schedule Jun 30th 03:00 - 03:45 PM place Kalinga Hall 2 people 69 Interested

It's a common scenario - you are starting a new job as a test engineer, excited to improve your new organization's testing culture, eager to instill confidence in the software, and BAM! instead of writing new tests, you are tasked with fixing legacy ones! If this sounds familiar, this talk is for you. I'll cover strategies for taking inventory of flaky tests and setting goals to address flakiness across a test suite. In addition, you will learn to get to the bottom of what's causing flakiness in your tests, and to communicate with other engineering teams about those tests. Although everyone's application has its own quirks, and every organization has its own workflows, this talk aims to give advice that anyone can use in their own tests and when dealing with their fellow engineers. By the end of this talk, you will feel confident enough to debug your own flaky tests, and get to the fun part - writing new ones!

Here are some alternate presentation titles, for your amusement:

  • Keep The Flakiness for Your Croissant: How to Un-Flake Your Automated Test
  • Bake The Flake Out of Your Tasty Test Cake
  • Flake It Off, Flake It Off: How to Un-Flake Your Flaky Test (T-Swift theme)
 
14 favorite thumb_down thumb_up 7 comments visibility_off  Remove from Watchlist visibility  Add to Watchlist
 

Outline/Structure of the Talk

  • Getting a 'lay of the land' at your company as a new hire
    • Understanding the test stack
    • Understanding the application under test (don't forget this step!)
    • Are the tests white box/grey box/black box?
    • Understanding state in the test. Are there strategies in place to deal with state?
    • Are there any ways of dealing with common problems throughout the suite? Are there any 'heavy hammers', aka, tools that are overapplied?
    • Is there a central repository for test run results? Can you collect and analyze these?
  • Creating a long term strategy to tackle flakiness
    • Collecting and classifying flaky tests
    • Creating trackable goals for the test suite
    • Establishing benchmarks for asserting that the test code is un-flaked before committing it
    • Establishing a workflow for adding new flaky tests to the collection of flaky tests, as the tests are run
    • Holding test engineers accountable to the rest of the engineering org
  • Techniques for debugging flaky tests
    • Writing scripts to automatically put break points in your automated test run, at key points, so you can step through the test code
    • Reducing variables in your investigation of the culprit of flakiness
    • Capturing and analyzing screenshots and HTML upon test fail
    • Diffing HTML with KDiff to 'catch' the point of change in the application
    • Turning on the Selenium $DEBUG log
    • Chrome debugging tips
      • Watching the network tab as the test continues exection
      • Using the chrome dev console
  • How to talk about flakiness with other engineers

Learning Outcome

Everyone leaving my session should feel more confident forming a strategy to tackle flakiness in their test suites. They should also have some techniques for uncovering what's really happening in the DOM, and eliminating variables when it comes to what's wrong in a test.

Target Audience

New test engineers, or test engineers that deal with flaky test code

Prerequisites for Attendees

This talk should be broadly accessible to anybody who uses Selenium, understands client/server architecture, and understands ruby code, although the code examples should transcend language.

schedule Submitted 1 year ago

Public Feedback

comment Suggest improvements to the Speaker
  • Pooja Shah
    By Pooja Shah  ~  1 year ago
    reply Reply
    Interesting and here I see a lot of potentials to take away the learnings. Thank you Zachary for proposing this.
    - Curious, if you are going to cover each section along with more and more real-time examples so that the attendees could relate more or you plan to do a demo something like keep going gradual story building on the fly which showcases the problem and addressing the debugging/prevention/solution or what's your approach.
    - It would be great if you could provide any of your previous talk link (any topic) or if none then share a 1-2 minutes video shoot (Just an idea: can be just briefing why/what you are going to talk, something like why as an attendee, someone should join this talk)
    • Zachary Attas
      By Zachary Attas  ~  1 year ago
      reply Reply

      Hey Pooja, thanks so much for your comment.  Each major section above will come with real examples.  And yes, in the talk I will be building a story as I go through each section.  The story is my first couple of months as a Senior Test Engineer at ReviewTrackers, but I'll also throw in other examples from my time as a Test Engineer at another company called Jellyvision.  

      The biggest section is 'Techniques for debugging flaky tests'.  For this section, I have a collection of GIFs that illustrate different phenomena in the DOM that can cause flaky tests, and my approaches for finding and solving them. I consider myself a very scrappy developer and I have many techniques for pinpointing what's gone wrong.  I'll show some brief code examples as well.  The audience will be able to apply these techniques to their code bases. However, the sections after that are brief, but important. They deal with the soft skills when discussing flaky tests with other engineers, and making test architecture decisions to prevent flaky tests. 

      I don't have any previous talk links, but I'll shoot a quick video showing why someone should attend this talk!

       

      • Pooja Shah
        By Pooja Shah  ~  1 year ago
        reply Reply

        cool, this sounds like a plan and now I can visualise how you plan to go through. 

        Awaiting the video.

        • Zachary Attas
          By Zachary Attas  ~  1 year ago
          reply Reply

          Hey Pooja,  I just posted the video, it's in the links section of my proposal.  Hope you like it!

          • Naresh Jain
            By Naresh Jain  ~  1 year ago
            reply Reply

            Thanks, Zachary. We are interested in this topic. But need some time to finalize. Please stay tuned.

            • Zachary Attas
              By Zachary Attas  ~  1 year ago
              reply Reply

              For sure, I understand. Let me know if there's any additional information I can provide!  And I'm very open to recommendations or tips concerning things to focus on or structure differently in the talk.

  • Jen
    By Jen  ~  1 year ago
    reply Reply

    This sounds amazing!!


  • Liked David Kirk
    keyboard_arrow_down

    David Kirk - Selenium Grid at Scale through Docker Swarm

    David Kirk
    David Kirk
    Sr. DevOps Engineer
    ReviewTrackers
    schedule 1 year ago
    Sold Out!
    45 Mins
    Tutorial
    Beginner

    Selenium Grid enables developers to significantly parallelize their testing pipeline. This strength can be empowered even further by leveraging the capabilities of Docker Swarm to efficiently utilize available resources and dynamically scale your testing capabilities. With the tools provided by Selenium and Docker, we can build a robust, scalable, and cost efficient browser testing platform without a significant degree of difficulty.

    In this talk, we will cover how to build a Selenium Grid through docker swarm, how to scale that grid, how to test against it, what sort of computing resource requirements should be provided, and my personal experiences implementing this system.

  • Liked Ivan Krutov
    keyboard_arrow_down

    Ivan Krutov - Selenoid: get rid of Selenium server!

    Ivan Krutov
    Ivan Krutov
    Developer
    Aerokube
    schedule 1 year ago
    Sold Out!
    90 Mins
    Tutorial
    Beginner

    Browser tests are known to be the flakiest ones. This is partly because browser infrastructure is complicated to maintain. But the second reason is – main stream browser automation tools such as Selenium server are far from being efficient.

    As your company grows – your browser automation infrastructure should easily adapt to increased loads and new requirements. Unfortunately this is not so easy to get really scalable Selenium cluster using only the main stream tools. But there are some new open-source tools and approaches that can be used to deliver really scalable Selenium cluster.

    This tutorial covers an emerging browser automation tool – Selenoid, a truly efficient replacement of the standard Selenium server. I will explain how it works and why it is so fast. I will show in details features like video recording, sending logs and statistics to centralized storage and how to easily visualize this data.