How to Un-Flake Flaky Tests- a New Hire's Toolkit
It's a common scenario - you are starting a new job as a test engineer, excited to improve your new organization's testing culture, eager to instill confidence in the software, and BAM! instead of writing new tests, you are tasked with fixing legacy ones! If this sounds familiar, this talk is for you. I'll cover strategies for taking inventory of flaky tests and setting goals to address flakiness across a test suite. In addition, you will learn to get to the bottom of what's causing flakiness in your tests, and to communicate with other engineering teams about those tests. Although everyone's application has its own quirks, and every organization has its own workflows, this talk aims to give advice that anyone can use in their own tests and when dealing with their fellow engineers. By the end of this talk, you will feel confident enough to debug your own flaky tests, and get to the fun part - writing new ones!
Here are some alternate presentation titles, for your amusement:
- Keep The Flakiness for Your Croissant: How to Un-Flake Your Automated Test
- Bake The Flake Out of Your Tasty Test Cake
- Flake It Off, Flake It Off: How to Un-Flake Your Flaky Test (T-Swift theme)
Outline/Structure of the Talk
- Getting a 'lay of the land' at your company as a new hire
- Understanding the test stack
- Understanding the application under test (don't forget this step!)
- Are the tests white box/grey box/black box?
- Understanding state in the test. Are there strategies in place to deal with state?
- Are there any ways of dealing with common problems throughout the suite? Are there any 'heavy hammers', aka, tools that are overapplied?
- Is there a central repository for test run results? Can you collect and analyze these?
- Creating a long term strategy to tackle flakiness
- Collecting and classifying flaky tests
- Creating trackable goals for the test suite
- Establishing benchmarks for asserting that the test code is un-flaked before committing it
- Establishing a workflow for adding new flaky tests to the collection of flaky tests, as the tests are run
- Holding test engineers accountable to the rest of the engineering org
- Techniques for debugging flaky tests
- Writing scripts to automatically put break points in your automated test run, at key points, so you can step through the test code
- Reducing variables in your investigation of the culprit of flakiness
- Capturing and analyzing screenshots and HTML upon test fail
- Diffing HTML with KDiff to 'catch' the point of change in the application
- Turning on the Selenium $DEBUG log
- Chrome debugging tips
- Watching the network tab as the test continues exection
- Using the chrome dev console
- How to talk about flakiness with other engineers
Everyone leaving my session should feel more confident forming a strategy to tackle flakiness in their test suites. They should also have some techniques for uncovering what's really happening in the DOM, and eliminating variables when it comes to what's wrong in a test.
New test engineers, or test engineers that deal with flaky test code
Prerequisites for Attendees
This talk should be broadly accessible to anybody who uses Selenium, understands client/server architecture, and understands ruby code, although the code examples should transcend language.
schedule Submitted 3 years ago
People who liked this proposal, also liked:
David Kirk - Selenium Grid at Scale through Docker SwarmDavid KirkSr. DevOps EngineerReviewTrackers
schedule 3 years agoSold Out!
Selenium Grid enables developers to significantly parallelize their testing pipeline. This strength can be empowered even further by leveraging the capabilities of Docker Swarm to efficiently utilize available resources and dynamically scale your testing capabilities. With the tools provided by Selenium and Docker, we can build a robust, scalable, and cost efficient browser testing platform without a significant degree of difficulty.
In this talk, we will cover how to build a Selenium Grid through docker swarm, how to scale that grid, how to test against it, what sort of computing resource requirements should be provided, and my personal experiences implementing this system.
Ivan Krutov - Selenoid: get rid of Selenium server!Ivan KrutovDeveloperAerokube
schedule 3 years agoSold Out!
Browser tests are known to be the flakiest ones. This is partly because browser infrastructure is complicated to maintain. But the second reason is – main stream browser automation tools such as Selenium server are far from being efficient.
As your company grows – your browser automation infrastructure should easily adapt to increased loads and new requirements. Unfortunately this is not so easy to get really scalable Selenium cluster using only the main stream tools. But there are some new open-source tools and approaches that can be used to deliver really scalable Selenium cluster.
This tutorial covers an emerging browser automation tool – Selenoid, a truly efficient replacement of the standard Selenium server. I will explain how it works and why it is so fast. I will show in details features like video recording, sending logs and statistics to centralized storage and how to easily visualize this data.