How to fight with unstable tests

The talk is about flaky, or unstable tests – why it’s the biggest trouble in test automation and how to overcome it. I will show my collection of real-life examples of flaky tests from my experience. These are really tricky investigations that lasted many days, weeks and even years!
The talk will be interesting to both testers and developers – anyone who loves to solve unsolvable puzzles.

 
 

Outline/Structure of the Case Study

  1. Intro: what are flaky tests and why it's a disaster? [5 min]
  2. My collection of flaky tests [25 min]
  3. Reasons of flakiness [5 min]
  4. How to fight them? [5 min]
  5. Best practices to prevent flaky tests [5 min]

 

Learning Outcome

You will learn the typical reasons of flaky tests.

You will see how Selenide can help with most of them. 

You will watch several breathtaking real-life examples of flaky tests with videos or detailed explanation. 

We will sum up best practices how to prevent flaky tests.

Target Audience

QA engineers tired of flaky tests :)

Prerequisites for Attendees

no prerequisities

Slides


schedule Submitted 1 year ago

  • Sai Krishna
    keyboard_arrow_down

    Sai Krishna / Srinivasan Sekar - Advanced Appium 2.0

    480 Mins
    Workshop
    Advanced

    At this workshop, you'll learn about advanced concepts in Appium. We will learn how to write a single script for Android, iOS, and mobile web apps. Once our tests are established, we will work on the framework design and report. We will also have a detailed look at how can we automate various mobile gestures using Actions API and mobile endpoints in Appium. The workshop will be based on the latest Appium's Java Client.

     

     

  • Srinivasan Sekar
    keyboard_arrow_down

    Srinivasan Sekar / Sai Krishna - Build your own Appium 2.0 Driver

    45 Mins
    Case Study
    Beginner

    What if you had to create a custom driver for your use case in your project and with Appium 1.0 it was not possible without making any code changes to the actual appium code base. This would lead us to maintain a fork and make our required changes. Which in turn is overkill as we have to always keep up with upstream. 

    With Appium 2.0 architecture we can create drivers and plugins for our unique needs. At this talk, we will talk about Appium 2.0 architecture, how to make custom Appium drivers and what other breaking changes we brought in Appium 2.0.

  • Benjamin Bischoff
    keyboard_arrow_down

    Benjamin Bischoff - Of plugins and decorators - trivago's e2e test framework in the spotlight

    45 Mins
    Talk
    Intermediate

    Most often, test frameworks are developed from scratch and gradually extended with the required functionality. This approach is perfectly normal, but carries the risk of creeping technical debt as well as poor maintainability, transparency and extensibility. These risks can be minimized by using clean software architecture.

    In this talk, I will present two key parts of our internal Selenium-based end-to-end testing framework, which has been successfully used to test multiple web projects using desktop and mobile browsers during the past five years:

    1. the plugin infrastructure using Java's Service Provider Interface
    2. the custom WebElement decorator that enables more resilient Selenium WebElements

    I will explain the rationale for using these architectural tools and how these approaches mean that we can still evolve this framework quickly and easily to meet new needs.

  • Gaurav Singh
    keyboard_arrow_down

    Gaurav Singh - Hello Espresso! Start with Android Gray box automation

    Gaurav Singh
    Gaurav Singh
    Software Engineer
    Meta
    schedule 1 year ago
    Sold Out!
    90 Mins
    Tutorial
    Beginner

    TL;DR Summary

    • Espresso is a powerful UI automation framework for the Android platform that offers gray box automation capabilities and has a simple, concise, and rich API. 
    • Onboarding to use espresso can however sometimes take time. In this talk, I will explain how to get started with Espresso in the shortest time possible and we’ll look at some recipes on how to automate common app scenarios with espresso API.

    Abstract

    • It’s 2022 and most businesses these days are mobile-first with a presence on major platforms. UI automation is a critical part of the testing strategy of businesses looking to release high-quality apps with confidence. While unit testing is a much wider adopted practice, writing scalable UI automation is often a challenge due to its higher fidelity.
    • Espresso is the dominant UI automation library for android from Google that provides a rich and concise API to test your android app's UI with confidence without sacrificing reliability and with minimal flakiness due to amazing integration with underlying instrumentation.
    • For an n00b engineer approaching their first Espresso test, it could be especially tough to wrap your head around its idioms and android context and get set up quickly. 
    • Well, In this talk:
      • we’ll solve for that exact persona and provide a quick understanding of an espresso test structure
      • Follow up by diving deeper into its API
      • Discuss different ready to use recipes that developers could use to solve some common use cases with mobile UI automation
      • Provide a boilerplate framework
      • I'll share Github repo links that have app source code along with their Espresso UI tests to follow along or revisit later.

     

  • Puja Chiman Jagani
    keyboard_arrow_down

    Puja Chiman Jagani - Selenium has a new trick up its sleeve to track failures

    Puja Chiman Jagani
    Puja Chiman Jagani
    Team Lead
    Browserstack
    schedule 1 year ago
    Sold Out!
    45 Mins
    Talk
    Beginner

    As our systems and tests grow more and more complex we need to make sure that we have the tools to capture the root causes without spending hours or days chasing them down. This is where Observability becomes our best friend. Observability allows us to see what is going on inside a system based on what we think is crucial without trawling through logs! Just like any piece of software should be robust, scalable, maintainable, and reliable, it should also be observable. Observability makes the journey from identifying unexpected problems to identifying the root cause easier.

    To do so, the code should record as much useful granular information as possible. Metrics, logs, and traces are three known ways of encapsulating granular information. They are the primary sources of information to help determine the state of the system at any given point in time. 

    Selenium 4 introduced a fully distributed Grid with multiple components that communicate over the network. Troubleshooting and diagnosing problems in this setup is a challenge. To tackle this, Selenium integrated OpenTelemetry’s tracing and event logs.  This feature is now available out of the box by default when using Selenium.
    The users now have more power in their hands!

    I will dive into Selenium's observability journey by discussing: 

    1. What is observability?

    2. Need for observability

    3. Understanding the three pillars of observability: Metrics, Logging, and Tracing

    4. Generating telemetry data alone does not suffice. It is a process from design to deployment. 

    5. Full-stack tracing in Selenium (Grid and Java client library)

    6. Explain how we, at BrowserStack, are benefiting and exposing this information to our users.

  • Matthias Zax
    keyboard_arrow_down

    Matthias Zax - DevSecOps on steroids - Powered by your Selenium tests

    45 Mins
    Talk
    Beginner

    Continuous Security testing is becoming more and more a key factor for success. Especially if we consider that the development and release process is speeding up enormously. Just imagine that your potential shippable product is going to production with a huge vulnerability or a back door open. The damage to your company and bad reputation would be even not measurable.

    So how can we avoid this? How can we build- security -in? Let's leave the stone age behind, break down the security silo and implement DevSecOps.

    During my talk, I will tell you where you can implement and improve security testing. What different kinds of functional and non-function security testing methods are available and what are the low-hanging fruits.
    On a high level, I will explain SAST / DAST / IAST / RASP and how your team could implement these methods with examples. I present how you can use your existing selenium scripts to drive OWASP ZAP and get more out of it!
    Then I will lift it to the next level and show how you can add security testing to your pipeline to get fast feedback to fix the vulnerabilities at a very early stage (shift left). By showing where to implement security tests in your software development lifecycle, I will explain where it makes sense to have security as a deep skill part of your team and go for DevSecOps!

    After I increased our transparency of security and showed you how to deal with "the four fists", I will close my talk by presenting the 10 successful steps to DevSecOps.

  • Andrei Solntsev
    keyboard_arrow_down

    Andrei Solntsev - Extending open-source libraries on the example of Selenide & Selenium

    Andrei Solntsev
    Andrei Solntsev
    Software developer
    Codeborne
    schedule 1 year ago
    Sold Out!
    45 Mins
    Talk
    Advanced

    You use lots of open-source libraries, but do you know how to customize every single piece of their behaviour? 

    How to make every piece of your software customizable?

    When I had to implement plugins in Selenide, I didn't find any good materials on this topic, and started investigating how other popular libraries do. In this presentation, I will show the results of my research. 

  • Anton Angelov
    keyboard_arrow_down

    Anton Angelov - Become Black Ops QA with Selenium 4 BiDi-DevTools APIs

    Anton Angelov
    Anton Angelov
    CTO
    Automate The Planet
    schedule 1 year ago
    Sold Out!
    45 Mins
    Talk
    Intermediate

    Anton Angelov will explain in many details and code examples how to automate hard-to-test web app functionalities using Selenium 4 BiDirectional and Chrome DevTools APIs. We will discuss test automation of progressive web applications, web sockets, memory leak checks. In addition, you will learn how to perform internationalization, localization, and network performance testing. Finally, we will investigate how you can use the protocols to speed up your tests using the black hole proxy pattern.

  • Boni García
    keyboard_arrow_down

    Boni García - WebDriverManager: the Swiss Army Knife for Selenium WebDriver

    45 Mins
    Tutorial
    Beginner

    Selenium WebDriver is a library that allows controlling web browsers (e.g., Chrome, Firefox, Edge, etc.) programmatically using different languages (such as Java, JavaScript, Python, Ruby, or C#). Internally, Selenium WebDriver uses the native support implemented by each browser to carry out the automation process. For this reason, we need to place a component called driver (e.g., chromedriver, geckodriver, msedgedriver, etc.) between the script using the Selenium WebDriver API and the browser. WebDriverManager is an open-source Java library that carries out the management (i.e., download, setup, and maintenance) of the drivers required by Selenium WebDriver in a fully automated manner. In addition, as of version 5, WebDriverManager provides other relevant features, such as discovering browsers installed in the local system and building WebDriver objects, and running browsers in Docker containers seamlessly. The last feature added to WebDriverManager is related to log gathering. As of version 5.2.0, WebDriverManager provides seamless integration with BrowserWatcher, an open-source browser extension that allows gathering the browser console programmatically (even for Firefox, which was not possible to date). 

  • Gayathri Mohan
    keyboard_arrow_down

    Gayathri Mohan / Pallavi Vadlamani - How to approach Continuous Testing of Cross-Functional Requirements?

    45 Mins
    Talk
    Advanced

    Cross-functional requirements (CFRs), predominantly referred to as Non-functional Requirements (NFRs), form an integral part of software quality and testing for them continuously is an absolute necessity for any team that promises to deliver high-quality software to their customers and end-users. Often, the emphasis that is placed on continuous testing (CT) the functional requirements are not equally placed on continuous testing the cross functional requirements in software delivery teams and by the business stakeholders. There could be multiple reasons catering to this phenomenon but one we believe could be a prominent reason is the lack of awareness on how to approach CFRs testing as they come across really vague, say, reliability or maintainability, for example. 

    In this talk, I, author of O'Reilly's Full Stack Testing book, and my colleague from Thoughtworks, Pallavi Vadlamani, would like to elaborate why it is essential to do continuous testing for cross-functional requirements and introduce a holistic approach to continuous testing of all the cross-functional requirements (the approach is carved as part of Gayathri's book). We will cover how different testing techniques such as static code analysis, architecture tests, visual tests, infrastructure tests, load tests, etc., cater to automating a variety of CFRs, including those really vague ones, and therefore, aid in continuous testing and building quality into the software. 

    We would continue the talk with applying the approach to a couple of CFRs specifically, say, security and accessibility, to give the audience a solid grasp on the approach. By the end of the talk, the audience should get a clear idea of how to approach continuous testing of any given CFR that is necessary for them to deliver a high-quality software.

  • Cameron Bradley
    keyboard_arrow_down

    Cameron Bradley - Why does building a robust automation framework matter?

    45 Mins
    Talk
    Beginner

    This talk will dive into the importance of building a robust automation framework architecture that leverages Object Orientated Design Principles. 

    Why does automation architecture matter? Clean code, Flexible, Easy to read, interpret, maintain and build upon.

    Come and see how this automation framework approach has enabled multiple companies, projects and team members (across a variety of skill levels) to become truly efficient automation engineers.

  • Eran Kinsbruner
    keyboard_arrow_down

    Eran Kinsbruner - Comparing Selenium 4, Cypress and Playwright Frameworks

    Eran Kinsbruner
    Eran Kinsbruner
    Global Evangelist
    Perfecto
    schedule 1 year ago
    Sold Out!
    45 Mins
    Talk
    Intermediate

    As the cross-browser testing landscape evolves, there is a stabilization around 3 main leading frameworks that are used the most. Some are more adopted by frontend developers, while some are adopted by SDETs. To better make a selection or even choose more than a single framework you need to realize the core differences and use cases as well as benefits each framework provides to the practitioner.

    In this session, Eran Kinsbruner, Chief DevOps evangelist, best-selling author from Perforce (Perfecto and Blazemeter specifically) will go through the core differences and advantages of all 3 frameworks with focus on Selenium 4 vs. the latest versions of Cypress and Playwright and provide a nice outline to better guide test automation engineers as they look for their next test framework.

  • Parasar Saha
    keyboard_arrow_down

    Parasar Saha - Cloud First Automation Approach – Road to Scalable and Faster Test Automation Pipelines

    Parasar Saha
    Parasar Saha
    CEO
    Digy4
    schedule 1 year ago
    Sold Out!
    45 Mins
    Talk
    Advanced

    Cloud has changed the way the software world works. Companies have accelerated their development and deployment cycles with 3x or more speed by adopting the cloud. But Testing in most organizations either is running locally or not taking advantage of the full potential of cloud infrastructure. Testing is the home-alone kid, left to spend the Christmas weekend without the bells the whistles that cloud has to offer.

     

    Digital transformation in IT and Agile delivery model are pushing the testing teams to complete their testing in shorter testing windows. 26 % of the organization are moving to daily releases. 70% of the organization have adopted Agile in delivery. Even after having a high amount of automation coverage teams are struggling to keep up with the pace of testing asked by the business teams.

     

    This talk is about how you can change the story and accelerate cloud adoption in your testing organization with Selenium-based frameworks. Through cloud adoption you benefit from box features of the cloud: On-demand scaling, Cost optimation, Security, Reliability, Geo -routing of your test infrastructure. Cloud adoption doesn’t need to limit to the execution of selenium tests but can help you in test management of automated selenium tests, test data generation, test reporting, test artifact store, and many more. It is not enough to have an Automation First Approach these days, you also need to have Cloud-First Approach for test infrastructure to get the best value from automation.

    The talk elaborates on factors to consider while building up the cloud-based testing pipeline and how they will benefit your organization in terms of time to market, cost optimization, and raising the bar of quality.

    Finally talks about the moonshot of having a Testing Cloud – AWS of Testing and what will it take to reach there as a testing community.

    Take your testing to the cloud, and fly your flag high in quality!

  • Mahesh Mal
    keyboard_arrow_down

    Mahesh Mal - "I am a tester and I can build my own test infrastructure & environments" - what does it take for a tester to say this confidently

    Mahesh Mal
    Mahesh Mal
    Engineering Manager
    Ellucian
    schedule 1 year ago
    Sold Out!
    45 Mins
    Talk
    Beginner

    We have seen a lot of times testing teams complaining that they have been waiting days and months to get a test environment, and this becomes a pain when they have to test something to meet deadlines. Depending on external teams for the testing environment is pain, we transitioned and built a testing team that was capable Of building their own infrastructure, environments for all testing activities, and how the entire process was automated based on test team requirements

  • Sidhartha Shukla
    keyboard_arrow_down

    Sidhartha Shukla - Continuous Testing With Selenium Automation

    Sidhartha Shukla
    Sidhartha Shukla
    SDET-II
    Amazon
    schedule 1 year ago
    Sold Out!
    20 Mins
    Talk
    Intermediate

    Continuous testing is the process of executing automated test cases as part of the software delivery pipeline to obtain immediate feedback on the business risks and add deployment blocker in Pipelines in case of any Error during test suite execution. “Continuous Testing allows any change made in the code to be tested immediately. This avoids the problems created by having “bigbang” testing left to the end of the cycle such as release delays and quality issues.

    Continuous testing is decisive for Full CI-CD in the project and to achieve it, we need to make our Automation very robust and impeccable.To implement continuous testing, we have to automate tests and create test suites(e.g. Regression, Smoke, Sanity, BVT, End to End, Functional). To design the automation suite, we will use the most stable automation framework (SELENIUM). Selenium allows the tests to be automated quickly, and execution is done as soon as a piece of code is ready at their respective stages of the release pipeline. So we will show in this topic that how quickly we can automate using selenium and it can be integrated into our our existing pipeline with ease. One point is preeminent to note that "Automation is the most integral part of continuous testing", and Selenium is the framework which helps us to achieve it.

    Continuous testing can be achieved using any CI-CD like Bamboo, CircleCI, Amazon CDK pipeline, and the best part is that Selenium can be integrated with all the CI-CD easily and efficiently. So just to give the audience a proper visualization of Continuous testing, we will use Jenkins as our CI-CD tool.


    Target Plan of Action :

    ==> Define Scope and Estimation.

    ==> Identify important features and scenarios, which will be part of Smoke/Sanity/BVT suite.

    ==> Develop Automation Framework with Selenium

    ==> Design test suite.

    ==> Automate Deployment process for both DEV/QA/UAT environment.

    ==> Create Jenkins job for automation suite by integrating pom.xml with Jenkins.

    ==> Create pipeline with deployment and automation jobs.

    ==> Integrate Email notification for Jenkins job execution status.

    ==> Add GIT web hook to auto trigger jobs with code push.

     

    Benefit Of Continous Testing:

    ==> We can add Deployment blocker in Pipelines in case of any Error during suite execution

    ==> More Test Reliability

    ==> Faster Release Rate

    ==> Reduce Costs

    ==> Easy Maintenance and Updates

    ==> Auto triggered jobs

    ==> Testing with more accuracy

    ==> Automated Email Notification for deployment and automation suite execution

    ==> Flawless and quick deployment

    Advance Plan of Action:

    ==> Automation regression suite with maximum coverage

    ==> Use script for job creation in Jenkins

    ==> Code quality with sonarqube integration

    ==> Slack notification for jenkins job execution report

    ==> Coverage report

    ==> Stabilize automation test scripts

  • Andrei Solntsev
    keyboard_arrow_down

    Andrei Solntsev - How to migrate from Selenium to Selenide: less boilerplate, more stable tests!

    Andrei Solntsev
    Andrei Solntsev
    Software developer
    Codeborne
    schedule 1 year ago
    Sold Out!
    45 Mins
    Workshop
    Intermediate

    Selenium WebDriver is a great tool, but it's not a testing library. It's a browser manipulation tool. Still, many QA projects are built on pure Selenium. 

    Selenide is a testing library based on Selenium for making stable and readable tests.

    In this session, I will show how you can easily migrate your Java project from pure Selenium to Selenide. You will see how to deal with webdriver initialization, enable automated screenshots, simplify page objects and use additional Selenide features. 

    And no, you don't ultimately need to refactor the entire project. You can leave old code as is, and use Selenide only in some critical parts and new code. 

  • Marco Cruz
    keyboard_arrow_down

    Marco Cruz - Deep dive into Selenium 4 architecture and features

    Marco Cruz
    Marco Cruz
    Founder
    automateNow
    schedule 1 year ago
    Sold Out!
    45 Mins
    Demonstration
    Beginner

    Selenium 4 boasts new and exciting features as well as a redesigned architecture. The architectural changes help modernize Selenium, while new features will give you many unique advantages such as increased productivity, reduced complexity, and superpowers with access to the Chrome DevTools Protocol (CDP)!
    Come learn all about what's new and how to put it to use.

  • Thivya Lakshimi
    keyboard_arrow_down

    Thivya Lakshimi / Abinaya Veluswamy - Shift left your Cross Functional Requirement testing

    45 Mins
    Talk
    Beginner

    Over the recent years, agile teams have started practicing the shift left test approach to speed up their product deliveries and remove testing bottlenecks. Often teams tend to shift functional testing to the left of delivery cycle addressing the cross functional requirements(CFR) tests at the end.

    The functional requirements of the product defines the behaviour of the system whereas the Cross-Functional Requirements(CFRs) define the operational attributes of the system like performance, availability, reliability, usability, accessibility, audit, logging, recoverability, user volume, security, localisation, etc.

    Consider a case where an online retail application has launched its big sale days with exceptionally great deals, triggering the enthusiasm of millions of users to access the application at the same time. The moment the big sale started, the website received a peak load from its users. Unable to withstand the load, the application was inaccessible until a suitable fix was done by the engineering team. Would this make the customers happy with online shopping experience? How many people would stay in the application until it’s up again? Would they shop in the application if this repeats?  If CFRs had been tested effectively with proper focus, we could avoid facing this situation of losing hundreds of customers.

    This session talks about the importance and need to shift CFR tests to the left of delivery cycle for high confident and quality releases. It gives step by step explanation of shift left process and practices that helps build CFRs into the product right from project kickoff till delivery along with an example case study.

  • Syam Sasi
    keyboard_arrow_down

    Syam Sasi - Integrating the desktop and mobile browser testing into the continuous delivery pipeline

    Syam Sasi
    Syam Sasi
    Senior Software Engineer
    Carousell
    schedule 1 year ago
    Sold Out!
    45 Mins
    Talk
    Beginner

    Automated Web UI testing can be challenging when the UI flows on the desktop web and mobile web are different because the product experience is separately optimized for each screen size.

     

    In this talk, I will be discussing about

    • Accessing native device components on the mobile web
    • Minimising the complexity with the help of  the business layer concept
    • Executing the desktop and mobile tests into the continuous delivery pipeline using Selenium, Docker, and Jenkins pipeline.
    • Sharing a template for selecting which cloud provider suits you for web automation testing.

    Techstack Used: Java, Selenium, Appium, Python, Docker

help