How we are reducing Test Failure Analysis time using Machine Learning at Expedia

location_city Virtual Platform schedule Sep 12th 11:30 AM - 12:15 PM place Online Meeting 2 people 42 Interested

How many of us have spent those hours just to check Automation run reports to first determine whether it was an actual bug or environment specific issue or an automation issue. We agree that no matter how much robust we make our UI Automation frameworks, we always encounter Automation/Environment specific failures which increase the time spent on analysing those failures and spotting actual issues/defects. At some point, all of us have had a bad day where we see so many flaky tests, that we lose confidence in the reliability of UI Automation results and tend to ignore the results over a period of time. We propose a solution to help us overcome this problem using the most trending technology of last decade - Machine Learning. The fact that we run around 130k test runs a day with around 2.3 million test records saved in MongoDB every month motivated us to look into Machine Learning as an approach for the problem statement we have.

What if we can use ML algorithms to find patterns in the day-to-day UI Automation error messages that we see, to tell us if it is - an actual bug or not !!! All of us use various Selenium based Test Automation frameworks like Cucumber, TestNG, Scalatest, Nightwatch JS etc. which have their own libraries to report any Test Validation/Automation failures, hence vary quite a lot in the formats. Also, it's difficult to find a common pattern in user defined error messages. A typical error message from UI Automation would have the - Message Stack Traces , other error data dumped from Selenium etc. We can argue that we can take only the Message part of this as input to be able to predict the outcome for us. However, we have seen many instances where these Messages are not very self-explanatory and we will have to look into the trace/error details to actually determine the root cause. Considering the whole error message is not as easy as it sounds.

Hence, the problem we have at hand is unstructured text data. Our approach includes the steps -> Collect Training data (viz., pre-classified errors), Clean the data to be fed to the model, Identify a simple yet powerful algorithm such as SVM, Random Forest, Naive Bayes etc., Classifiers to work with, Tune the model to identify the right metrics to help us calculate the reliability of the resultant predictions.

This can also be extended to other error messages like Javascript Error Messages, Splunk or Trace logs as well.


Outline/Structure of the Talk

  1. Introduction to Problem Statement - Anuradha
  2. Solution Outline - Keertika
  3. How did we collect the training data - Keertika
  4. Various Algorithms Evaluated - Keertika
  5. Challenges we have faced - Keertika
  6. Expose the Model as a Service using an API Endpoint - Anuradha
  7. End-to-End setup using Docker, AWS Sagemaker, Lambda, S3, API Gateway, ECR etc. - Anuradha
  8. Integration with any UI Automation Frameworks/Libraries - Anuradha
  9. Usage Examples - Anuradha
  10. Impact/ Future Work - Keertika
  11. Queries/Discussion


  • 1 => 5 mins
  • 2 to 5 => 15 mins
  • 6 to 9 => 15 mins
  • 10 => 5 mins
  • 11 => 3-5 mins

Learning Outcome

  • Importance of Error Classification in Test Automation Space
  • Challenges with Unstructured Text Data (Raw Data)
  • How Machine Learning can help
  • End to End flow and how to deploy an ML model

Target Audience

Software Engineers, Software Engineers in Test, Technical Project Mangers, Scrum Masters, Engineering Managers

schedule Submitted 1 year ago

Public Feedback

    • Dimpy Adhikary

      Dimpy Adhikary - Workshop: Automate web application using selenium and UI framework objects/APIs

      Dimpy Adhikary
      Dimpy Adhikary
      Test Architect
      Happiest Minds
      schedule 1 year ago
      Sold Out!
      90 Mins

      We all love the tagline - Selenium automates browsers. Selenium provides self sufficient APIs for automating any user workflow exactly like an end user does. But as a automation engineer we see both side of the coin together. There is nothing simpler than selenium to interact with browser but same time selenium comes with a set of prerequisite before doing what exactly we want. Locator lookup strategy, page rendering frequency, event triggering and last but not least the wait context are some of those challenges we face in day to day basis while automating our apps. Other than that most of the latest UI frameworks comes with different set of UI controls which are sometime challenging to interact with only using selenium. In this workshop I would like to target fixing few of those issues by using the UI framework internals of application under test. Some specific use cases I will be sharing as part of this workshop are are as follows:

      • Performing actions directly using ui framework objects/APIs
      • Handling dynamic ids
      • Automating charts/graphs rendered in a canvas element

      For demonstration purpose I would limit the scope of this workshop to three web UI frameworks - AngularJs, ExtJs and ReactJs and see how we can use application framework internals to fix issues high lighted above along with selenium.