How we are reducing Test Failure Analysis time using Machine Learning at Expedia

How many of us have spent those hours just to check Automation run reports to first determine whether it was an actual bug or environment specific issue or an automation issue. We agree that no matter how much robust we make our UI Automation frameworks, we always encounter Automation/Environment specific failures which increase the time spent on analysing those failures and spotting actual issues/defects. At some point, all of us have had a bad day where we see so many flaky tests, that we lose confidence in the reliability of UI Automation results and tend to ignore the results over a period of time. We propose a solution to help us overcome this problem using the most trending technology of last decade - Machine Learning. The fact that we run around 130k test runs a day with around 2.3 million test records saved in MongoDB every month motivated us to look into Machine Learning as an approach for the problem statement we have.

What if we can use ML algorithms to find patterns in the day-to-day UI Automation error messages that we see, to tell us if it is - an actual bug or not !!! All of us use various Selenium based Test Automation frameworks like Cucumber, TestNG, Scalatest, Nightwatch JS etc. which have their own libraries to report any Test Validation/Automation failures, hence vary quite a lot in the formats. Also, it's difficult to find a common pattern in user defined error messages. A typical error message from UI Automation would have the - Message Stack Traces , other error data dumped from Selenium etc. We can argue that we can take only the Message part of this as input to be able to predict the outcome for us. However, we have seen many instances where these Messages are not very self-explanatory and we will have to look into the trace/error details to actually determine the root cause. Considering the whole error message is not as easy as it sounds.

Hence, the problem we have at hand is unstructured text data. Our approach includes the steps -> Collect Training data (viz., pre-classified errors), Clean the data to be fed to the model, Identify a simple yet powerful algorithm such as SVM, Random Forest, Naive Bayes etc., Classifiers to work with, Tune the model to identify the right metrics to help us calculate the reliability of the resultant predictions.

This can also be extended to other error messages like Javascript Error Messages, Splunk or Trace logs as well.

 
 

Outline/Structure of the Talk

  1. Introduction to Problem Statement - Anuradha
  2. Solution Outline - Keertika
  3. How did we collect the training data - Keertika
  4. Various Algorithms Evaluated - Keertika
  5. Challenges we have faced - Keertika
  6. Expose the Model as a Service using an API Endpoint - Anuradha
  7. End-to-End setup using Docker, AWS Sagemaker, Lambda, S3, API Gateway, ECR etc. - Anuradha
  8. Integration with any UI Automation Frameworks/Libraries - Anuradha
  9. Usage Examples - Anuradha
  10. Impact/ Future Work - Keertika
  11. Queries/Discussion

Duration

  • 1 => 5 mins
  • 2 to 5 => 15 mins
  • 6 to 9 => 15 mins
  • 10 => 5 mins
  • 11 => 3-5 mins

Learning Outcome

  • Importance of Error Classification in Test Automation Space
  • Challenges with Unstructured Text Data (Raw Data)
  • How Machine Learning can help
  • End to End flow and how to deploy an ML model

Target Audience

Software Engineers, Software Engineers in Test, Technical Project Mangers, Scrum Masters, Engineering Managers

schedule Submitted 7 months ago

Public Feedback

comment Suggest improvements to the Speaker
  • Anuradha Konduri
    By Anuradha Konduri  ~  4 months ago
    reply Reply

    Outline/Structure of the Talk

    1. Introduction to Problem Statement  - Anuradha
    2. Solution Outline - Keertika
    3. How did we collect the training data - Keertika
    4. Various Algorithms Evaluated - Keertika
    5. Challenges we have faced - Keertika
    6. Expose the Model as a Service using an API Endpoint - Anuradha
    7. End-to-End setup using Docker, AWS Sagemaker, Lambda, S3, API Gateway, ECR etc. - Anuradha
    8. Integration with any UI Automation Frameworks/Libraries - Anuradha
    9. Usage Examples - Anuradha
    10. Impact/ Future Work - Keertika
    11. Queries/Discussion

    Duration

    • 1 => 5 mins
    • 2 to 5 => 15 mins
    • 6 to 9 => 15 mins
    • 10 => 5 mins
    • 11 => 3-5 mins
  • Deepti Tomar
    By Deepti Tomar  ~  6 months ago
    reply Reply

    Hello Anuradha & Keertika,

    Thanks for your proposal!

    Could you please reply to Maaret's questions?

    In addition, Could you please update the Outline/Structure section of your proposal with a time-wise breakup of how you plan to use 20 mins for the topics you've highlighted?

    And, to help the program committee understand your presentation style, can you provide a link to your past recording or record a small 1-2 mins trailer of your talk and share the link to the same? 

    Thanks!

    • Anuradha Konduri
      By Anuradha Konduri  ~  4 months ago
      reply Reply

      Hi Deepti,

      We have recorded a quick trailer of our presentation and its uploaded here : https://youtu.be/fpRL3f7qnRg

      Thanks,

      Anuradha

      • Manoj Kumar
        By Manoj Kumar  ~  4 months ago
        reply Reply

        The main idea was to have you(the speakers) in the video and speak. We don't need a screen share and a voice over on it, but actual video of the person talking so we can see the speaking ability.

    • Anuradha Konduri
      By Anuradha Konduri  ~  5 months ago
      reply Reply

      Hi Deepti,

      I have shared the youtube link to a quick trailer of our presentation we recorded as a Private message accessible to the Program Team. Please let me know if you are able to access it.

      Thanks!

      • Manoj Kumar
        By Manoj Kumar  ~  5 months ago
        reply Reply

        We're able to access, thank you.

    • Keertika Gangwar
      By Keertika Gangwar  ~  5 months ago
      reply Reply

      Hi Deepti, 

      Somehow my reply was trimmed, but now I have updated the comment again. We are working on small demo we'll share and update this proposal shortly. 

      Thanks,

      Keertika

  • Maaret Pyhajarvi
    By Maaret Pyhajarvi  ~  6 months ago
    reply Reply

    This is marked as a 20 minute talk with 8 discussion points and 2 presenters. Is the time marked correctly? And for 2 presenters, why does this topic (and your experience in particular) need two presenters rather than one from an audience takeaways perspective? 

    Interesting experience of cleaning up data and feeding that into ML - curious on whether the bottom line is that this was a learning experience (not useful results) or if this saved you from the efforts you outlined as motivation when you started.

    • Keertika Gangwar
      By Keertika Gangwar  ~  5 months ago
      reply Reply

      Hi Maaret, thanks for your comment and suggestion,  Anuradha and I have collaboratively worked on this project, in which my part was setting up the machine learning  model and overcome data challenges, while Anuradha helped in deployment of this model in production using AWS sagemaker.  That’s why our talk will be divided in two parts where I’ll be covering ML and data challenge part and Anuradha will be covering deployment part.

      As per your second ask the implementation of this model is completed and pushed to production, Team is using this automated categorization model for error segregation. Which helped in saving manual efforts and time.

       


  • Liked Dimpy Adhikary
    keyboard_arrow_down

    Dimpy Adhikary - Workshop: Automate web application using selenium and UI framework objects/APIs

    Dimpy Adhikary
    Dimpy Adhikary
    Test Architect
    Happiest Minds
    schedule 7 months ago
    Sold Out!
    90 Mins
    Workshop
    Intermediate

    We all love the tagline - Selenium automates browsers. Selenium provides self sufficient APIs for automating any user workflow exactly like an end user does. But as a automation engineer we see both side of the coin together. There is nothing simpler than selenium to interact with browser but same time selenium comes with a set of prerequisite before doing what exactly we want. Locator lookup strategy, page rendering frequency, event triggering and last but not least the wait context are some of those challenges we face in day to day basis while automating our apps. Other than that most of the latest UI frameworks comes with different set of UI controls which are sometime challenging to interact with only using selenium. In this workshop I would like to target fixing few of those issues by using the UI framework internals of application under test. Some specific use cases I will be sharing as part of this workshop are are as follows:

    • Performing actions directly using ui framework objects/APIs
    • Handling dynamic ids
    • Automating charts/graphs rendered in a canvas element

    For demonstration purpose I would limit the scope of this workshop to three web UI frameworks - AngularJs, ExtJs and ReactJs and see how we can use application framework internals to fix issues high lighted above along with selenium.