Helping Test & Test Automation with AI-ML

"To implement or use AI (mostly creating a solution to help in testing) you don't need to be an expert or some certified data scientist etc. My talk revolves around how being a normal automation tester, we see some challenges and with limited knowledge, we start leveraging AI to help in our test. Talk create a mindset and case study how AI may help you in some of your day to day challenges in testing."

The impact of AI has penetrated our lives and increasing daily. Related to AI-ML existence in Testing, two scenarios are:

- Testing AI programs

- AI helping Testing

This talk is related to later aspects of "AI Helping Testing". There are a number of possible ways how it is impacting and many companies are already working on developing tools around same and many test solutions are already available in market.

Whenever AI keywords are intercepted, the common image is of "autonomous car, robots" and then the question, how these cars and robots will help us in testing, rather the thought should be these the outputs of AI.

AI/ML can be leveraged in the number of areas and scenarios to solve and help in our day to day testing activities. This talk would discuss about AI/ML, its impact, some existing solutions available, doing brainstorming ideas, so you can identify in your project. Also, USE CASE how we took AI benefit to solve our Automation problem.

Use Case - Problem Statement

1- Multiple automation suites running daily and sharing reports. Each report is having some failures. To do defect triaging for multiple failures is difficult.

Solution: Displaying consolidated reports of actual and new failures suggested by the prediction model (qa analysis on failures was reduced by 80%) based on classification & Deep learning..

2- Auto analysis while bug reporting directly to bug management tool.

Solution: Model predicting the defect is already raised in bug management tool, as per the score it would take appropriate action to create new, update, no action, only notification to team etc.

Talk includes below takeaways:

1- Understanding how is AI/ML/Deep learning specifically in software testing.

2- Brainstorming how to leverage AI to help in your tests.

3- Initial steps to start for any model.

4- Tools to leverage


Outline/Structure of the Case Study

  • Understanding AI/ML
  • Popular AI solutions in Test
  • AI Unexplored Areas in Test
  • Case Study - Problem Statement
  • AI Prediction Path
    • Finalizing the expected data which solves Problem Statement
    • Identifying it is AI or Multiple Approach
    • Brainstorming data availability
    • Categorizing under supervised or unsupervised learning
    • Programming Model to learn (Classification, Clustering, Regression, Deep Learning ...)
    • Cleansing data
    • Model preparation and prediction
    • Model feedback & adjustment
    • Model as a service
    • Quantitative analysis of this implementation
  • Overview of other case studies

Learning Outcome

  • Understanding of AI-ML and how it's affecting in testing
  • Sharing popular AI-ML solutions on which current IT in working
  • Knowledge how AI-ML is working in our testing industry.
  • Implementation demo of AI solution for our customized need we needed in our project.

Target Audience

Anyone having basic knowledge of what is testing, and why we do

schedule Submitted 8 months ago

Public Feedback

comment Suggest improvements to the Author
    By LBP LBP  ~  1 month ago
    reply Reply

    Mr. Vaibhav Singhal is an outstanding trainer. 
    He always goes right to the point, with practical perspectives, mastering the topic and plain English explanations.
    It would be just a pleasure to learn from him about "AI Helping Testing": state of the art, his own opinions, ideas and approaches.

  • Pallavi ...
    By Pallavi ...  ~  6 months ago
    reply Reply

    Hi Vaibhav

    can you change the outline/structure of the talk to include the case studies you have mentioned and focus more around it. 

    there is a lot of theory in the talk, or so it seems while looking at it, although needed but can we do it more on talking about how AI/ML help in Test, which your title suggest and focus around it. And showcase tools/technologies - case studies to audience for it. 

    let me know what you think. 

    • Vaibhav Singhal
      By Vaibhav Singhal  ~  6 months ago
      reply Reply

      Hello Pallavi, thanks for taking a deeper look into the proposal contents :)

      "AI/ML help in test" name is given considering the idea is not to showcase how AI/ML can help you or many ways they can help you.

      rather to represent the usecase or problem statement as mentioned above and how we resolved our problem using AI/ML. Since talk would be more focussed on this use case and problem statement so it would be demo solution. 

      AI/ML theory is not in scope for this proposal.




      • Pallavi ...
        By Pallavi ...  ~  6 months ago
        reply Reply

        Understood, so can you accordingly please update the outline structure of the talk with time division to present those use cases as well. 

        And if possible can you make that title a bit more specific something like problem solving , so as to differentiate it with other proposals out there. 

        its just a suggestion.

  • Suhas Bharadwaj
    By Suhas Bharadwaj  ~  7 months ago
    reply Reply

    Good topic and helpful for the current trend in the market.

  • Robin Gupta
    By Robin Gupta  ~  7 months ago
    reply Reply

    I believe that most of the content on the outline focuses on machine/deep learning concepts rather than Selenium or Test Automation. Can you please update the deck or the content here to highlight overlapping sections with the theme of Selenium conference?

    • Vaibhav Singhal
      By Vaibhav Singhal  ~  7 months ago
      reply Reply

      Hello Robin, I agree that the topic is not directly related to Selenium but to test automation it is.

      Please check the use cases mentioned in my listed talk description which are related to test reporting and bug analysis. These two features are usually done manually once we receive the test automation result, we present the solution using machine learning concepts how we automated these problems statement. 

      Often we talk about "AI, machine learning etc" but to use them, we have to go to qa tools having AI features implemented for now. Talk is related to mindset how we can start using machine learning to solve our day to day automation/manual testing challenges, whenever needed or required as per your requirement.




  • Ravi Muragani
    By Ravi Muragani  ~  7 months ago
    reply Reply

    Great topic Vaibhav. Looking forward to this.

  • Sumit kar
    By Sumit kar  ~  8 months ago
    reply Reply

    This is an interesting topic and very much relevant in today's job scenario. Thanks for bringing this up!

  • Haritha Busaareddy
    By Haritha Busaareddy  ~  8 months ago
    reply Reply

    It seems very interesting talk!!  which i'm waiting for this topic

  • Potru Satish  Kumar
    By Potru Satish Kumar  ~  8 months ago
    reply Reply

    very interesting topic! looking forward to know more about this concept.

  • Ankisha Gupta
    By Ankisha Gupta  ~  8 months ago
    reply Reply

    This sounds interesting and looking forward to read more about this

  • Umesh Kumar
    By Umesh Kumar  ~  8 months ago
    reply Reply

    It seems very interesting talk!!  Looking forward for this. 

  • rama sharma
    By rama sharma  ~  8 months ago
    reply Reply
    Vaibhav is a great trainer. So this is great opportunity   to learn from him about AI


    • Vaibhav Singhal
      By Vaibhav Singhal  ~  8 months ago
      reply Reply

      thanks, rama for these encouraging words :)

  • Prabh Preet Singh
    By Prabh Preet Singh  ~  8 months ago
    reply Reply

    I am really looking forward to it.

    • Vaibhav Singhal
      By Vaibhav Singhal  ~  8 months ago
      reply Reply

      thanks Prabh, hope I would meet expectations. 

  • Liked Virender Singh

    Virender Singh - Rise of Shadow DOM - Lets solve it through WebDriver

    45 Mins

    Shadow DOM is kind of web standard that developers use to encapsulate their custom HTML code and style components so that no other global style components can override their code. It ensures that a component will work in any environment, even if other CSS or JavaScript is run on the page i.e. Shadow DOM hides away the complexity of the components from the rest of the page because it renders separately from main DOM.

    Selenium legacy selectors are not able to interact these custom elements under the shadow DOM so we need java script to interact with shadow DOM elements and we also require JavascriptExecutor interface for their execution.

    We will also talk about the WebdriverIO tool v5.5.0, which implicitly support Shadow DOM elements.

  • Liked Vinayaka Mayura G G

    Vinayaka Mayura G G - Search Relevancy Testing: QA in Machine Learning Models

    Vinayaka Mayura G G
    Vinayaka Mayura G G
    schedule 8 months ago
    Sold Out!
    45 Mins

    The adoption of Artificial Intelligence is getting more traction, it is in need to enhance QA capabilities to cope up with these skills. Machine Learning is used extensively in retail applications for solving complex problems, one of them is solving the search relevancy. Showing the appropriate results for the user is important for the conversion rate to go high. As Machine Learning poses different challenges such as a Test Oracle, Fairness, Correctness and Robustness to do QA, We may need to follow different approaches and testing techniques to do the QA for Machine Learning models.

    Different Machine learning types such as Supervised and Unsupervised Models have different characteristics and are used for different types of problems. Though these solves different complex problems, Machine learning Models also a unit of software code that needs to be verified as a normal software system. When a Machine learning model is seen as a whole system, it may look complex and unsolvable. We can group them into small modules and verify for quality. Black box and White box testing techniques can be applied to verify the functionality. Data, Feature Engineering and Algorithms are the major part of the Machine Learning model. We will see how we applied different techniques to validate these.

    This talk is focused on viewing the Machine Learning software as a whole and performing the Quality Analysis for it. We look at how different is testing a machine learning model from typical software testing. We will discuss the challenges that came across, the Process involved in building an ML model. We take an example of Search Relevance for an explanation. We will dive into the areas where quality is assessed. The significant factors considered here are measuring Accuracy and Efficiency. We will look into the different black box testing techniques for different Algorithms. Let us also see how traditional testing is different from testing machine learning applications. I will go through different black-box testing techniques with examples following a live demo.

  • Liked Krishnan Mahadevan

    Krishnan Mahadevan - My experiments with Grid

    45 Mins

    Everyone starts off with a simple grid setup which involves a hub and one or more nodes.

    This traditional setup is a good start but the moment one starts to get serious with the selenium grid and decide to house their own selenium grid for their local executions, that is when issues start.

    My experiences with the Selenium grid in the past couple of years has led me to get introduced some of the most prevalent problems with maintaining an in-house selenium grid.

    • Nodes get unhooked randomly due to network glitches.
    • Nodes introduce false failures due to memory leaks.
    • Selenium Grid running out of capacity.
    • Nodes require OS upgrades/patches etc.
    • Needing to deal with auto upgrades by browsers (especially chrome and firefox)

    Some of these issues I managed to fix by building a "Self Healing" Grid wherein the nodes automatically get restarted after they have serviced "n" tests. But that still didn’t solve many of these other problems.

    That was when I felt, what if there was an on-demand selenium grid.

    What if the Grid could do the following ?

    • The Grid auto scales itself in terms of the nodes based on the current load.
    • The Grid does not require a lot of infrastructure to support it.
    • The Grid can plug itself into some of the cloud providers or leverage a solution such as Docker so that the nodes can be spun and shutdown at will.

    That was how the idea of "Just Ask" an on-demand grid was born.

    Just-Ask is an on-demand grid. It has no nodes attached to it.

    It’s designed to spin off nodes on demand, run test against the newly spun off node and after test runs to completion, clean-up the node as well. The node can be backed by anything. It could be Docker (or) it could be a VM running on any of the popular clouds.

    The session aspires to walk the audience through with my experiments with the selenium grid, my learnings on the selenium grid internals and how I used all of that knowledge to build my own On Demand Selenium Grid. What better avenue to share these learnings than a Selenium Conference.

    The session will introduce the audience to the grid internals and their concepts such as

    • What is a Selenium Remote Proxy ? What is it used for? What can you do with it?
    • What is a Hub (or) Node level Servlet ? When would you need one ?
    • All of this followed by a quick demo on "Just Ask", the on-demand grid that I have built and open sourced here:

  • Liked Gaurav Singh

    Gaurav Singh - How to build an automation framework with selenium : patterns and practices

    Gaurav Singh
    Gaurav Singh
    Lead SDET
    schedule 9 months ago
    Sold Out!
    45 Mins

    With an ever increasing no of businesses being conducted on web the testing need to write automated tests for the app's UI is something that can never be ignored. As you all know Selenium provides an API that enables us to do this quite effectively.

    However, when tasked with setting up the automation framework, there are a lot of questions that arise in the minds of aspiring test developers regardless of what level they are in their career.

    Some of such questions are:

    1. How does one actually go about the business of building a robust and effective automation framework on top of selenium?
    2. What are the elementary building blocks to include in the framework that an aspiring automation developer should know of?
    3. How should we model our tests? XUnit style vs BDD?
    4. Are there good practices, sensible design patterns and abstractions that we can follow in our code?
    5. What are some of the anti patterns/common mistakes we should avoid

    A lot of literature, documentation and blogs exists on these topics on the web already.

    However In this talk,

    I would combine this existing knowledge and my years of experience in building automation frameworks and breakdown these elements and walk you through exactly the sort of decisions/considerations and practices that you can take while starting to implement or improve the UI automation for your team.

    Hope to see you there!

  • Liked Simon Stewart

    Simon Stewart / Jim Evans - Fix a Bug, Become a Committer

    480 Mins

    Have you ever wondered how Selenium works under the covers? Do you get frustrated with locators not locating, pages not loading, or browsers behaving inconsistently from one run to the next? Selenium is an attempt to unify thousands of disparate elements across a wide spectrum of challenges into a single, common interface that works seamlessly with all the major browsers - and yet only a handful of volunteers work to maintain this gigantic effort. If you would like to enhance your own Selenium experience while contributing back to the software that has defined so many of our careers, come to this workshop. In it we'll dissect the different elements of Selenium, dive into its internals, learn how it was built and how to make changes to it, and even write a unit test you can contribute on the same day!

  • 20 Mins

    In this talk, I will speak about the process and strategies on how can one analyze and select the right tools for test automation. We will look at some tools and processes that help to select the tools, frameworks, libraries, etc.

    I will demonstrate some helpful extensions and tools that will be very handy and useful during the process

    This process can be carried out before every test automation project or engagement. The selection of the right tools will help throughout the process of automation and will help to avoid delays, changes, re-work, incomplete automation, etc. and will save a lot of time, effort and resources for the organization.

    The session is designed for anyone in the Software Testing industry and the knowledge received from this session can be implemented right away in any of the upcoming or existing test automation projects.

  • Liked Amit Rawat

    Amit Rawat - Is Puppeteer better than Selenium

    45 Mins

    Puppeteer is a Node js library (developed by Google Chrome team) to control Chrome and Firefox and is getting lot of traction recently because of its amazing capabilities. It has already become so popular that it has got 50K+ stars on Github against Selenium's 15K+ stars.

    In the last Google I/O event, this tool's capability has been showcased and it has been perceived as the next generation Web Test Automation Tool.

    Is Puppeteer better than Selenium? The answer is 'No', and I will cover 'why' in detail during this talk. I will show some live examples to demonstrate that Selenium can also do all those advance things which Puppeteer promises to do.

  • Liked Sanjay Kumar

    Sanjay Kumar - ChroPath- Smarter way of writing automation script!!

    Sanjay Kumar
    Sanjay Kumar
    ChroPath Creator
    schedule 10 months ago
    Sold Out!
    45 Mins
    Case Study
    • Want to save 70-80% manual effort of automation script writing!!
    • Wasting time in verifying xpath one by one?
    • Want to complete automation script without wasting much time?
    • Are you still wasting time in writing english manual test cases?
  • Liked Pallavi ...

    Pallavi ... - Learn Selenium with JavaScript

    480 Mins

    Selenium is a popular open source test automation tool, so popular it has its own conference event which run for 3 whole days across continents and is attended by hundreds of enthusiastic testers, and people wanting to know more about testing and automation. In my little experience of coaching for last 7 years, i feel what english language has done for common man[those from non english speaking countries] in earning bread and butter and giving them an elite status, selenium has done for manual testers.

    Anyone and everyone wants to learn Selenium. The magic of selenium or the flexibility is that it can be learnt with your choice of programming languages. I have taught people Selenium with Java, C#, Python and Ruby largely in past these years. I wish to delve into the world of Javascript now and see what this has to offer to us. So this workshop is for those open minded souls out there who wish to pick up another language and learn, unlearn, relearn selenium with it.

  • Liked Babu Narayanan Manickam

    Babu Narayanan Manickam - Deep Learning Based Selenium Test Failure-triage Classification Systems

    45 Mins

    Problem Statement:

    While running thousands of automated test scripts on every nightly test schedule, we see a mixed test result of pass and failures. The problem begins when there is a heap of failed tests, we caught in the test-automation trap: unable to complete the test-failure triage from a preceding automated test run before the next testable build was released.

    Deep Learning Model:

    The classification was achieved by introducing Machine Learning in the initial months and followed by Deep Learning algorithms into our Selenium, Appium automation tests. Our major classification was based on the failed test cases: Script, Data, Environment, and Application Under Test and that internally had hundreds of sub-classifications.

    To overcome this problem, we started to build and train an AI using Deep Learning, which simulates a human to categorize the test case result suite. Based on the test result failure for each test, the AI model would predict an outcome through API, categorizes and prioritize on the scale of 0 to 1. Based on the prediction model, the algorithm takes appropriate response actions on those test cases that are failed like re-run test again or run for different capabilities. We kick-started this by gathering the historical data set of 1.6 million records, which was collected over a 12 months period, data including the behavior of the test case execution and the resulting suite.

    This Deep Learning-based algorithm has been provided the quality to break down the new defects based on its category, and a classification score is given on a scale of 0-1. We’ve also established a cutoff threshold based on its accuracy of improving, and to group the failed test cases based on its similarity. Classification of the test cases is done in high granularity for sophisticated analysis, and our statistical report states that the classification of the defects has been increased with 87% accuracy over a year. The system has been built based on the feedback adapting models, where for each right classification it will be rewarded and for the wrong, a penalty is given. So whenever receiving a penalty the system will automatically enhance itself for the next execution.

    The algorithm has a powerful model for detecting false-positive test results calculated using the snapshot comparisons, test steps count, script execution time and the log messages. Also, the model has been built with other features like – duplicate failure detection, re-try algorithms and defect logging API, etc.

    The entire classification system has been packaged and deployed in the cloud where it can be used as a REST service. The application has been built with its own reinforcement learning where it uses the classification score to enhance itself and this is programmed to perform in an inconclusive range.

    In sum, this deep learning solution can assist all Selenium testers to classify their test results in no-time and can assist to take next steps automatically and allow team could focus its efforts on new test failures.


  • Liked Srinivasu gangam

    Srinivasu gangam - Zero Touch Automation using NLP (Natural language processing) & AI

    Srinivasu gangam
    Srinivasu gangam
    Sr Manager
    Cotiviti India
    schedule 10 months ago
    Sold Out!
    45 Mins

    Problem Statement:

    As part of SDLC process:

    1. Is your product quality impacted due to a smaller number of QA resources available in the team?
    2. Are you waiting for QA resources to certify your code every time when you deploy? Is this impacting your product lead time (Speed to Market)?
    3. Is your Product delivery timelines are impacted due to last minute defects identified?
    4. Do you have your QA resources only in one location, but you want to “follow-the-sun” approach for Software delivery across multiple locations?
    5. Do you have manual testers who are not skilled in programming, but you want them to execute automated test scripts w/o any training efforts and automation setup?
    6. Would you like your team more agile and cross functional with Delivery?
    7. Would you like to increase your QA team’s productivity while they invest more time in script development rather than script execution?

    If answer is ‘Yes’ for above questions, "Zero touch automation" is the solution for above challenges that we have been facing part of SDLC.

    Solution: Zero touch automation with cutting-edge technologies

    In this session, I will cover how we solved this problem using innovative solutions, Cutting-edge technologies like NLP (Natural language processing), AI & Cloud solutions.

    You will learn how AI, NLP integrated with core automation components to achieve Zero touch automation.

    This solution is not just revolutionary, it is paradigm shift in test automation to get results to your email with detailed analysis of failure categorization with recommended actions to users.

    I will also cover how E2E automation will be driven with decisions taken by machines based on what user is looking for . There is no manual intervention in this process. NLP and AI play key role to help machines to take decisions.

    We will also cover how we empowered developer/release manager/any team member/Manager to trigger the scripts from their cell phone and get the detailed execution report without having any automation software installed in their computer or Phone.

    We will be demonstrating how the request will be initiated from User, understand the need from user using NLP & AI , Fetching the code from bitbucket to select appropriate automation scripts , running them on Selenoid/docker server , storing results to MongoDB , receiving email with test results and Failure analysis.

    What is the value of zero touch automation?

    1. Enable speed to market: Now that Developers does not need to wait for QA resource, Changes can be certified quickly and ready to push to production. Lead time will be significantly reduced.
    2. Increase quality: Now that test automation is easy and it can run multiple times in each environment, most of the defects will be uncovered and addressed before code goes to production.
    3. Ease of test execution: Test execution will be very easy, no automation or framework setup required from user side. Test execution can be done 24*7.
    4. Productivity: Increase QA team’s Productivity to focus more on script development rather than focusing on script execution and failure analysis .
  • Liked Naveen Khunteta

    Naveen Khunteta - Best Practices to implement the test automation framework starting from Design - To -> Infrastructure - To -> Execution.

    45 Mins

    Best Practices - How to get the best 'Return ON Investment' (ROI) from your Test Automation.

    This has been observed that, most of the test frameworks wont be able to survive due to lack of expertise, no maintenance, no best practices being followed, and finally your test automation will be dead after few months, and there is no "Return ON Investment" from this. This is the most common problem, most of the companies are struggling and finally back to square to the Manual testing.

    My proposal : HOW to leverage your test automation in terms of best practices, best ROI, and how to adopt best automation culture in your organisation.
    I strongly propose some of the important points/suggestions to achieve this in your Organisation/Team.
    1. Test Automation Practices:
    • Design Patterns (Web/Mobile/API)
    • What to Automate/Not to Automate
    2. Common Automation Frameworks at Org Level:
    • How to design Generic Utilities, Libraries and different Components, which can be suitable for all the teams in the same Org.
    • Best practices to design your Tests (Automation).
      • Common Design Patterns
      • Common application level and Page libraries
      • Best Practices to use Assertions in your Tests (How and What to write for assertions). Most of the people don't write proper assertions and this is making your test unreliable and no defects found during execution.
    3. Inclusion of API/Backend libraries in your UI test automation as an external Maven/Gradle Dependencies to avoid un-necessary tasks, some of the important points to be considered here:
    • User Creation from APIs (No need to automate user creation from web/app for all the test cases)
    • API tests are stabled most of the time
    • API calls takes lesser time as compared to web, hence include API calls in your UI/App framework to save time.
    • Less flaky test

    4. Best Code Review Process (Do not merge your code into Master without proper Code Review)

    • Implement PR (PULL Request) Process
    • Static Code Analysis using SonarQube, Cobertura, JACOCO etc..
    • Get the benefits of Best Test Automation Quality Matrices
    • Sometimes, Manual (Functional Tester) should review your code (Assertion, test steps and use cases) to get the best coverage
    5. Quality is A Team responsibility:
    • Developers, POs, Manual QEs and Automation engineers should be included to get an overview of test automation coverage.
    6. Maintenance of the Frameworks
    • After couple of months, it should not make your life miserable if you don't maintain your libraries and framework properly.
    • Do not use Hard Coded values, make it simple and Generic.
    7. Infrastructure Setup for Test Design and Test Execution:
    • Proper Browser - OS lab setup
    • Proper Mobile Labs setup with different Devices - IOT, iOS, Android, iPad, Tablets
    • Proper CI - CD common configuration using Jenkins, Dev Ops, AWS, Docker and Cloud setup
    • Handling multiple Docker nodes using Kubernates (use of Selenoid, GRID on Cloud)
  • Liked Vishal Srivastava

    Vishal Srivastava / Khyati Sehgal - Setup Grid Infra with AWS Nodes & Autoscaling of AWS Nodes

    45 Mins

    When it comes to creating a project we always search for ideas that are open-source, easily scalable, ready to use.

    We as humans always have a tendency to grow and expand and when it comes to customizable infrastructure it's a big Yes.

    In this talk, we will be focussing on how we can save the cost of actual servers with easily scalable and manageable open-source software using an open-source tool - selenium grid. How we set up the nodes and hub altogether to make cross-browser, multi-platform execution takes place.

  • Liked Argneshu Gupta

    Argneshu Gupta - Get rid off Xpath in iOS Native App Automation

    Argneshu Gupta
    Argneshu Gupta
    QA Lead
    schedule 8 months ago
    Sold Out!
    45 Mins

    It is always challenging for us while dealing with Xpath and Css in iOS Mobile Automation.Even we can't think to do iOS native app mobile automation without them .

    So problems we need to face are

    • High Maintenance
    • High Execution Time
    • Complex installation
    • Additional Web server

    Do we have some solutions for this problem - The answer is Yes !!! -> KIF(Keep It Functional)