1. Are you Confident about your test coverage?
2. Do you have millions of permutations and combinations to be covered in testing?
3. is automation the solution to cover million combinations?
4. why can not my system decide what test cases to be executed to give complete test coverage?
5. As part of CI/CD , instead of executing same test cases irrespective changes going in, would you like to execute more appropriate test cases ?

I will be presenting quick demo on statistical approach and strategy which can help us to generate appropriate number of test cases automatically based on the impacted areas and business metrics . Let me share an example below.

Let’s say we are testing any eCommerce application Flipcart or amazon. We will have to cover multiple combinations like different categories of items, different payment types, shipping types, Promotions, user types etc.. . I will use little bit of sets theory here.

Let us take below 5 sets as an example. Assume that there is new payment type is getting introduced as “Paypal” as enhancement. If we need to make sure this new payment type is working with all products, all shipments types , all user types etc… we will have to cover all permutations and combinations which is nothing but Cartesian product of all these sets. The Cartesian product of below 5 sets is - 6*3*2*4*3 =432 test cases.

Do we really need to execute 432 test cases just for one new payment type ? we just need to execute 24 test case to give confident test coverage .This can be done with OATS tool which is developed with intelligence with Business user Metrics + Orthogonal array testing approach ( OATS) approach . OATS tool will generate required number of test cases with simple input and one click .

There are 2 key areas of input that we consider before we generate optimized tests.

1. Business user metrics - Example : if you look at the below payment set , we would like to know what % of customers using each payment type in overall successful transactions. let us assume 60% Credit card payments , 20 % paytm and 20% rest of payment types . These metrics will help tool to give more weightage to Credit card test cases over other payment types. This make sense because we need to focus more on the areas our end users are using.

2. Impacted areas: let us take an example that there is new payment type (AmazonPay) is getting introduced as en enhancement. We should give more weightage to payment set and make sure all existing payment types are tested with different combinations.

Set1 - PAYMENT = { Cash, Credit card , Paytm , AmazonPay, Debit card}- # elements - 6

Set2 – SHIPPING – {Shiptoday, Prime Shipping, ship Tomorrow}-# elements - 3

Set3 – USER TYPE - { Guest User , Logged in user }-# elements - 2

Set4 – PRODUCT TYPE – {Electronics, Groceries, Books , Stationary}- # elements - 4

Set5 – PROMO TYPE – { Promo1 , Promo2 , Promo3}-# elements – 3

I will be covering how this can be done in detail during presentation along with Tool Demo which was built in with an intelligence using sets theory . you will get an answer on why only 24 test cases are good enough instead of 432 test cases for the change new payment type got introduced.

Advantages :

1. Optimized coverage with less test cases

2. Tester will be confident about test coverage

3. This tool output can be integrated with Selenium key word driven framework . All this happens during run time w/o manual intervention.

4. This strategy is good fit for automation testing .

5. if any combinations are invalid to test, those combinations will be excluded while generating tests.

6. There are tools in the market to provide statistics about untested code (Ex: JaCoCO ). These statistics can be used as input for OATS tool. We have not done this yet.

Limitations:

1. We have seen great value (reduced bug escape) applying this strategy to automated tests. However there is limitation in applying the same strategy to manual exploratory , Adhoc testing.

2. We have not tried this strategy for performance testing yet.

3. There are corner scenarios where human can only think will not be covered by this tool . These scenarios to be appended to generated test cases .

Problem Statement:

As part of SDLC process:

1. Is your product quality impacted due to a smaller number of QA resources available in the team?
2. Are you waiting for QA resources to certify your code every time when you deploy? Is this impacting your product lead time (Speed to Market)?
3. Is your Product delivery timelines are impacted due to last minute defects identified?
4. Do you have your QA resources only in one location, but you want to “follow-the-sun” approach for Software delivery across multiple locations?
5. Do you have manual testers who are not skilled in programming, but you want them to execute automated test scripts w/o any training efforts and automation setup?
6. Would you like your team more agile and cross functional with Delivery?
7. Would you like to increase your QA team’s productivity while they invest more time in script development rather than script execution?

If answer is ‘Yes’ for above questions, "Zero touch automation" is the solution for above challenges that we have been facing part of SDLC.

Solution: Zero touch automation with cutting-edge technologies

In this session, I will cover how we solved this problem using innovative solutions, Cutting-edge technologies like NLP (Natural language processing), AI & Cloud solutions.

You will learn how AI, NLP integrated with core automation components to achieve Zero touch automation.

This solution is not just revolutionary, it is paradigm shift in test automation to get results to your email with detailed analysis of failure categorization with recommended actions to users.

I will also cover how E2E automation will be driven with decisions taken by machines based on what user is looking for . There is no manual intervention in this process. NLP and AI play key role to help machines to take decisions.

We will also cover how we empowered developer/release manager/any team member/Manager to trigger the scripts from their cell phone and get the detailed execution report without having any automation software installed in their computer or Phone.

We will be demonstrating how the request will be initiated from User, understand the need from user using NLP & AI , Fetching the code from bitbucket to select appropriate automation scripts , running them on Selenoid/docker server , storing results to MongoDB , receiving email with test results and Failure analysis.

What is the value of zero touch automation?

1. Enable speed to market: Now that Developers does not need to wait for QA resource, Changes can be certified quickly and ready to push to production. Lead time will be significantly reduced.
2. Increase quality: Now that test automation is easy and it can run multiple times in each environment, most of the defects will be uncovered and addressed before code goes to production.
3. Ease of test execution: Test execution will be very easy, no automation or framework setup required from user side. Test execution can be done 24*7.
4. Productivity: Increase QA team’s Productivity to focus more on script development rather than focusing on script execution and failure analysis .