Babu Narayanan Manickam
Member since 3 years
Babu Narayanan Manickam
Specialises In (based on submitted proposals)
A Management Professional with over 16 years of experience with expertise in IT Project Management, Application Development, Onsite IT QA Support, IT Services Delivery, People Management and Quality Management achieving revenue objectives
Reinforced Selenium test selection using Recommender SystemsBabu Narayanan ManickamDirectorTestLeaf SoftwareSarath Muthuswamy-
schedule 1 month agoSold Out!
One of our enterprise oracle customers has more than 30,000+ UI & API regression tests for end to end Apps Integration testing. The average time it takes to run these tests (depends on parallelization) is anywhere between 11-12 hours.
With considerable time to complete all regression tests, the objective was to find a working AI model to identify the probable failing test scripts and prioritize them early along with the most critical business automated tests.
Selecting the most promising selenium test scripts to detect application defects may be harder considering the uncertainties on the impact of committed code changes and breakage of many traceability links between code and automated tests. The design was to automatically choose the test scripts and prioritize in CI tool with the goal to minimize round-trip-times between code commits and developer feedback on failed test scripts.
In our devops environment, where new test scripts are created and obsolete tests are deleted constantly, the reinforced method learns to prioritize error-prone test scripts higher under the guidance of a reward function and by observing previous test cycles from historical data. By applying reinforced learning techniques on data extracted, it is evident that reinforcement learning enables better automatic adaptive test script selection and prioritization in CI and automated regression testing.
As the first step, we created a predictive model that estimates the probability of each test failing for a newly proposed build. Instead of defining the model manually, we built it by using a large data set containing results of tests on historical builds/releases and then applying the standard machine learning technique - gradient-boosted decision tree.
- Code changes based on build metadata
- Code Owner Information
- Historical test runs
With this model, we can analyze a particular code change to find all potentially impacted tests that transitively depend on modified files, and then estimate the probability of that test detecting a regression introduced by the change. Based on those estimates, the system selects the tests that are most likely to fail for a particular change. The diagram below shows which tests (shown in blue) would be chosen for a change affecting the two files from the previous example, where the likelihood of each considered is represented by a number between zero and one.
In sum, Our recommender system with reinforcement learning algorithms helps to identify the priority test scripts in runtime, the model helps to move up or down the test scripts from the sequences set at the beginning.
Deep Learning Based Selenium Test Failure-triage Classification SystemsBabu Narayanan ManickamDirectorTestLeaf Software
schedule 2 months agoSold Out!
While running thousands of automated test scripts on every nightly test schedule, we see a mixed test result of pass and failures. The problem begins when there is a heap of failed tests, we caught in the test-automation trap: unable to complete the test-failure triage from a preceding automated test run before the next testable build was released.
Deep Learning Model:
The classification was achieved by introducing Machine Learning in the initial months and followed by Deep Learning algorithms into our Selenium, Appium automation tests. Our major classification was based on the failed test cases: Script, Data, Environment, and Application Under Test and that internally had hundreds of sub-classifications.
To overcome this problem, we started to build and train an AI using Deep Learning, which simulates a human to categorize the test case result suite. Based on the test result failure for each test, the AI model would predict an outcome through API, categorizes and prioritize on the scale of 0 to 1. Based on the prediction model, the algorithm takes appropriate response actions on those test cases that are failed like re-run test again or run for different capabilities. We kick-started this by gathering the historical data set of 1.6 million records, which was collected over a 12 months period, data including the behavior of the test case execution and the resulting suite.
This Deep Learning-based algorithm has been provided the quality to break down the new defects based on its category, and a classification score is given on a scale of 0-1. We’ve also established a cutoff threshold based on its accuracy of improving, and to group the failed test cases based on its similarity. Classification of the test cases is done in high granularity for sophisticated analysis, and our statistical report states that the classification of the defects has been increased with 87% accuracy over a year. The system has been built based on the feedback adapting models, where for each right classification it will be rewarded and for the wrong, a penalty is given. So whenever receiving a penalty the system will automatically enhance itself for the next execution.
The algorithm has a powerful model for detecting false-positive test results calculated using the snapshot comparisons, test steps count, script execution time and the log messages. Also, the model has been built with other features like – duplicate failure detection, re-try algorithms and defect logging API, etc.
The entire classification system has been packaged and deployed in the cloud where it can be used as a REST service. The application has been built with its own reinforcement learning where it uses the classification score to enhance itself and this is programmed to perform in an inconclusive range.
In sum, this deep learning solution can assist all Selenium testers to classify their test results in no-time and can assist to take next steps automatically and allow team could focus its efforts on new test failures.
No more submissions exist.
No more submissions exist.