Using Computer Vision to Reduce Test Automation Blind Spots

The standard test automation toolkit easily completes web and mobile automation, but it fails to detect elements on desktop and mobile content-based applications. Computer vision (CV) replicates the human eye using deep learning technology and can determine objects in pictures, which helps machines orient in space and perform repetitive detection tasks. Let's see how a CV can help automate the testing of a much wider software product list.

Anton Angelov will present to you a solution that combines functional tests written on WebDriver W3C protocol with a CV engine based on SikuliX. You will see examples of how his teams managed to create automated tests for verifying complex functionalities such as PDFs, charts, etc. At the end of the presentation, you will know how you can build a similar library in your native programming language to leverage the benefits of the combination between WebDriver and CV.

 
 

Outline/Structure of the Demonstration

1. Will define the problem we had, what we needed to test- give examples. PDFs and charts from our customer websites

2. We will talk about what the computer vision is and give some examples

3. Describe the overall architecture of the solution we designed and see how it can be used to solve our problem

4. Explain what a SikuliX is

5. Demos- real coding how we can create the mentioned design - combining WebDriver/WinAppDriver with SikuliX

6. Give real-world examples

Learning Outcome

understanding what computer vision is and how it can be used to help test automation
practical knowledge about SikuliX and its usages
understand an overall design of implementing a solution for CV with a combination of functional WebDriver tests

Target Audience

automation QAs, software developers in test

Prerequisites for Attendees

basic WebDriver knowledge, intermediate Java, C# or similar OOP language

schedule Submitted 1 week ago

Public Feedback

comment Suggest improvements to the Speaker