How do you user Selenium to test non-HTML UI components? By integrating Sikuli with Selenium we were able to solve this problem. In this demo, we would like to share our experience in increasing the automation scope to include non-HTML UI components with the help of Sikuli.

Typically learning new tools to increase automation coverage is a challenge for Selenium users. Especially if those screens or components are not in the scope of Selenium locators (e.g. Non-HTML UI, Native Windows, Desktop Apps, SVG components etc.) Integration with Sikuli makes Selenium automation comprehensive. Sikuli complements Selenium capabilities, and integrates with Selenium seamlessly.

We have empowered Selenium to identify non-HTML UI by extending Selenium locators (by.xpath, by.CSS) to identify the components with "by.image."


Outline/Structure of the Demonstration

Challenges in automating non-HTML UI with Selenium. – 5 Mins

How to leverage Sikuli to automate non-HTML UI - 5 Min

Quick demo of integrating Selenium with Sikuli - 10 Mins

Benefits of adding internal support in Selenium to locate components with images. -5 Mins

Q&A - 5 Mins 


Learning Outcome

1. Typical challenges faced by Selenium when identifying non-HTML components.

2. How to leverage Sikuli to increase the scope of Selenium to test non-HTML components.

Target Audience

Automation Tester, Automation Framework developer,

schedule Submitted 6 years ago

Public Feedback