How do you user Selenium to test non-HTML UI components? By integrating Sikuli with Selenium we were able to solve this problem. In this demo, we would like to share our experience in increasing the automation scope to include non-HTML UI components with the help of Sikuli.

Typically learning new tools to increase automation coverage is a challenge for Selenium users. Especially if those screens or components are not in the scope of Selenium locators (e.g. Non-HTML UI, Native Windows, Desktop Apps, SVG components etc.) Integration with Sikuli makes Selenium automation comprehensive. Sikuli complements Selenium capabilities, and integrates with Selenium seamlessly.

We have empowered Selenium to identify non-HTML UI by extending Selenium locators (by.xpath, by.CSS) to identify the components with "by.image."


Outline/Structure of the Demonstration

Challenges in automating non-HTML UI with Selenium. – 5 Mins

How to leverage Sikuli to automate non-HTML UI - 5 Min

Quick demo of integrating Selenium with Sikuli - 10 Mins

Benefits of adding internal support in Selenium to locate components with images. -5 Mins

Q&A - 5 Mins 


Learning Outcome

1. Typical challenges faced by Selenium when identifying non-HTML components.

2. How to leverage Sikuli to increase the scope of Selenium to test non-HTML components.

Target Audience

Automation Tester, Automation Framework developer,

schedule Submitted 5 years ago

Public Feedback

comment Suggest improvements to the Speaker
  • sidharth
    By sidharth  ~  5 years ago
    reply Reply

    Looks like an interesting seminar

  • Dave Haeffner
    By Dave Haeffner  ~  5 years ago
    reply Reply

    Awesome topic, and a well written/structured submission.

    I think you can lower the bar from Advanced to Intermediate.

    I'd also like to see a mention of the cons/limitations of this approach -- e.g., Selenium Grid/Cloud platform integration (or at the very least, be prepared to answer questions about it during the Q&A).

    Also, will you be demonstrating against an example application?



    Dave H

  • Pankaj Dhoot
    By Pankaj Dhoot  ~  5 years ago
    reply Reply

    This is a really interesting topic...Must attend ....

  • Madhan
    By Madhan  ~  5 years ago
    reply Reply

    As you are using sikuli the image has to be in focus to perform operation right..

    Will it support if we run testcases using selenium grid..