Diagnostic techniques for Selenium tests

One of the main benefits of end-to-end tests with Selenium is the simulation of real user scenarios in an automated fashion. Nevertheless, end-to-end tests are often criticized because these kinds of tests do not isolate failures, and therefore troubleshooting failures can be difficult and costly for testers. This talk pretenses different techniques aimed to solve this problem. The objective is to provide useful assets to find the root cause of a failure in a Selenium test. The first one is screenshotting. Once a failure is detected, the state of the browser just before the failure happens can provide the clue to determine the nature of the problem. If the screenshot is not enough, the next step is to watch the browser during the test to observe the evolution of the web under test. This process is straightforward when running a test with the local browser, but it can be tricky in the usual case of running a test in a headless environment, such as Jenkins or Travis to name a few. The solution proposed is twofold. On the one hand, a Selenium session can be recorded using Virtual Frame Buffer (xvfb) and FFmpeg. On the other hand, we can use web browsers and mobile devices in Docker containers, in which the GUI session can be easily recorded using VNC and FFmpeg. The last mechanism proposed is the access to browser logs to debug the JavaScript traces of our applications. Unfortunately, the Selenium API only allows us to do it when Chrome is used as a browser. The solution proposed is to use monkey patching in the console object at JavaScript level in a cross-platform browser extension which is used to instrumentalize the browser (e.g., Firefox, Opera) controlled with Selenium WebDriver. All these mechanisms have been implemented in Selenium-Jupiter, a JUnit 5 extension for Selenium providing seamless integration with Docker. To ease the access to such information, Selenium-Jupiter provides seamless integration with Jenkins and Slack, publishing the resulting files (PNG screenshots, MP4 recordings, and TXT logs) in the Jenkins GUI or Slack channel when a test fails.

 
 

Outline/Structure of the Talk

  1. End-to-end tests: pros and cons
  2. Diagnostic techniques in Selenium
    1. Screenshotting
    2. Session recording
    3. Log gathering
  3. Agnostic implementation
    1. Bash script for session recording
    2. Xvfb session recording
    3. Cross-platform log gathering
  4. Framework-based implementation
    1. Screenshotting in different binding languages
    2. Java implementation: Selenium-Jupiter
      1. Integration with Jenkins
      2. Integration with Slack

Learning Outcome

  • Discover several mechanisms which allow to find the underlying cause of a failure Selenium test
  • Learn agnostic and framework-based solutions for screenshoting, session recording, and log gathering
  • Discover Selenium-Jupiter, the JUnit 5 extension for Selenium and Docker

Target Audience

Developers/testers/CTOs

schedule Submitted 1 year ago

Public Feedback