From Trigger to Report - The End to End Test Pipeline at trivago

Learn how the end to end tests for trivago's hotel search website are triggered, how we run them and what we do in terms of reporting and visibility of tests results. This is mainly about the technological aspects and challenges but will also touch on how and why we established certain processes.

This talk will touch on a range of topics like

  • How the test automation team works at trivago
  • What challenges we faced on our long way towards automated end to end testing
  • How our complete test pipeline works
  • Why we decided against some available frameworks and libraries

Outline/Structure of the Case Study

The basic outline is work in progress but will likely follow these topics:

  • Technology
    • Job triggers
    • Jenkins jobs/pipelines
    • Test data preparation with Cucable
      • Why an own solution?
  • Test runs
    • Maven Failsafe
    • Docker container
    • Rerun functionality
    • Test Framework (Cucumber 2.3.1, Selenium 3.10.0)
      • Why an own solution?
    • Selenium Grid
    • Reporting
      • Report generation with Cluecumber
        • Why an own solution?
      • Additional reporting and dashboards
    • Processes / QA collaboration

The attached slides are from my last talk "The Bumpy Road Towards Continuous Delivery" from SeleniumConf Berlin 2017 which was much less technical than what I want to present now.

Learning Outcome

Participants will have an idea about the challenges of setting up a complete and stable test environment. The presented solutions are not necessarily the ideal ones for every use case but may trigger ideas on how to solve certain problems like

  • running tests continuously
  • user friendly reporting
  • dealing with flaky tests
  • running inhouse vs. on an external Selenium grid

Target Audience

QA engineers, Developers, POs

Prerequisites for Attendees

Participants should know the benefits of test automation and when it should be used.

schedule Submitted 1 year ago

Public Feedback

comment Suggest improvements to the Speaker
  • Anand Bagmar
    By Anand Bagmar  ~  1 year ago
    reply Reply

    Hi Benjamin,

    Will you be talking about the challenges and why certain solutions came up for you? Can you share some of the challenges here as an example?

    Also, how will you ensure the learning outcomes are not specific to Trivago, but attendees can actually learn and apply them (appropriately) in their own context as well?

    • Benjamin Bischoff
      By Benjamin Bischoff  ~  1 year ago
      reply Reply

      Hi Anand,

      you are exactly right. In this talk I want to follow our pipeline from start to end, mention our challenges in every step and how we overcame them. So the structure of the talk should be pretty straight forward.

      Some challenges were:

      • Test runtimes of almost 90 minutes that were not acceptable 
      • Test flakiness due to environment issues like a temporarily dropped connection to our Selenium provider
      • Reporting that spread over different sources and was hard to follow and analyze by our QA

      The learning outcomes are not specific to trivago since I will present solutions that are universally usable, e.g. how to rerun Cucumber tests in case of specific failures, parallel execution of tests and better reporting. So either these things can be directly used (like the mentioned open source plugins) or be easily adaptable to other user contexts as well.