So you have got a few UI tests and they are running in parallel, great! However, life will not be so sweet once these 'a few' turns into 'a lot'. We grew from a few to 1500 UI tests (although not particularly proud of this number, there are situations and reasons)

We started with a simple parallel distribution of tests 3 years ago. As test count increased failure count run time increased along with increased flaky tests. Mobile tests had their own challenges (eg. device dropping-off, random wi-fi issues, etc) To keep up with this, we created a queue and workers based solution which could distribute the tests more efficiently (https://github.com/badoo/parallel_cucumber). Over time, we made more improvements, in particular:

• Segregation of failures based on infrastructure issues and re-queue the tests
• If a device/emulator malfunction, rescue the tests to another device
• Repeating a single test on 100s of the worker in parallel to detect flakiness
• Repeat a test if a known network issue
• Terminating the build early if more than a certain number of tests have failed
• Health check of each device, before each test to ensure reliability
• Muting a test if failure is known, and highlight outdated mutes if the related task is fixed

In this talk, I will talk about the initial challenges with running UI tests in parallel (Selenium and Appium), how we approached the queue based solution and continuous improvement of this solution; finally, how attendees can use it at their workplace or create their own solution based on our learnings.

Challenges in Testing & Automating Games

Testing real-time, multi-user games built for native apps and / or browser-based on phones / mobile devices / tablets / desktop browsers makes testing of regular products as apps, or websites appear like a piece of cake. Testing such real-time and multi-user games becomes even more challenging when you think about automating the same.

I got an opportunity to build Functional Test Automation for a suite of games – and what an exciting time it turned out to be!

These games are built either using Cocos2d-x, or Unity (cross platform game engines for mobile games, apps and other cross platform interactive GUI and are known for their speed, stability, and ease-of-use).

The key challenges I encountered here were:

• Millions of users, playing games on a huge variety of devices (Android & iOS native apps, Mobile-Web, and Desktop Web)
• Limited unit testing
• API testing & Functional Testing done in isolation (mini-waterfall approach)
• Usage of Cocos2d-x & Unity for game rendering – which cannot be automated via Appium
• Limited Functional Automation (for native apps)

An approach to Functional Automation of Real-Time, Multi-User Game scenarios

I overcame the above mentioned challenges by doing the following:

• A better way of working (you can call it ‘Agile’)
• Break down the walls by fostering a mindset of “build quality-in, as opposed to test for quality”
• Built a new functional automation framework using java / testing / appium-test-distribution / reportportal / jenkins with focus on – specify test intent once and run on all supported channels (ex: Android, iOS, Mobile-Web, and Web)
• Built a solution for Cocos2d-x layer automation
• Created a vision of CI-CD for the organization, and setup code-based CI pipelines to enable end-2-end visibility
• Made the framework extensible by providing ability to use same framework for multiple games

The focus of this talk will be to:

• Share an example of a particular use case

• Share the solutions, including code snippets, implemented for:
  • Functional Test Automation Framework Architecture & Design
  • Ease of Test Implementation, while maintaining code quality and promoting reuse
  • Test Execution on local Vs CI, on-demand as well as on every new build

Selenoid is an alternative lightning fast open-source Selenium protocol implementation running browsers and Android emulators inside Docker containers. It is distributed with a set of ready-to-use Docker images corresponding to the majority of popular browser versions, has a one-command installation utility and works slightly more efficiently than traditional Selenium Grid.

This tutorial shows how to build efficient and scalable browser automation infrastructure using Selenoid and related tools. You will be taught why running browsers in containers is so efficient, how to easily install Selenoid and use its powerful features.