Challenges in Testing & Automating Games

Testing real-time, multi-user games built for native apps and / or browser-based on phones / mobile devices / tablets / desktop browsers makes testing of regular products as apps, or websites appear like a piece of cake. Testing such real-time and multi-user games becomes even more challenging when you think about automating the same.

I got an opportunity to build Functional Test Automation for a suite of games – and what an exciting time it turned out to be!

These games are built either using Cocos2d-x, or Unity (cross platform game engines for mobile games, apps and other cross platform interactive GUI and are known for their speed, stability, and ease-of-use).

The key challenges I encountered here were:

  • Millions of users, playing games on a huge variety of devices (Android & iOS native apps, Mobile-Web, and Desktop Web)
  • Limited unit testing
  • API testing & Functional Testing done in isolation (mini-waterfall approach)
  • Usage of Cocos2d-x & Unity for game rendering – which cannot be automated via Appium
  • Limited Functional Automation (for native apps)

An approach to Functional Automation of Real-Time, Multi-User Game scenarios

I overcame the above mentioned challenges by doing the following:

  • A better way of working (you can call it ‘Agile’)
  • Break down the walls by fostering a mindset of “build quality-in, as opposed to test for quality”
  • Built a new functional automation framework using java / testing / appium-test-distribution / reportportal / jenkins with focus on – specify test intent once and run on all supported channels (ex: Android, iOS, Mobile-Web, and Web)
  • Built a solution for Cocos2d-x layer automation
  • Created a vision of CI-CD for the organization, and setup code-based CI pipelines to enable end-2-end visibility
  • Made the framework extensible by providing ability to use same framework for multiple games

The focus of this talk will be to:

  • Share an example of a particular use case
  • Share the solutions, including code snippets, implemented for:
    • Functional Test Automation Framework Architecture & Design
    • Ease of Test Implementation, while maintaining code quality and promoting reuse
    • Test Execution on local Vs CI, on-demand as well as on every new build


Outline/Structure of the Case Study

  • Challenges of testing real-time, multi-user games (3 min)
  • Testing and Test Automation Strategy to give early feedback (3 min)
  • Functional Test Automation Framework design, architecture including bots, game engine interaction (20 min)
  • Specify test once, implement and run on multiple channels (5 min)
  • Test Execution & Reporting (4 min)
  • Overcoming other challenges (3 min)
  • Next steps (2 min)
  • Q&A (5 min

Learning Outcome

  • A case study of the automation strategy for real-time, multi-user games
  • Test Automation Framework design & architecture to support multiple products, and multiple channels per product
  • Solution for automating Cocos2d-x layer
  • CI-CD setup, with multi-branch pipeline
  • Upskilling team members on this approach (coding practices, PRs, design patterns, etc.)

Target Audience

Test Automation Engineers of all levels, Developers

Prerequisites for Attendees

Basic understanding of multi-player games

If any experience in testing games, then even better



schedule Submitted 1 year ago

  • Anand Bagmar

    Anand Bagmar / Diego Molina / Jim Evans / Manoj Kumar / Simon Stewart / Sri Harsha - Q&A with the Selenium Committee

  • Rajdeep varma

    Rajdeep varma - The Joy Of Green Builds - Running Tests Smartly

    Rajdeep varma
    Rajdeep varma
    Automation Lead
    schedule 1 year ago
    Sold Out!
    45 Mins

    So you have got a few UI tests and they are running in parallel, great! However, life will not be so sweet once these 'a few' turns into 'a lot'. We grew from a few to 1500 UI tests (although not particularly proud of this number, there are situations and reasons)

    We started with a simple parallel distribution of tests 3 years ago. As test count increased failure count run time increased along with increased flaky tests. Mobile tests had their own challenges (eg. device dropping-off, random wi-fi issues, etc) To keep up with this, we created a queue and workers based solution which could distribute the tests more efficiently ( Over time, we made more improvements, in particular:

    • Segregation of failures based on infrastructure issues and re-queue the tests
    • If a device/emulator malfunction, rescue the tests to another device
    • Repeating a single test on 100s of the worker in parallel to detect flakiness
    • Repeat a test if a known network issue
    • Terminating the build early if more than a certain number of tests have failed
    • Health check of each device, before each test to ensure reliability
    • Muting a test if failure is known, and highlight outdated mutes if the related task is fixed

    In this talk, I will talk about the initial challenges with running UI tests in parallel (Selenium and Appium), how we approached the queue based solution and continuous improvement of this solution; finally, how attendees can use it at their workplace or create their own solution based on our learnings.

  • Rajdeep varma

    Rajdeep varma - Appium-Android: From Surface To Core

    Rajdeep varma
    Rajdeep varma
    Automation Lead
    schedule 1 year ago
    Sold Out!
    90 Mins

    Over the past few years, Appium has gained tremendous popularity due to its support of many languages and multi-platform automation possibilities. It is the most popular tool for test automation engineers as of today.
    However, there are times when beginner Appium users have a curiosity about how it works under the surface. Some intermediate/advanced users always wonder how to debug their automation code in conjunction with the Appium source code to find out why that element on UI can’t be located or why the click is not working on that custom element.
    In this workshop, we will start with the tools that are building a block of higher-level android automation frameworks such as espresso, uiautomator2, etc. Later, we will deep dive into Appium with espressodriver.

  • Ivan Krutov

    Ivan Krutov - Workshop: Efficient Selenium Infrastructure with Selenoid

    Ivan Krutov
    Ivan Krutov
    schedule 1 year ago
    Sold Out!
    480 Mins

    Selenoid is an alternative lightning fast open-source Selenium protocol implementation running browsers and Android emulators inside Docker containers. It is distributed with a set of ready-to-use Docker images corresponding to the majority of popular browser versions, has a one-command installation utility and works slightly more efficiently than traditional Selenium Grid.

    This tutorial shows how to build efficient and scalable browser automation infrastructure using Selenoid and related tools. You will be taught why running browsers in containers is so efficient, how to easily install Selenoid and use its powerful features.