Building a Mobile Testing Infrastructure: What I Learned

location_city Bengaluru schedule Jun 15th 12:00 - 12:45 PM IST place Grand Ball Room 2 people 98 Interested

Mobile is everywhere, and everyone wants a piece of it. All companies want to have mobile apps for their products, lots developers want to code mobile apps, and lots testers want to build awesome tests and frameworks for mobile apps…

But, what about the infrastructure? Does someone want to spend endless days building a test infrastructure for Android and iOS apps? Not so many people have the time and patience to put all the pieces together and make them work like a Swiss watch. It just seems that many people forget that tests also need a solid infrastructure to be reliable.

In this talk, I will narrate the journey I went through while doing the job that not so many people want, building a mobile testing infrastructure. The attendants will see all the steps taken in the process of planning the system architecture, how all the components were selected, and how they were put together.

More importantly, I will share the lessons learned and the open source components that are being used in the final solution, and also guidelines on how to scope the needed resources when building your own mobile testing infrastructure.


Outline/Structure of the Case Study

Reinventing the Wheel?

  • Does it make sense to build a mobile testing infrastructure from scratch or is there something out there we can reuse?
  • How to build a proof of concept before investing months of work.

Main Challenges

  • Keeping an immutable and stateless mobile infrastructure for iOS and Android
  • Real Devices or Emulators
  • Managing Android and iOS versions, also Chrome and Google Play Services versions in Android
  • Resource monitoring (CPU and memory), how to know when one emulator more can be started.

Components of the Solution

  • Open source and free tools used to create a mobile testing infrastructure
  • Modularising Appium, ChromeDriver, Android tools and friends into different components to simplify upgrading of versions
  • Using software provision management tools, like Puppet or Ansible to keep a clean environment for iOS
  • Retrying? Yes, sometimes things don’t start as expected and we need to cleanup and try again.

Final words

  • Pros and cons of building your own testing infrastructure
  • Having a testing infrastructure is an ongoing task, related activities for maintenance

Learning Outcome

Presentation takeaways:

  • Real Devices vs. Emulators, what to consider when building infrastructure
  • What are the main challenges, and how to overcome them
  • What to consider when scoping a mobile testing infrastructure, how to decide what frameworks and platforms to support
  • Open source resources that can be used to build a mobile testing infrastructure
  • Get an overview of existing tools that can help to tackle the challenge of having a stable testing infrastructure, like software provision management tools

Target Audience

Test Infrastructure and DevOps Engineers, QAs and Automation Engineers in charge of a small-middle sized testing lab.

Prerequisites for Attendees

Some basic Understanding and notions of Docker, general Appium internals, Android and Android Debug Bridge commands would be useful to get more into the presentation.



schedule Submitted 3 years ago

  • Justin Ison
    Justin Ison
    Sr. Software Engineer
    schedule 3 years ago
    Sold Out!
    45 Mins

    In today’s agile world the time to market is becoming increasingly shorter. There is a constant desire to release ASAP to keep ahead of the competition and to please users with updated/new features. Because of this, we have less time to fully do manual and exploratory testing of our apps. Especially, when you consider all the combinations of OS's, Locales, Accessibility, Orientations & Resolutions apps support. Running anywhere from 1 to 100's of Appium crawler bots (covering all of those combinations) at once we can discover more issues quickly and efficiently without having to write a line of code.

    UI Automation also has its limitations as it only tests for expected results. Crawler bots test the unexpected, by collecting metadata such as logs, app strings, screenshots, memory and reporting back it’s finding for review so we can test all these combinations quickly and more efficiently. In this talk, I will go over the current challenges we face in today's development world, why we need more tools to help us keep pace, and cover how you can build your own Appium crawler.

    I've open sourced this tool and is available here for everyone to use:

  • Wim Selles

    Wim Selles - Building your own native app visual testing module with Appium

    Wim Selles
    Wim Selles
    Sr. Solutions Architect
    Sauce Labs
    schedule 3 years ago
    Sold Out!
    45 Mins
    Case Study

    We all know that Appium can do (almost) everything what you tell it to do, but in the end we always ask it to do the same thing. We ask Appium to automate our happy and or our error flow. The result can give us the confirmation that the user can still buy a product, or verify if his bank account still has a positive balance.

    But what if we need to know more than that - what if we also need to verify the layout of our app? The answer would be easy, as there are plenty of open-source or paid solutions that can help you with that. However, almost all of these solutions focus on web/hybrid apps, not on native iOS and or Android apps.

    I faced this challenge when I was automating a React Native app for my customer. There was no image comparison tool that could do what I wanted it to do, so I rolled up my sleeves and started building my own. In this talk we’ll be walking down the path I’ve been taking and I’ll explain:

    • Why I used ResembleJS as my core visual comparison solution
    • The things I’ve learned about Appium in the process (pro tips!)
    • The differences between iOS and Android and how I managed to solve this in one cross-platform solution
    • The pros and cons of my solution
  • 45 Mins
    Case Study

    Experimentation and quick feedback is the key to success of any product, while of course ensuring a good quality product with new and better features is being shipped out at a decent / regular frequency to the users.

    In this session, we will discuss how to enable experimentation, get quick feedback and reduce risk for the product by using a case study of a media / entertainment domain product, used by millions of users across 10+ countries - i.e. - we will discuss Testing Strategy and the Release process an Android & iOS Native app - that will help enable CI & CD.

    To understand these techniques, we will quickly recap the challenges and quirks of testing Native Apps and how that is different than Web / Mobile Web Apps.

    The majority of the discussion will focus on different techniques / practices related to Testing & Releases that can be established to achieve our goals, some of which are listed below:

    • Functional Automation approach - identify and automate user scenarios, across supported regions
    • Testing approach - what to test, when to test, how to test!
    • Manual Sanity before release - and why it was important!
    • Staged roll-outs via Google’s Play Store and Apple’s App Store
    • Extensive monitoring of the release as users come on board, and comparing the key metrics (ex: consumer engagement) with prior releases
    • Understanding Consumer Sentiments (Google’s Play Store / Apple’s App Store review comments, Social Media scans, Issues reported to / by Support, etc.)

  • Ajay Balamurugadas

    Ajay Balamurugadas - Appium or No Appium? Tales of Test Automation Failure

    45 Mins

    Automation is everywhere and mobile app automation using Appium is a boon for many. This talk is about the multiple times' people have burned their fingers trying to automate without proper planning beforehand. Instead of diving deep into writing automated checks, it is a good idea to crosscheck against few guidelines. A realistic plan would help everyone and save cost, increase value and reduce the risk. A simple mistake multiple testers do is to pick a tool first and then figure out what to automate. The reason to automate could be driven by management, development teams or peer pressure. When we think of automation, it actually amplifies, enables, extends and maximizes our intended tasks.

    If we choose the right tasks to automate, we would save time, money and effort. At the same time, give the tool garbage and we get more garbage quicker at a higher cost. This is simple to understand, yet a majority of teams fail to understand the basic philosophy behind what and when to automate.

    Let us look at common tales of test automation failure and how teams can recover from them. We will discuss a few tough questions related to adding automation to test plan and how to derive value from automation efforts.

  • Manjunath Halakoppa

    Manjunath Halakoppa - A rollercoaster ride with Appium

    45 Mins

    We love Appium and we realized it a bit late. We did not start with Appium and the ride was a bit bumpy to our dislike. Once we picked Appium, it was not a straightforward journey either. New challenges popped up and we started loving how we solved them with Appium. This talk highlights our journey from MonkeyTalk to using Appium on Windows for Android apps and slowly moving on to Appium on Mac. Few interesting stories along the way like the migration challenge or the lack of community support or the last minute surprises with the autogenerated scripts will be shared with the audience. With our realistic and relatable stories, join us in this talk to know how we conquered our fear of Appium and finally use it to its fullest potential bringing smiles on our clients’ faces.

    The rollercoaster ride also includes our discovery of Appium’s power and how we moved away from emulators once for all. If you have just started your journey with Appium, this talk is a good preview of some of the challenges you might face and get inspired by how we solved them for everyone’s satisfaction.