Creating a mobile app is now the new cross platform problem. The major mobile platforms tend to gear their development tool chain towards individuals and their workstations.  But what if you want to introduce a CI solution to this environment? What if your app is launching on more than one platform and there's a team of 20+ developers working on it? What if your tests are more than just Selenium based?

This is normally where you can look to the cloud for scale but mobile has a ton of challenges to do so.  Come and learn from some of the challenges and pitfalls I've encountered while working towards this goal.

 
7 favorite thumb_down thumb_up 1 comment visibility_off  Remove from Watchlist visibility  Add to Watchlist
 

Outline/structure of the Session

 

  • Discuss CI Systems in General
  • General overview of how AWS or Azure could be used (and other Selenium cloud based providers)
  • Pitfalls of scaling iOS build and test
  • Pitfalls of scaling Android build and test
  • Pitfalls of scaling Windows and Windows Phone build and test
  • How to create a reproducible build and test environment for mobile

 

Learning Outcome

The talk will give engineers information to help bootstrap their own mobile CI setups

Target Audience

People who want to scale out a CI system for mobile

schedule Submitted 2 years ago

Comments Subscribe to Comments

comment Comment on this Proposal
  • Dave Haeffner Test
    By Dave Haeffner Test  ~  2 years ago
    reply Reply

    Please provide more information about your approach for creating a reproducble build and test environment for mobile.


  • Liked Anand Bagmar
    keyboard_arrow_down

    Anand Bagmar - To Deploy or Not-to-Deploy - decide using TTA's Trend & Failure Analysis

    Anand Bagmar
    Anand Bagmar
    Director - Quality
    Vuclip Inc.
    schedule 2 years ago
    Sold Out!
    45 mins
    Demonstration
    Intermediate

    The key objectives of organizations is to provide / derive value from the products / services they offer. To achieve this, they need to be able to deliver their offerings in the quickest time possible, and of good quality!

    In order for these organizations to to understand the quality / health of their products at a quick glance, typically a team of people scramble to collate and collect the information manually needed to get a sense of quality about the products they support. All this is done manually.

    So in the fast moving environment, where CI (Continuous Integration) and CD (Continuous Delivery) are now a necessity and not a luxury, how can teams take decisions if the product is ready to be deployed to the next environment or not?

    Test Automation across all layers of the Test Pyramid (be it Selenium-based UI tests, or, xUnit based unit tests, or, Performance Tests, etc.) is one of the first building blocks to ensure the team gets quick feedback into the health of the product-under-test. 

    The next set of questions are:
        •    How can you collate this information in a meaningful fashion to determine - yes, my code is ready to be promoted from one environment to the next?
        •    How can you know if the product is ready to go 'live'?
        •    What is the health of you product portfolio at any point in time?
        •    Can you identify patterns and do quick analysis of the test results to help in root-cause-analysis for issues that have happened over a period of time in making better decisions to better the quality of your product(s)?

    The current set of tools are limited and fail to give the holistic picture of quality and health, across the life-cycle of the products.

    The solution - TTA - Test Trend Analyzer

    TTA is an open source product that becomes the source of information to give you real-time and visual insights into the health of the product portfolio using the Test Automation results, in form of Trends, Comparative Analysis, Failure Analysis and Functional Performance Benchmarking. This allows teams to take decisions on the product deployment to the next level using actual data points, instead of 'gut-feel' based decisions.

  • Anand Bagmar
    Anand Bagmar
    Director - Quality
    Vuclip Inc.
    schedule 2 years ago
    Sold Out!
    45 mins
    Demonstration
    Intermediate

    Typically in organizations, there are multiple projects / products. These products may be of implemented using tech-stacks over many years. Yet - they interact with each other in some way. To manage the complexity around Test Automation, many organizations prefer to have a common Test Automation solution across these products in an effort to build, standardize and maintain the framework.

    However, this is not a good idea! With this approach one potentially ends up having to compromise on the quality of automation that can be done for each product, limited by the toolset.

    The better approach would be to use the tools and technologies that are "right" for each product. This does have other disadvantages, but you would ensure each product is well tested! The only missing piece which remains is that these different products talk with each other. You need to test the integration between them in an automated way to verify all is well.

    "TaaS" is an open-source product solution that allows you do achieve the "correct" way of doing integration testing across a variety of products via Test Automation.

    Example: 

    For one set of products, Selenium-based toolset may be the right choice, where as for legacy reasons, QTP may be used for some other product. With TaaS - you will be able to automate the Integration Testing between these products, by re-using the tests already implemented in the individual product suites.

     

  • moiz
    moiz
    Software Engineer
    Saucelabs
    schedule 2 years ago
    Sold Out!
    45 mins
    Talk
    Beginner

    Appium, often dubbed "Selenium for mobile", at heart its a web server written in NodeJs. Its architecture is modular, which means that it is composed of many small, independently maintained and tested modules. Testing Appium is challenging, but clearly very important, since thousands of users depend on it for their testing. Appium also has all the usual challenges of a large open source project, for example, ensuring consistency of JavaScript code style across hundreds of contributors. It's important to have high-quality and readable code.

     
    I will be discussing approaches to and strategies for testing these kinds of large, modular applications. On the Appium team, we use a combination of unit, functional, and integration tests. Modern services like GitHub, Travis CI, and Sauce Labs make it possible for large open source projects to be tested thoroughly, keeping the code and the app at high quality. I will also discuss the use of tools like JSLint and Gulp, which help prevent code style issues.
     
    Testing the tool which is used for testing is clearly very important. This talk aims to showcase how testing should be approached for large, modular projects which has many collaborators.
  • Liked Rémi
    keyboard_arrow_down

    Rémi - Mobile end to end testing at scale: stable, useful, easy. Pick three.

    Rémi
    Rémi
    Software engineer
    Facebook
    schedule 2 years ago
    Sold Out!
    45 mins
    Talk
    Beginner

    This talk is about how Facebook turned a great idea with a terrible track record into a great tool for thousands of developers.

    The promise of E2E testing — complex, real-world test scenarios from the point of view of and end user — is appealing.
    Many attempts have been made over the years at automating large parts of companies' and developers' testing and release processes, yet most of these efforts ended up in bitter and hard learned lessons about the inherent challenges of the whole approach.

    My work at Facebook over the last two years has been making mobile end to end testing at scale a reality.
    When others said it couldn't be done, or fell by the wayside, we relentlessly pushed forward, solving problems deemed intractable, and finding new, untold vistas of horror before us

    We've come a long way: E2E testing is now an integral part of Facebook's mobile development and release process.
    We'll cover what challenges we faced, and how we chose to solve or make them irrelevant.

  • Liked Jonathan Lipps
    keyboard_arrow_down

    Jonathan Lipps - The Mobile JSON Wire Protocol

    45 mins
    Talk
    Intermediate

    The JSON Wire Protocol (JSONWP) is the version of the WebDriver spec currently implemented by all the Selenium clients. It defines an HTTP API that models the basic objects of web automation---sessions, elements, etc... The JSON Wire Protocol is the magic that powers Selenium's client/server architecture, enables services like Selenium Grid or Sauce Labs to work, and gives you the ability to write your test scripts in any language.

    The JSONWP has served Selenium faithfully for a number of years, but the future of automated testing lies beyond the borders of the web browser. Mobile automation is an essential ingredient in any build, and tools like Appium or Selendroid have made it possible to run tests against mobile apps using the JSONWP. The JSONWP's current incarnation isn't enough to automate all the new behaviors that mobile apps support, however. Complex gestures, multiple device orientations, airplane mode, and the ability to use both native and web contexts, for example, are all essential to mobile automation.

    For this reason the leaders of the Selenium project, in concert with other Selenium-based projects like Appium and Selendroid, met to discuss the future of the JSONWP. We've been working on its next version, called the "Mobile JSON Wire Protocol" (MJSONWP). Appium and Selendroid already implement much of the MJSONWP spec. In this talk I'll dive into the specifics of the MJSONWP extensions, how they relate to the original JSONWP, and how the Selenium clients have begun to implement them.

    Finally, I will talk about the future of the MJSONWP and how it's related to the current and future versions of the WebDriver spec. I'll share how you can get help with the creation of the MJSONWP, and discuss issues with the authors of the new spec before the API is set in stone. We need the help of everyone who's involved in mobile automation to come up with the best and most future-proof version of the MJSONWP. Ultimately, your understanding of how Selenium works will be improved, and you'll have a much better handle on how projects like Appium and Selenium work together to make sure you have the best automation methods available.

  • Liked Justin Woolley
    keyboard_arrow_down

    Justin Woolley / David Anderson / Marvin Ojwang - Automating for the Second Screen with WebdriverJS

    45 mins
    Talk
    Intermediate

    There has been a recent explosion in second-screen technologies such as Chromecast, but designing test automation for second-screen applications is far from straightforward. This new paradigm lacks major automated tool support, and coordinating test execution across multiple devices is tricky and error-prone.

    Our automation solution uses WebdriverJS and WebSockets to perform end-to-end test automation that covers our web player controller and second screen application.

    Learn about our approach to second-screen automation which we’ve used to build a reactive, responsive test suite. We’ll describe our solutions to synchronizing test flow between the controller and target device, validation on the device, targeting different integration components, and device management.

  • Liked James Farrier
    keyboard_arrow_down

    James Farrier / Xiaoxing Hu - Making Your Results Visible - A Test Result Dashboard and Comparison Tool

    45 mins
    Demonstration
    Intermediate

    If a test fails in the woods and no one is there to see it does anyone care, does anyone even notice. What happens when failing tests become the norm and you can't see the wood from the trees? 

     

    After watching last years Allure Report presentation I was inspired.  Selenium tests (and automation tests in general) are often poorly understood by the team as a whole.  Reports/emails go unread with tests failing becoming an expected outcome rather than a glaring red flag.  We looked at what Allure brought to the table and from that base created a dashboard which was designed to:

    • Display the results of test runs in a way that was useful to managers, testers and the rest of the development team.  Including tools to filter out specific test runs and view the overall trend of the test run results.
    • Make debugging tests easier by grouping errors, displaying history of test results, filtering tests and offering visual comparison of test runs.
    • Help mitigate the problems flaky tests cause with test run result reporting (say that three times fast).
    • Help with our mobile device certification process, by easily providing a view to compare test runs across devices.

    Since it's creation the dashboard has been used and praised by managers through to developers.  With our full suite of tests from unit to integration to selenium and appium being stored on the dashboard.  We've managed to:

    • Decrease the time taken to debug test cases.
    • Increase the visibility of all our test suites, with managers having a better idea of how our selenium test suite is progressing and testers better understanding the coverage of unit tests.
    • Focus the organization on quality.

    We are working with legal at present to have this project open sourced and available to all prior to Selenium Conf 2015.

  • Liked Ragavan Ambighananthan
    keyboard_arrow_down

    Ragavan Ambighananthan - Distributed Automation Using Selenium Grid / AWS / Autoscaling

    45 mins
    Talk
    Advanced

    Speed of UI automation has always been an issue when it comes to Continuous Integration / Continuous Delivery. If UI automation suite takes 3 hours to complete, then any commit happens during this time will not be visible in test environment, because the next deployment will happen only after 3 hours. 

    With 2000+ developers and average 250+ checkins per day, the above issues is replicated 250+ times every day. This is not productive and feedback cycle is super slow!

    Another issue is , with 35+ different project teams using 10 or more different jenkins jobs to run their UI automation. So many jobs means (350+), individual teams need to go through the pain of managing their own jenkins job, its a duplicate effort and waste of time. Automation teams need to spend time on writing reliable automation and not managing jenkins jobs.

    Solution is to reduce the UI automation run time from hours to minutes and also use only handful of jobs to run the Distributed Automation!

    Goal: To run all UI automation scenarios within the time take by the longest test case

  • Liked James Eisenhauer
    keyboard_arrow_down

    James Eisenhauer - An Introduction to the World of Node, Javascript & Selenium

    45 mins
    Talk
    Beginner

    Ever wanted to write Selenium code in Node.js?  There seems to be a new javascript library written every hour!  Entering the world of Node.js can be a daunting task.  This session will teach you everything you need to know to make the right decisions when selecting what libraries you should implement on your new Node.js Selenium project and what the possible challenges will be.

     

     

  • Liked Russell Rutledge
    keyboard_arrow_down

    Russell Rutledge - Blazing Fast UI Validation - 5000 Reliable Tests in 10 Minutes on One Machine

    Russell Rutledge
    Russell Rutledge
    Senior Technical Lead
    Nike
    schedule 2 years ago
    Sold Out!
    45 mins
    Talk
    Advanced

    A big blocker for putting a website on truly continuous production delivery is the amount of time it take to validate that the site works correctly.  Tests themselves take time to run, and test results are unreliable to the point where it takes a human to investigate and interpret them.  When counting the time that it takes to both run and interpret results, test runs for an enterprise web site can take an entire day from inception to useful result.

    This session describes common points of failure in test execution that add both latency and unreliability and what can be done to overcome them while still preserving the value of UI validation.  We'll discuss why, after addressing these concerns, UI can be unblocked to reliably field thousands of validation scenarios on a local machine in a matter of minutes. 

  • Liked David Giffin
    keyboard_arrow_down

    David Giffin - A Large-scale, Data-driven Company's Journey of Going From Manual to Automated Testing In 6 Months

    David Giffin
    David Giffin
    Technologist
    TrueCar
    schedule 2 years ago
    Sold Out!
    45 mins
    Talk
    Beginner

    Manual Testing.  Depending on how you've been influenced by those two simple words, reactions vary from slight disgust to full-on depression.  Of course, the solution is clear: automate, but how do you get there when your company is continually pushing out the next big feature?  As the set of features to cover increases, the lack of scalability of manual testing becomes more apparent.

     

    This is a problem that we struggled with at our company.  Automation tactics were explored and implemented, but problems persisted as proposed solutions did not cater to the demands of the manual testers.

     

    After years of failure and disappointment, our latest stint resulted in success.  Not only do we have hundreds of automated tests across various platforms (mobile and web) and products, but manual testing has been eliminated with zero casualties.  As we move forward towards Continuous Delivery and improved automation performance, we wanted to take this moment to look back and share stories of failure and success.

  • Liked Titus Fortner
    keyboard_arrow_down

    Titus Fortner - What Are We Testing, Anyway?

    45 mins
    Talk
    Intermediate

    Testing strategies and the role of DOM to Database Testing in a world of micro services and client side MVCs.

    The trends in software development are making UI testing increasingly difficult. Sites are leveraging more dynamic interactions and moving toward Single Page Applications. Gone are the days when the term “and the page finishes loading” makes any sense. This shift is dramatically increasing the number of flaky tests as well as the costs of such testing relative to the benefits, leaving many organizations wondering if they are worth doing at all. 

    The approach to testing that is “good enough” for any given organization is going to vary by context. In this talk, I’ll cover some different testing options and the advantages and disadvantages to each. We’ll discuss the dangers of mocking and stubbing, the problems with relying on testing journeys, and dealing with bloated test suites that are difficult to maintain.

    Another trend in software development is away from monolithic architectures and toward micro services and service oriented architectures. This approach provides opportunities for decreasing the costs and overhead of UI testing while still maintaining all of the benefits of DOM to Database verification.

  • Liked Justin Ison
    keyboard_arrow_down

    Justin Ison - Android Mobile Device Grid & CI - Getting Started

    Justin Ison
    Justin Ison
    Senior Software Engineer
    Microsoft
    schedule 2 years ago
    Sold Out!
    45 mins
    Talk
    Intermediate

    In the modern era, we have many different cloud testing services to choose from. These cloud services are useful and help reduce the burden of building and maintaining your own Selenium Grid environment. However, there are many scenarios in which you need your tests running locally and quickly, such as you work for the government (or agency), you have sensitive software/data you cannot expose to the cloud, or service costs are too expensive for your organization.

    This presentation will feature getting started with setting up your own mobile device grid, running your tests in parallel, running in CI (Jenkins), and the lessons I have learned along the way.

     

  • 45 mins
    Talk
    Intermediate

    Responsive Website Design have enabled mobile phones and tablets to fundamentally changed how we interact with the internet. Now we have instant access to any website we choose to visit and this causes headaches for testers, especially automated testers.

    This changes how automation, specifically Selenium, is implemented as the test suite needs to be maintainable, which is difficult and will get unruly if not maintained.

    The talk will be specifically about responsive websites however the same techniques can be applied to native app testing.

    Utilizing a test case generator allows for the test conditions(browser, OS and resolution) to exist outside of the test itself allows a single test to be able to test against all testing combinations without having to code for the other options explicitly. With the different options outside of the test the driver is easily instantiated and the browser windows are modified prior to test execution.

    As a sole(or two man team) automated test engineer for 4 years in over 5 projects these are my tools and techniques on how to make your automated test suite not only maintainable but adaptable to any device that you need to test with minimal overhead.

  • Liked Anand Bagmar
    keyboard_arrow_down

    Anand Bagmar - Create your Future!

    Anand Bagmar
    Anand Bagmar
    Director - Quality
    Vuclip Inc.
    schedule 2 years ago
    Sold Out!
    45 mins
    Talk
    Beginner

    With all the focus on Test practices and tools getting better for the present, it is easy to forget about the Future of Testing, especially in terms of Tools and Infrastructure? A question that always comes up in my mind - “Are we so caught up in the past and present, that we will not be as effective in the future?”

    In this talk, we will go on a journey to figure out what new challenges are coming up in the future, and more importantly, what do we need to do next to prepare for it. Also, just preparing for the future is not sufficient. We have an opportunity to stretch beyond our current set of skills, capabilities and boundaries to influence out future! The question is - will we make use of this opportunity? Will Selenium help us take the step in the right direction? Or, will it hold us back?

  • Liked Sarah Thompson
    keyboard_arrow_down

    Sarah Thompson - DevOps meets QA - Using Puppet to set up and manage your Selenium Grid

    Sarah Thompson
    Sarah Thompson
    QE UI Toolsmith
    Puppet Labs
    schedule 2 years ago
    Sold Out!
    45 mins
    Talk
    Intermediate

    For testers, setting up and maintaining a Selenium Grid infrastructure can be timely and costly. A lot of the time, we are asked to do this as part of our day to day job when we really want to focus on testing the product!

    There are some great cloud based alternatives out there that allow you to easily run your tests on a wide range of Operating Systems and against multiple browser types (at a price).

    But what if you already have plenty of devices available within your own company (be it physical machines, virtual machines or cloud based resources) and you want to be able to setup and manage your own grid infrastructure:

    • to allow you greater control over the configuration (perhaps you want to have a headless browser like PhantomJS available on your grid)
    • to run your tests faster (the cloud based alternatives are a good bit slower for obvious reasons)
    • or to simply save money.
  • Liked Satyajit Malugu
    keyboard_arrow_down

    Satyajit Malugu - Adapting testing and automation for Mobile First

    Satyajit Malugu
    Satyajit Malugu
    Senior mobile tester
    Godaddy
    schedule 2 years ago
    Sold Out!
    45 mins
    Talk
    Intermediate

    Mobile-first is a paradigm shift in developing software where products are created with an emphasis on mobile experiences, rather than desktop. Mobile sites, native apps become the new centerfold for the company around which various other initiatives and goals are driven. This paradigm has been in adoption by major industry players including Google, facebook, Airbnb, ebay, shopify etc.

     

    From a development standpoint it is not really different from traditional websites after you get UX but testing is quite different. You have to consider various devices, screen sizes, performance and visual testing. Automation should focus on tackling the most time consuming manual aspects which include functionality, visual parity and device compatibility.

     

    Our $5B company has been in a mobile-first transformation this year, spanning many legacy products and teams. As a senior SDET leading the testing efforts I have been involved in various phases of this transformation. I believe this talk would give practical insights in mobile web development and testing.

  • Liked Manoj Pahuja
    keyboard_arrow_down

    Manoj Pahuja - Measuring the metrics to make them reliable, fast and valuable- What, why and How?

    Manoj Pahuja
    Manoj Pahuja
    Test Architect
    Godaddy
    schedule 2 years ago
    Sold Out!
    45 mins
    Talk
    Intermediate

    UI Tests are fragile by nature. The reliability and execution times are common challenges. If you add scaling UI tests for multiple runs, multiple times a day and environmental instabilities, it becomes very difficult to find the right information related to these tests. Running through and Understanding test results across different builds takes a lot of time. You get too many false alarms and now you have reached a point where nothing is an alarm. Get to know some practical approaches and success stories on how you can distribute all tests across multiple machines, VMs, cloud services and still be able to combine all the results in a easy to read and understandable report. Fix reliability and improve speed of your tests.

  • Liked Dmytro Zharii
    keyboard_arrow_down

    Dmytro Zharii - Smoke Tests and Page Objects: Let's do the right thing fast!

    45 mins
    Talk
    Intermediate

    "Write the Smoke Tests" -- this is the ultimate answer for the questions:
    - How do I start a new test automation project?
    - What is the way to test that my PageObjects are still working correctly after the "nightly" product changes?
    - What is the fastest way for my Test Automation Framework to bring the real value for my teammates and managers?

    Can you imagine you can generate the Page Objects code in a fast and easy way with SWD Page Recorder?
    Can you imagine you can generate the Smoke Tests for your PageObjects and the Application Under Test by pressing one button?

    Let me show how you can increase your productivity in average 45% by using good tooling, patterns and practices during your daily work.

  • Liked Aaron Evans
    keyboard_arrow_down

    Aaron Evans - Looking ahead: testing responsive, mobile, and native apps with Selenium

    Aaron Evans
    Aaron Evans
    Test Consultant
    One Shore
    schedule 2 years ago
    Sold Out!
    45 mins
    Talk
    Intermediate

    Selenium made test automation easier and affordable for many software development teams, but it had many limitations.  It was limited to DOM manipulation in the browser, it depended on explicit waits.  

    Webdriver helped overcome some of these deficiencies and took Selenium to the next level.  Other extensions like Appium have enabled us to use the familiar Selenium API for testing mobile apps.  Proprietary frameworks allow you to integrate Selenium with native extensions and ALM tools.  But a new category of apps is coming with responsive UIs, rich client side Javascript frameworks, touch screens (with pinch/zoom, swipe, rotation, etc) and interact with native device features (such as GPS, accelerometer, local storage) and apps are becoming a collection of interactive services.  

    Is Selenium becoming outdated?  What can we do to keep up with these new interfaces and architectures?

    In this talk, we'll discuss some of the challenges and limitations facing testers using Selenium with this new generation of apps.  We'll cover some of the solutions people are using today, and propose a new way to address these issues and others going forward.