Measuring the metrics to make them reliable, fast and valuable- What, why and How?

UI Tests are fragile by nature. The reliability and execution times are common challenges. If you add scaling UI tests for multiple runs, multiple times a day and environmental instabilities, it becomes very difficult to find the right information related to these tests. Running through and Understanding test results across different builds takes a lot of time. You get too many false alarms and now you have reached a point where nothing is an alarm. Get to know some practical approaches and success stories on how you can distribute all tests across multiple machines, VMs, cloud services and still be able to combine all the results in a easy to read and understandable report. Fix reliability and improve speed of your tests.

 
12 favorite thumb_down thumb_up 3 comments visibility_off  Remove from Watchlist visibility  Add to Watchlist
 

Outline/structure of the Session

  • Understand problems related to reliability and executions times.
  • Metrics that can be used to measure reliability and speed.
  • Terms and Technologies (Grid, Elastic Search, Log stash, Kibana, RSpec, Mocha, NodeUnit etc.)
  • Tools and measures that can be used to better debug tests, manage and improve reliability and speed automatically.
  • Alerting that calls for action rather than yet another notification.

Learning Outcome

Measure the right metrics to have an effective, maintainable and measurable value of automation.

Target Audience

Managers, Framework Designers, Testers,Automation engineers

schedule Submitted 2 years ago

Comments Subscribe to Comments

comment Comment on this Proposal
  • Ru Vu
    By Ru Vu  ~  2 years ago
    reply Reply

    Getting some information on making tests more stable would be useful.

    One of the challenges for us has been getting the tests fixed and then who to alert and how. 

  • Adam Knapp
    By Adam Knapp  ~  2 years ago
    reply Reply
    • Manoj Pahuja
      By Manoj Pahuja  ~  2 years ago
      reply Reply

      Things that are going to be overlap are:

      1. Test History - How we can use the history of a test to aid in debugging, focusing on the visual comparison for selenium/appium tests.
      2. Allure Reporting
      3. Filtering Results

      Obviously at high level the concept of creating the dashboard for measuring different metrics is the same but there may be differences in the technical implementations and how generic the dashboard can be.

      Besides the points mentioned above, this talk includes

      • Reasons for instable tests and how to manage those more effectively and automatically.
      • What kind of information you can capture and use to also improve the execution times for the tests.
      • After the dashboard has been created, what kind of alerts would result in an Action e.g. fix the failing test, fix the bug in the application.

      I wish there was a way to be able to connect with other speakers. Would love to hear about all the work, learnings, outcomes from the other talk. Certainly validates the concepts.

       


  • 45 Mins
    Demonstration
    Intermediate

    Typically in organizations, there are multiple projects / products. These products may be of implemented using tech-stacks over many years. Yet - they interact with each other in some way. To manage the complexity around Test Automation, many organizations prefer to have a common Test Automation solution across these products in an effort to build, standardize and maintain the framework.

    However, this is not a good idea! With this approach one potentially ends up having to compromise on the quality of automation that can be done for each product, limited by the toolset.

    The better approach would be to use the tools and technologies that are "right" for each product. This does have other disadvantages, but you would ensure each product is well tested! The only missing piece which remains is that these different products talk with each other. You need to test the integration between them in an automated way to verify all is well.

    "TaaS" is an open-source product solution that allows you do achieve the "correct" way of doing integration testing across a variety of products via Test Automation.

    Example: 

    For one set of products, Selenium-based toolset may be the right choice, where as for legacy reasons, QTP may be used for some other product. With TaaS - you will be able to automate the Integration Testing between these products, by re-using the tests already implemented in the individual product suites.

     

  • Liked Priyanka Gupta
    keyboard_arrow_down

    Priyanka Gupta / Sarah Eisen - Automation Alchemy On a Mass Scale: Turning Costly Manual Tests Into Automation Gold

    45 Mins
    Talk
    Beginner

    Do you want to hear a story about overcoming obstacles and achieving seemingly unattainable goals at a massive scale? Well, we have one to tell - it’s a true story, and like all good stories, teaches us some valuable lessons. We have gone through the ups and downs of this tale and come out better and smarter. We would love to share those experiences and learning with everyone.

    The story starts with a mission...automate 5000 hours of manual tests for our enterprise product. Like many other product based companies, we had one big monolithic application to test. The mission was to be accomplished with the resources available - no new magical dream team, we had to work with the resources we had - QA analysts with no technical background, a very small automation team, and a huge offshore manual testing group. Go figure! There was another twist - we had to accomplish our mission without dropping the current level of support for testing our enterprise application, including regression and new feature tests. Doesn't it all sound very familiar?

     

    This presentation will cover all aspects of our journey from the beginning to the end. We went through a lot of ups and down, and every single decision we made taught us a great deal. It is those experiences that we want to share with everyone.

    • We created a tool that wrapped the Selenium API in order to make it easy for non developers to write tests. The tests were written in a Domain Specific Language that made Selenium API calls with some application specific logic added in.
    • We needed to build our own execution framework to support our growing automated test base. The framework offered many customized features and was able to sustain 60,000 hours of tests running every single day.
    • We wrote our own best practices and worked closely with the QA team to make sure everyone wrote high quality tests.
    • The results from the tests needed to be displayed in a way that made sense. We created several different dashboards for that purpose and had many different views of the test suite performance, including a heat map to highlight problem areas.
    • Elasticsearch and Kibana were instrumental in helping us parse through the massive volume of test results and make sense of them, giving us metrics in different forms.
    • Daily environment setup for this execution was also massive - 100 or so slaves and several SUTs for every codeline, with support for 3 codelines meant that we needed a big lab setup.


    We successfully completed the mission of automating the manual test behemoth and gained a rich understanding of test automation at scale along the way.

  • Liked rubytester
    keyboard_arrow_down

    rubytester - Docker Selenium. Getting Started

    45 Mins
    Demonstration
    Beginner
    • `docker-selenium` project is about packaging selenium grid as docker containers (https://github.com/seleniumhq/docker-selenium)
      To me this means I don't have to build any selenium infrastructure machines. I just run the provided images by docker-selenium project (https://hub.docker.com/r/selenium/).
    • I don't have to install selenium jar, java, browsers and other runtime dependencies. They are already built in a docker image and I can just run them as either selenium grid with hub and nodes or as standalone selenium on any docker engine enabled vm. 

    In this talk/demo/case study I will show you how you can use `docker-selenium` project to build several pipelines starting from running on your local dev box to public cloud for quick tests and finally to a stable private cloud for your team.

     

  • Liked Adam Carmi
    keyboard_arrow_down

    Adam Carmi - Advanced Automated Visual Testing With Selenium

    Adam Carmi
    Adam Carmi
    Co-Founder and VP R&D
    Applitools
    schedule 2 years ago
    Sold Out!
    45 Mins
    Talk
    Intermediate

    Automated visual testing is a major emerging trend in the dev / test community. In this talk you will learn what visual testing is and why it should be automated. We will take a deep dive into some of the technological challenges involved with visual test automation and show how modern tools address them. We will review available Selenium-based open-source and commercial visual testing tools, demo cutting edge technologies that enable running cross browser and cross device visual tests at large scale, and show how visual test automation fits in the development / deployment lifecycle.

    If you don't know what visual testing is, if you think that Sikuli is a visual test automation tool, if you are already automating your visual tests and want to learn more on what else is out there, if you are on your way to implement Continuous Deployment or just interested in seeing how cool image processing algorithms can be, this talk is for you!

  • Liked Jonathan Lipps
    keyboard_arrow_down

    Jonathan Lipps - The Mobile JSON Wire Protocol

    45 Mins
    Talk
    Intermediate

    The JSON Wire Protocol (JSONWP) is the version of the WebDriver spec currently implemented by all the Selenium clients. It defines an HTTP API that models the basic objects of web automation---sessions, elements, etc... The JSON Wire Protocol is the magic that powers Selenium's client/server architecture, enables services like Selenium Grid or Sauce Labs to work, and gives you the ability to write your test scripts in any language.

    The JSONWP has served Selenium faithfully for a number of years, but the future of automated testing lies beyond the borders of the web browser. Mobile automation is an essential ingredient in any build, and tools like Appium or Selendroid have made it possible to run tests against mobile apps using the JSONWP. The JSONWP's current incarnation isn't enough to automate all the new behaviors that mobile apps support, however. Complex gestures, multiple device orientations, airplane mode, and the ability to use both native and web contexts, for example, are all essential to mobile automation.

    For this reason the leaders of the Selenium project, in concert with other Selenium-based projects like Appium and Selendroid, met to discuss the future of the JSONWP. We've been working on its next version, called the "Mobile JSON Wire Protocol" (MJSONWP). Appium and Selendroid already implement much of the MJSONWP spec. In this talk I'll dive into the specifics of the MJSONWP extensions, how they relate to the original JSONWP, and how the Selenium clients have begun to implement them.

    Finally, I will talk about the future of the MJSONWP and how it's related to the current and future versions of the WebDriver spec. I'll share how you can get help with the creation of the MJSONWP, and discuss issues with the authors of the new spec before the API is set in stone. We need the help of everyone who's involved in mobile automation to come up with the best and most future-proof version of the MJSONWP. Ultimately, your understanding of how Selenium works will be improved, and you'll have a much better handle on how projects like Appium and Selenium work together to make sure you have the best automation methods available.

  • Liked Justin Woolley
    keyboard_arrow_down

    Justin Woolley / David Anderson / Marvin Ojwang - Automating for the Second Screen with WebdriverJS

    45 Mins
    Talk
    Intermediate

    There has been a recent explosion in second-screen technologies such as Chromecast, but designing test automation for second-screen applications is far from straightforward. This new paradigm lacks major automated tool support, and coordinating test execution across multiple devices is tricky and error-prone.

    Our automation solution uses WebdriverJS and WebSockets to perform end-to-end test automation that covers our web player controller and second screen application.

    Learn about our approach to second-screen automation which we’ve used to build a reactive, responsive test suite. We’ll describe our solutions to synchronizing test flow between the controller and target device, validation on the device, targeting different integration components, and device management.

  • Liked James Farrier
    keyboard_arrow_down

    James Farrier / Xiaoxing Hu - Making Your Results Visible - A Test Result Dashboard and Comparison Tool

    45 Mins
    Demonstration
    Intermediate

    If a test fails in the woods and no one is there to see it does anyone care, does anyone even notice. What happens when failing tests become the norm and you can't see the wood from the trees? 

     

    After watching last years Allure Report presentation I was inspired.  Selenium tests (and automation tests in general) are often poorly understood by the team as a whole.  Reports/emails go unread with tests failing becoming an expected outcome rather than a glaring red flag.  We looked at what Allure brought to the table and from that base created a dashboard which was designed to:

    • Display the results of test runs in a way that was useful to managers, testers and the rest of the development team.  Including tools to filter out specific test runs and view the overall trend of the test run results.
    • Make debugging tests easier by grouping errors, displaying history of test results, filtering tests and offering visual comparison of test runs.
    • Help mitigate the problems flaky tests cause with test run result reporting (say that three times fast).
    • Help with our mobile device certification process, by easily providing a view to compare test runs across devices.

    Since it's creation the dashboard has been used and praised by managers through to developers.  With our full suite of tests from unit to integration to selenium and appium being stored on the dashboard.  We've managed to:

    • Decrease the time taken to debug test cases.
    • Increase the visibility of all our test suites, with managers having a better idea of how our selenium test suite is progressing and testers better understanding the coverage of unit tests.
    • Focus the organization on quality.

    We are working with legal at present to have this project open sourced and available to all prior to Selenium Conf 2015.

  • Liked Ragavan Ambighananthan
    keyboard_arrow_down

    Ragavan Ambighananthan - Distributed Automation Using Selenium Grid / AWS / Autoscaling

    45 Mins
    Talk
    Advanced

    Speed of UI automation has always been an issue when it comes to Continuous Integration / Continuous Delivery. If UI automation suite takes 3 hours to complete, then any commit happens during this time will not be visible in test environment, because the next deployment will happen only after 3 hours. 

    With 2000+ developers and average 250+ checkins per day, the above issues is replicated 250+ times every day. This is not productive and feedback cycle is super slow!

    Another issue is , with 35+ different project teams using 10 or more different jenkins jobs to run their UI automation. So many jobs means (350+), individual teams need to go through the pain of managing their own jenkins job, its a duplicate effort and waste of time. Automation teams need to spend time on writing reliable automation and not managing jenkins jobs.

    Solution is to reduce the UI automation run time from hours to minutes and also use only handful of jobs to run the Distributed Automation!

    Goal: To run all UI automation scenarios within the time take by the longest test case

  • Liked Sveta Kostinsky
    keyboard_arrow_down

    Sveta Kostinsky - Selenium Today vs. Selenium Tomorrow: Digital as the Convergence of Mobile & Web Programs

    45 Mins
    Talk
    Beginner

    Today, mobile is increasingly trumping web as the most important brand engagement point; enterprises are moving away from mobile and web projects independent of each other. The rapid adoption of responsive web encourages teams to discover one approach to measuring software quality regardless of form factors.

     

    Selenium is current market leading solution for web testing, but how does it stand with mobile? The truth is that working with Selenium presents a few challenges, including:

    • Building and maintaining an internal structure to support it
    • Bridging an architectural gap
    • Requirements demand support for unattended test execution
    • Lack of real network conditions for mobile testing

     

    There is a solution to address these challenges!

    Let’s work through a demo and show how to test mobile & web in parallel with Selenium

  • Liked Russell Rutledge
    keyboard_arrow_down

    Russell Rutledge - Blazing Fast UI Validation - 5000 Reliable Tests in 10 Minutes on One Machine

    Russell Rutledge
    Russell Rutledge
    Senior Technical Lead
    Nike
    schedule 2 years ago
    Sold Out!
    45 Mins
    Talk
    Advanced

    A big blocker for putting a website on truly continuous production delivery is the amount of time it take to validate that the site works correctly.  Tests themselves take time to run, and test results are unreliable to the point where it takes a human to investigate and interpret them.  When counting the time that it takes to both run and interpret results, test runs for an enterprise web site can take an entire day from inception to useful result.

    This session describes common points of failure in test execution that add both latency and unreliability and what can be done to overcome them while still preserving the value of UI validation.  We'll discuss why, after addressing these concerns, UI can be unblocked to reliably field thousands of validation scenarios on a local machine in a matter of minutes. 

  • Liked David Giffin
    keyboard_arrow_down

    David Giffin - A Large-scale, Data-driven Company's Journey of Going From Manual to Automated Testing In 6 Months

    David Giffin
    David Giffin
    Technologist
    TrueCar
    schedule 2 years ago
    Sold Out!
    45 Mins
    Talk
    Beginner

    Manual Testing.  Depending on how you've been influenced by those two simple words, reactions vary from slight disgust to full-on depression.  Of course, the solution is clear: automate, but how do you get there when your company is continually pushing out the next big feature?  As the set of features to cover increases, the lack of scalability of manual testing becomes more apparent.

     

    This is a problem that we struggled with at our company.  Automation tactics were explored and implemented, but problems persisted as proposed solutions did not cater to the demands of the manual testers.

     

    After years of failure and disappointment, our latest stint resulted in success.  Not only do we have hundreds of automated tests across various platforms (mobile and web) and products, but manual testing has been eliminated with zero casualties.  As we move forward towards Continuous Delivery and improved automation performance, we wanted to take this moment to look back and share stories of failure and success.

  • Liked Surendran Ethiraj
    keyboard_arrow_down

    Surendran Ethiraj - VAS (Value Added Services) using Selenium Webdriver

    45 Mins
    Demonstration
    Intermediate

    The evolution of Test Automation started with automation tools that had record and playback features.  This allowed Automation Testers to record and structure the script in such a way that it could be reused. Tools like Selenium, which provided APIs, could interact with different browsers. The Automation Testers could use these APIs to interact with web applications. Additionally, it was possible to develop frameworks for reusing each of the components of the framework. Currently, the focus has shifted more and more towards the designing of frameworks rather than just the tools, so that the testing framework could be integrated with test management applications and continuous integration tools to aid test-driven development.

    With that background, we have come up with certain Value Added Services (VAS), a step ahead of developing functional automation scripts. Imagine creating an Automation Framework which will not just check for the functionality of the application, but also check for security, page performance, page layout, accessibility and have an output that can be a trigger to other aspects of testing.

    This paper presents three of the Value Added Services that we offer. We are working on creating many more such services on top of the Automation Framework.

  • Liked Dipesh Bhatewara
    keyboard_arrow_down

    Dipesh Bhatewara - UI Automation As A Web Service

    Dipesh Bhatewara
    Dipesh Bhatewara
    Staff Engineer
    VMware Software
    schedule 2 years ago
    Sold Out!
    45 Mins
    Talk
    Intermediate

    In modern cloud aeon, customers are delivered with UI and API for most of the Enterprise Products. The QE teams have to test the product for UI and APIs from functional as well as non functional perspective. For these they typically end up writing different test suites.

    There are lot of challenges one faces when automating test suites for these different purposes, like

    1. How do you integrate UI automation seamlessly with different test frameworks/suites?
    2. How do you test all these requirements (UI, API, Performance etc.) effectively without a lot of redundant or duplicate code?
    3. How do you use the best fit language/technology for UI automation and still do not impact the grand test automation strategies?

    There is an interesting solution to conquer these challenges which I have recently implemented in one of my projects. 

    The idea is to segregate tests from the UI Automation OR webdriver code. This code can be exposed as a web service APIs. Such service allows us to write common tests for UI as well as API. Tests written for API testing can be reused for UI testing with very minimal configuration modification like pointing to appropriate web service. These UI Automation APIs can also be called easily from any performance test requiring UI interaction.

    This approach will enable any test suite using any language or technology in your organization to reuse UI automation seamlessly. This will also make lot of things straight and simple for execution and management across the verticals.

     

    We will deep dive into the technical solution for this in this talk.

  • Liked Anand Bagmar
    keyboard_arrow_down

    Anand Bagmar - Create your Future!

    45 Mins
    Talk
    Beginner

    With all the focus on Test practices and tools getting better for the present, it is easy to forget about the Future of Testing, especially in terms of Tools and Infrastructure? A question that always comes up in my mind - “Are we so caught up in the past and present, that we will not be as effective in the future?”

    In this talk, we will go on a journey to figure out what new challenges are coming up in the future, and more importantly, what do we need to do next to prepare for it. Also, just preparing for the future is not sufficient. We have an opportunity to stretch beyond our current set of skills, capabilities and boundaries to influence out future! The question is - will we make use of this opportunity? Will Selenium help us take the step in the right direction? Or, will it hold us back?

  • Liked Jason Watt
    keyboard_arrow_down

    Jason Watt - Challenges of the Mobile Cloud

    Jason Watt
    Jason Watt
    Senior Software Engineer
    Salesforce
    schedule 2 years ago
    Sold Out!
    45 Mins
    Talk
    Intermediate

    Creating a mobile app is now the new cross platform problem. The major mobile platforms tend to gear their development tool chain towards individuals and their workstations.  But what if you want to introduce a CI solution to this environment? What if your app is launching on more than one platform and there's a team of 20+ developers working on it? What if your tests are more than just Selenium based?

    This is normally where you can look to the cloud for scale but mobile has a ton of challenges to do so.  Come and learn from some of the challenges and pitfalls I've encountered while working towards this goal.

  • Liked Mike Levin
    keyboard_arrow_down

    Mike Levin - Selenium meta hub – scalable and redundant infrastructure

    45 Mins
    Talk
    Intermediate
     
    Selenium grid is widely used in lots of companies and projects. Unfortunately with the current open source implementation one can not run more than one hub which can cause various problems due to hardware or network instability. Single hub architecture is also hardly scalable, at least it requires hardware upgrades for the hub.
    At the same time many teams implement their own internal solutions which are usually not shared because of the team/organization specific or custom selenium hacks.
    In Yandex we have multi hub solution for more than 5 years. At the same time we are doing our best to avoid making custom internal patches to selenium.
    During these years we were using client side balancing approach: client applications were always obtaining a browser via a special internal library. It knew the configuration of all the hubs and browsers and was performing the search for an available node on request.
    But when it came to the different test frameworks, different languages and different runtimes this approach became difficult to support. As long as test practices move from test engineers to development teams the diversity of frameworks and runtimes increases. So we come up with meta hub solution.
     
    Our meta hub solution has the following basis:
    • Stock versions of selenium and selenium grid
    • Stock web driver interface for the client
    • Virtual infrastructure. We use Openstack for all parts of our infrastructure. It’s not necessary but It makes sense.
    • Fixed load for each hub and node – scalability via adding new hubs with fixed volume.
    • Redundancy and scalability
    • Stateless solution for meta hub. No storage requires to keep state between several meta hub nodes
     
    We made solution that includes:
    • Proxy software between client and multi-hub grid installation
    • Some configuration adjustments for hubs/nodes.
     
    I'll talk about our solution. We going to go open source on the SeleniumConf conference or earlier.

     

  • Liked Sarah Thompson
    keyboard_arrow_down

    Sarah Thompson - DevOps meets QA - Using Puppet to set up and manage your Selenium Grid

    Sarah Thompson
    Sarah Thompson
    QE UI Toolsmith
    Puppet Labs
    schedule 2 years ago
    Sold Out!
    45 Mins
    Talk
    Intermediate

    For testers, setting up and maintaining a Selenium Grid infrastructure can be timely and costly. A lot of the time, we are asked to do this as part of our day to day job when we really want to focus on testing the product!

    There are some great cloud based alternatives out there that allow you to easily run your tests on a wide range of Operating Systems and against multiple browser types (at a price).

    But what if you already have plenty of devices available within your own company (be it physical machines, virtual machines or cloud based resources) and you want to be able to setup and manage your own grid infrastructure:

    • to allow you greater control over the configuration (perhaps you want to have a headless browser like PhantomJS available on your grid)
    • to run your tests faster (the cloud based alternatives are a good bit slower for obvious reasons)
    • or to simply save money.
  • Liked David Anderson
    keyboard_arrow_down

    David Anderson / Justin Woolley / Marvin Ojwang - Reactive Page Modeling using WebDriverJS and RxJs

    45 Mins
    Talk
    Intermediate

    Selenium is mostly used to test JavaScript/HTML applications.  However, several organizations use Java WebDriver bindings to develop their automation.  Our team has learned that WebDriverJS provides a number of benefits over the traditional approach.  Asynchronous constructs in Javascript, such as Promises and Reactive-Extensions (RxJs), help solve many of the challenges faced when developing automated tests for browser based applications. With the increase in popularity of Node.js, JavaScript is being utilized across the entire web development stack.  WebDriverJS complements this trend, by enabling functional tests to be written in the same language as unit tests and production code.


    Learn how your team can benefit from the adoption of WebDriverJS and complimentary JavaScript tools and libraries.  We will discuss our motivations, as well as the architectural and design decisions that applied to our use case.  The presentation will conclude with a real-world demo of using WebDriverJS and RxJs to implement the Page Object Model.

  • Liked Satyajit Malugu
    keyboard_arrow_down

    Satyajit Malugu - Adapting testing and automation for Mobile First

    Satyajit Malugu
    Satyajit Malugu
    Senior mobile tester
    Godaddy
    schedule 2 years ago
    Sold Out!
    45 Mins
    Talk
    Intermediate

    Mobile-first is a paradigm shift in developing software where products are created with an emphasis on mobile experiences, rather than desktop. Mobile sites, native apps become the new centerfold for the company around which various other initiatives and goals are driven. This paradigm has been in adoption by major industry players including Google, facebook, Airbnb, ebay, shopify etc.

     

    From a development standpoint it is not really different from traditional websites after you get UX but testing is quite different. You have to consider various devices, screen sizes, performance and visual testing. Automation should focus on tackling the most time consuming manual aspects which include functionality, visual parity and device compatibility.

     

    Our $5B company has been in a mobile-first transformation this year, spanning many legacy products and teams. As a senior SDET leading the testing efforts I have been involved in various phases of this transformation. I believe this talk would give practical insights in mobile web development and testing.

  • Liked Dmytro Zharii
    keyboard_arrow_down

    Dmytro Zharii - Smoke Tests and Page Objects: Let's do the right thing fast!

    45 Mins
    Talk
    Intermediate

    "Write the Smoke Tests" -- this is the ultimate answer for the questions:
    - How do I start a new test automation project?
    - What is the way to test that my PageObjects are still working correctly after the "nightly" product changes?
    - What is the fastest way for my Test Automation Framework to bring the real value for my teammates and managers?

    Can you imagine you can generate the Page Objects code in a fast and easy way with SWD Page Recorder?
    Can you imagine you can generate the Smoke Tests for your PageObjects and the Application Under Test by pressing one button?

    Let me show how you can increase your productivity in average 45% by using good tooling, patterns and practices during your daily work.