Blazing Fast UI Validation - 5000 Reliable Tests in 10 Minutes on One Machine

A big blocker for putting a website on truly continuous production delivery is the amount of time it take to validate that the site works correctly.  Tests themselves take time to run, and test results are unreliable to the point where it takes a human to investigate and interpret them.  When counting the time that it takes to both run and interpret results, test runs for an enterprise web site can take an entire day from inception to useful result.

This session describes common points of failure in test execution that add both latency and unreliability and what can be done to overcome them while still preserving the value of UI validation.  We'll discuss why, after addressing these concerns, UI can be unblocked to reliably field thousands of validation scenarios on a local machine in a matter of minutes. 

 
26 favorite thumb_down thumb_up 1 comment visibility_off  Remove from Watchlist visibility  Add to Watchlist
 

Outline/structure of the Session

Slides (bullets and diagrams) and demo

Learning Outcome

After attending this session you will understand common test environment setup (tools and product) that can add instability and latency to standard UI test run and also what strategy can be taken to sidestep these effects while still gaining the same value in test results.  You will understand the limitations of this setup and when a traditional environment still adds irreplaceable value to UI validation.

Target Audience

Website test architects and senior test engineers

schedule Submitted 2 years ago

Comments Subscribe to Comments

comment Comment on this Proposal
  • Dave Haeffner Test
    By Dave Haeffner Test  ~  2 years ago
    reply Reply

    Please provide more detail about the approach you're going to cover. Also, can you please provide a rough outline with approximate timing? We want to make sure the talk is not going to be too high level and that it can fill the time slot accordingly.


    • Liked Anand Bagmar
      keyboard_arrow_down

      Anand Bagmar - To Deploy or Not-to-Deploy - decide using TTA's Trend & Failure Analysis

      Anand Bagmar
      Anand Bagmar
      Director - Quality
      Vuclip Inc.
      schedule 2 years ago
      Sold Out!
      45 mins
      Demonstration
      Intermediate

      The key objectives of organizations is to provide / derive value from the products / services they offer. To achieve this, they need to be able to deliver their offerings in the quickest time possible, and of good quality!

      In order for these organizations to to understand the quality / health of their products at a quick glance, typically a team of people scramble to collate and collect the information manually needed to get a sense of quality about the products they support. All this is done manually.

      So in the fast moving environment, where CI (Continuous Integration) and CD (Continuous Delivery) are now a necessity and not a luxury, how can teams take decisions if the product is ready to be deployed to the next environment or not?

      Test Automation across all layers of the Test Pyramid (be it Selenium-based UI tests, or, xUnit based unit tests, or, Performance Tests, etc.) is one of the first building blocks to ensure the team gets quick feedback into the health of the product-under-test. 

      The next set of questions are:
          •    How can you collate this information in a meaningful fashion to determine - yes, my code is ready to be promoted from one environment to the next?
          •    How can you know if the product is ready to go 'live'?
          •    What is the health of you product portfolio at any point in time?
          •    Can you identify patterns and do quick analysis of the test results to help in root-cause-analysis for issues that have happened over a period of time in making better decisions to better the quality of your product(s)?

      The current set of tools are limited and fail to give the holistic picture of quality and health, across the life-cycle of the products.

      The solution - TTA - Test Trend Analyzer

      TTA is an open source product that becomes the source of information to give you real-time and visual insights into the health of the product portfolio using the Test Automation results, in form of Trends, Comparative Analysis, Failure Analysis and Functional Performance Benchmarking. This allows teams to take decisions on the product deployment to the next level using actual data points, instead of 'gut-feel' based decisions.

    • moiz
      moiz
      Software Engineer
      Saucelabs
      schedule 2 years ago
      Sold Out!
      45 mins
      Talk
      Beginner

      Appium, often dubbed "Selenium for mobile", at heart its a web server written in NodeJs. Its architecture is modular, which means that it is composed of many small, independently maintained and tested modules. Testing Appium is challenging, but clearly very important, since thousands of users depend on it for their testing. Appium also has all the usual challenges of a large open source project, for example, ensuring consistency of JavaScript code style across hundreds of contributors. It's important to have high-quality and readable code.

       
      I will be discussing approaches to and strategies for testing these kinds of large, modular applications. On the Appium team, we use a combination of unit, functional, and integration tests. Modern services like GitHub, Travis CI, and Sauce Labs make it possible for large open source projects to be tested thoroughly, keeping the code and the app at high quality. I will also discuss the use of tools like JSLint and Gulp, which help prevent code style issues.
       
      Testing the tool which is used for testing is clearly very important. This talk aims to showcase how testing should be approached for large, modular projects which has many collaborators.
    • Liked Priyanka Gupta
      keyboard_arrow_down

      Priyanka Gupta / Sarah Eisen - Automation Alchemy On a Mass Scale: Turning Costly Manual Tests Into Automation Gold

      45 mins
      Talk
      Beginner

      Do you want to hear a story about overcoming obstacles and achieving seemingly unattainable goals at a massive scale? Well, we have one to tell - it’s a true story, and like all good stories, teaches us some valuable lessons. We have gone through the ups and downs of this tale and come out better and smarter. We would love to share those experiences and learning with everyone.

      The story starts with a mission...automate 5000 hours of manual tests for our enterprise product. Like many other product based companies, we had one big monolithic application to test. The mission was to be accomplished with the resources available - no new magical dream team, we had to work with the resources we had - QA analysts with no technical background, a very small automation team, and a huge offshore manual testing group. Go figure! There was another twist - we had to accomplish our mission without dropping the current level of support for testing our enterprise application, including regression and new feature tests. Doesn't it all sound very familiar?

       

      This presentation will cover all aspects of our journey from the beginning to the end. We went through a lot of ups and down, and every single decision we made taught us a great deal. It is those experiences that we want to share with everyone.

      • We created a tool that wrapped the Selenium API in order to make it easy for non developers to write tests. The tests were written in a Domain Specific Language that made Selenium API calls with some application specific logic added in.
      • We needed to build our own execution framework to support our growing automated test base. The framework offered many customized features and was able to sustain 60,000 hours of tests running every single day.
      • We wrote our own best practices and worked closely with the QA team to make sure everyone wrote high quality tests.
      • The results from the tests needed to be displayed in a way that made sense. We created several different dashboards for that purpose and had many different views of the test suite performance, including a heat map to highlight problem areas.
      • Elasticsearch and Kibana were instrumental in helping us parse through the massive volume of test results and make sense of them, giving us metrics in different forms.
      • Daily environment setup for this execution was also massive - 100 or so slaves and several SUTs for every codeline, with support for 3 codelines meant that we needed a big lab setup.


      We successfully completed the mission of automating the manual test behemoth and gained a rich understanding of test automation at scale along the way.

    • Liked rubytester
      keyboard_arrow_down

      rubytester - Docker Selenium. Getting Started

      45 mins
      Demonstration
      Beginner
      • `docker-selenium` project is about packaging selenium grid as docker containers (https://github.com/seleniumhq/docker-selenium)
        To me this means I don't have to build any selenium infrastructure machines. I just run the provided images by docker-selenium project (https://hub.docker.com/r/selenium/).
      • I don't have to install selenium jar, java, browsers and other runtime dependencies. They are already built in a docker image and I can just run them as either selenium grid with hub and nodes or as standalone selenium on any docker engine enabled vm. 

      In this talk/demo/case study I will show you how you can use `docker-selenium` project to build several pipelines starting from running on your local dev box to public cloud for quick tests and finally to a stable private cloud for your team.

       

    • Liked Rémi
      keyboard_arrow_down

      Rémi - Mobile end to end testing at scale: stable, useful, easy. Pick three.

      Rémi
      Rémi
      Software engineer
      Facebook
      schedule 2 years ago
      Sold Out!
      45 mins
      Talk
      Beginner

      This talk is about how Facebook turned a great idea with a terrible track record into a great tool for thousands of developers.

      The promise of E2E testing — complex, real-world test scenarios from the point of view of and end user — is appealing.
      Many attempts have been made over the years at automating large parts of companies' and developers' testing and release processes, yet most of these efforts ended up in bitter and hard learned lessons about the inherent challenges of the whole approach.

      My work at Facebook over the last two years has been making mobile end to end testing at scale a reality.
      When others said it couldn't be done, or fell by the wayside, we relentlessly pushed forward, solving problems deemed intractable, and finding new, untold vistas of horror before us

      We've come a long way: E2E testing is now an integral part of Facebook's mobile development and release process.
      We'll cover what challenges we faced, and how we chose to solve or make them irrelevant.

    • Liked Adam Carmi
      keyboard_arrow_down

      Adam Carmi - Advanced Automated Visual Testing With Selenium

      Adam Carmi
      Adam Carmi
      Co-Founder and VP R&D
      Applitools
      schedule 2 years ago
      Sold Out!
      45 mins
      Talk
      Intermediate

      Automated visual testing is a major emerging trend in the dev / test community. In this talk you will learn what visual testing is and why it should be automated. We will take a deep dive into some of the technological challenges involved with visual test automation and show how modern tools address them. We will review available Selenium-based open-source and commercial visual testing tools, demo cutting edge technologies that enable running cross browser and cross device visual tests at large scale, and show how visual test automation fits in the development / deployment lifecycle.

      If you don't know what visual testing is, if you think that Sikuli is a visual test automation tool, if you are already automating your visual tests and want to learn more on what else is out there, if you are on your way to implement Continuous Deployment or just interested in seeing how cool image processing algorithms can be, this talk is for you!

    • Liked Jonathan Lipps
      keyboard_arrow_down

      Jonathan Lipps - The Mobile JSON Wire Protocol

      45 mins
      Talk
      Intermediate

      The JSON Wire Protocol (JSONWP) is the version of the WebDriver spec currently implemented by all the Selenium clients. It defines an HTTP API that models the basic objects of web automation---sessions, elements, etc... The JSON Wire Protocol is the magic that powers Selenium's client/server architecture, enables services like Selenium Grid or Sauce Labs to work, and gives you the ability to write your test scripts in any language.

      The JSONWP has served Selenium faithfully for a number of years, but the future of automated testing lies beyond the borders of the web browser. Mobile automation is an essential ingredient in any build, and tools like Appium or Selendroid have made it possible to run tests against mobile apps using the JSONWP. The JSONWP's current incarnation isn't enough to automate all the new behaviors that mobile apps support, however. Complex gestures, multiple device orientations, airplane mode, and the ability to use both native and web contexts, for example, are all essential to mobile automation.

      For this reason the leaders of the Selenium project, in concert with other Selenium-based projects like Appium and Selendroid, met to discuss the future of the JSONWP. We've been working on its next version, called the "Mobile JSON Wire Protocol" (MJSONWP). Appium and Selendroid already implement much of the MJSONWP spec. In this talk I'll dive into the specifics of the MJSONWP extensions, how they relate to the original JSONWP, and how the Selenium clients have begun to implement them.

      Finally, I will talk about the future of the MJSONWP and how it's related to the current and future versions of the WebDriver spec. I'll share how you can get help with the creation of the MJSONWP, and discuss issues with the authors of the new spec before the API is set in stone. We need the help of everyone who's involved in mobile automation to come up with the best and most future-proof version of the MJSONWP. Ultimately, your understanding of how Selenium works will be improved, and you'll have a much better handle on how projects like Appium and Selenium work together to make sure you have the best automation methods available.

    • 45 mins
      Demonstration
      Intermediate

      How many times do we test the same things at multiple layers, multiple levels, adding time to the build process and testing cycle, delaying the feedback?

      We know what to test and how to test, but what is the right place to test it?

      In this workshop, we will demonstrate how as QA’s we can identify what tests can be classified as unit tests, integration test and functional test. Using a case study, we will see how each component can be tested as part of unit testing; the integration of different parts and the functioning of a software system as a whole and how functional tests fit into this big picture. We will then bring all these tests together to understand and build the testing pyramid and how it enables us to build the right testing framework with fewer selenium i.e functional tests.

    • Liked Andrei Solntsev
      keyboard_arrow_down

      Andrei Solntsev - Selenide: Concise UI Tests in Java

      45 mins
      Demonstration
      Advanced

      Selenium WebDriver is a great tool, but it's not a testing library. It's a browser manipulation tool. There is a gap. 

      Selenide adds a possibility for easy and stable testing.

       

      Why yet another Selenium wrapper?

      There are several testing libraries around Selenium webdriver. But it seems that they do not resolve the main problems of UI tests. Namely, instability of tests caused by dynamic content, JavaScript, Ajax, timeouts etc. Selenide was created to resolve these problems. With Selenide, you can forget all these common timing issues and concentrate on business logic.

       
    • Liked Justin Woolley
      keyboard_arrow_down

      Justin Woolley / David Anderson / Marvin Ojwang - Automating for the Second Screen with WebdriverJS

      45 mins
      Talk
      Intermediate

      There has been a recent explosion in second-screen technologies such as Chromecast, but designing test automation for second-screen applications is far from straightforward. This new paradigm lacks major automated tool support, and coordinating test execution across multiple devices is tricky and error-prone.

      Our automation solution uses WebdriverJS and WebSockets to perform end-to-end test automation that covers our web player controller and second screen application.

      Learn about our approach to second-screen automation which we’ve used to build a reactive, responsive test suite. We’ll describe our solutions to synchronizing test flow between the controller and target device, validation on the device, targeting different integration components, and device management.

    • Liked James Farrier
      keyboard_arrow_down

      James Farrier / Xiaoxing Hu - Making Your Results Visible - A Test Result Dashboard and Comparison Tool

      45 mins
      Demonstration
      Intermediate

      If a test fails in the woods and no one is there to see it does anyone care, does anyone even notice. What happens when failing tests become the norm and you can't see the wood from the trees? 

       

      After watching last years Allure Report presentation I was inspired.  Selenium tests (and automation tests in general) are often poorly understood by the team as a whole.  Reports/emails go unread with tests failing becoming an expected outcome rather than a glaring red flag.  We looked at what Allure brought to the table and from that base created a dashboard which was designed to:

      • Display the results of test runs in a way that was useful to managers, testers and the rest of the development team.  Including tools to filter out specific test runs and view the overall trend of the test run results.
      • Make debugging tests easier by grouping errors, displaying history of test results, filtering tests and offering visual comparison of test runs.
      • Help mitigate the problems flaky tests cause with test run result reporting (say that three times fast).
      • Help with our mobile device certification process, by easily providing a view to compare test runs across devices.

      Since it's creation the dashboard has been used and praised by managers through to developers.  With our full suite of tests from unit to integration to selenium and appium being stored on the dashboard.  We've managed to:

      • Decrease the time taken to debug test cases.
      • Increase the visibility of all our test suites, with managers having a better idea of how our selenium test suite is progressing and testers better understanding the coverage of unit tests.
      • Focus the organization on quality.

      We are working with legal at present to have this project open sourced and available to all prior to Selenium Conf 2015.

    • Liked Ragavan Ambighananthan
      keyboard_arrow_down

      Ragavan Ambighananthan - Distributed Automation Using Selenium Grid / AWS / Autoscaling

      45 mins
      Talk
      Advanced

      Speed of UI automation has always been an issue when it comes to Continuous Integration / Continuous Delivery. If UI automation suite takes 3 hours to complete, then any commit happens during this time will not be visible in test environment, because the next deployment will happen only after 3 hours. 

      With 2000+ developers and average 250+ checkins per day, the above issues is replicated 250+ times every day. This is not productive and feedback cycle is super slow!

      Another issue is , with 35+ different project teams using 10 or more different jenkins jobs to run their UI automation. So many jobs means (350+), individual teams need to go through the pain of managing their own jenkins job, its a duplicate effort and waste of time. Automation teams need to spend time on writing reliable automation and not managing jenkins jobs.

      Solution is to reduce the UI automation run time from hours to minutes and also use only handful of jobs to run the Distributed Automation!

      Goal: To run all UI automation scenarios within the time take by the longest test case

    • Liked Sveta Kostinsky
      keyboard_arrow_down

      Sveta Kostinsky - Selenium Today vs. Selenium Tomorrow: Digital as the Convergence of Mobile & Web Programs

      45 mins
      Talk
      Beginner

      Today, mobile is increasingly trumping web as the most important brand engagement point; enterprises are moving away from mobile and web projects independent of each other. The rapid adoption of responsive web encourages teams to discover one approach to measuring software quality regardless of form factors.

       

      Selenium is current market leading solution for web testing, but how does it stand with mobile? The truth is that working with Selenium presents a few challenges, including:

      • Building and maintaining an internal structure to support it
      • Bridging an architectural gap
      • Requirements demand support for unattended test execution
      • Lack of real network conditions for mobile testing

       

      There is a solution to address these challenges!

      Let’s work through a demo and show how to test mobile & web in parallel with Selenium

    • Liked rajesh sarangapani
      keyboard_arrow_down

      rajesh sarangapani / Prabhu Epuri - Visualizing Real User Experience Using Integrated Open Source Stack (Selenium + Jmeter + Appium + Visualization tools)

      45 mins
      Demonstration
      Advanced

      Traditional approach in performance testing does not include client side processing time (i.e. DOM Content Load, Page Render, JavaScript Execution, etc.) as part of response times, performance tests has always been conducted to stress the server so tools like Jmeter have been very popular to execute tests. With increasing complexity of architectures (Web, Browser, Mobile) on the client side it has been important to understand the real user experience.   Commercial tools have started to provide features that can provide insights into real user experience after the bytes are transferred to the client end.  With the ability to call Selenium scripts via Jmeter the ability to conduct real user experience tests using open source stack has opened up new avenues to comment on real user experience.   This enables us to comment on

      • Provides Page load times similar to On Load time of real browsers
      • Generates HAR file with following statistics
      • Details of summary of request times and content types
      • Waterfall chart with page download time breakdown statistics such as  DNS resolution time, Connection time, SSL handshaking time, Request send time, wait time and receive time.

      By integrating the open source stack tools it enables us to provide the same insights which a commercial of the shelf tools would offer.   At Gallop we have implemented this at multiple clients providing them insights into various bottlenecks at the client side which helped us to provide greater value proposition

    • Liked James Eisenhauer
      keyboard_arrow_down

      James Eisenhauer - An Introduction to the World of Node, Javascript & Selenium

      45 mins
      Talk
      Beginner

      Ever wanted to write Selenium code in Node.js?  There seems to be a new javascript library written every hour!  Entering the world of Node.js can be a daunting task.  This session will teach you everything you need to know to make the right decisions when selecting what libraries you should implement on your new Node.js Selenium project and what the possible challenges will be.

       

       

    • Liked David Giffin
      keyboard_arrow_down

      David Giffin - A Large-scale, Data-driven Company's Journey of Going From Manual to Automated Testing In 6 Months

      David Giffin
      David Giffin
      Technologist
      TrueCar
      schedule 2 years ago
      Sold Out!
      45 mins
      Talk
      Beginner

      Manual Testing.  Depending on how you've been influenced by those two simple words, reactions vary from slight disgust to full-on depression.  Of course, the solution is clear: automate, but how do you get there when your company is continually pushing out the next big feature?  As the set of features to cover increases, the lack of scalability of manual testing becomes more apparent.

       

      This is a problem that we struggled with at our company.  Automation tactics were explored and implemented, but problems persisted as proposed solutions did not cater to the demands of the manual testers.

       

      After years of failure and disappointment, our latest stint resulted in success.  Not only do we have hundreds of automated tests across various platforms (mobile and web) and products, but manual testing has been eliminated with zero casualties.  As we move forward towards Continuous Delivery and improved automation performance, we wanted to take this moment to look back and share stories of failure and success.

    • Liked Trinath Babu
      keyboard_arrow_down

      Trinath Babu - Visual Test Automation using Selenium

      Trinath Babu
      Trinath Babu
      Sr. Manager
      Gallop Solutions
      schedule 2 years ago
      Sold Out!
      45 mins
      Talk
      Intermediate

      Visual Test Automation using Selenium

      Visual Testing is the method of verifying that the application’s GUI appears correctly to its users. Most of the people say visual testing is hard to automate. Given the number of web browsers, operating systems, screen resolutions, responsive design, internationalization, etc.) the nature of visual testing can be complex. But with existing open source and commercial solutions, this complexity is manageable, making it easier to automate than it once was, since verification with traditional automated functional testing tools can be very challenging.

      It can be easily achieved by integrating Selenium with Applitools. This talk mainly focuses on verifying the application’s graphical user interfaces (GUI) and finding the visual bugs using Applitools. It is very helpful for all sites having graphical functionalities like (charts, graph, dashboards etc).  Verify that the GUI appears correctly across all devices & browsers. The nature of visual testing can be complex. But with existing open source and commercial solutions, this complexity is manageable, making it easier to automate than it once was. And the payoff is well worth the effort.

      Take pressure off manual QA: increase coverage, test faster & more accurately.  Reduce maintenance efforts: automatically propagate changes across execution environments. Release faster, with confidence & flawless.

      Applitools Eyes Express captures the screen you want to test, and compares it to a baseline image – instantly, with a single click. No extra testing code necessary, no boring error logs.

      For example, a single automated visual test will look at a page and assert that every element on it has rendered correctly. Effectively checking hundreds of things and telling you if any of them are out of place. This will occur every time the test is run, and it can be scaled to each browser, operating system, and screen resolution you care about.

      Put another way, one automated visual test is worth hundreds of assertions. And if done in the service of an iterative development workflow, then you’re one giant leap closer.

       

      Each of these tools follows some variation of the following work flow:

      1. Drive the application under test (AUT) and take a screenshot
      2. Compare the screenshot with an initial “baseline” image
      3. Report the differences
      4. Update the baseline as needed
    • Liked Titus Fortner
      keyboard_arrow_down

      Titus Fortner - What Are We Testing, Anyway?

      45 mins
      Talk
      Intermediate

      Testing strategies and the role of DOM to Database Testing in a world of micro services and client side MVCs.

      The trends in software development are making UI testing increasingly difficult. Sites are leveraging more dynamic interactions and moving toward Single Page Applications. Gone are the days when the term “and the page finishes loading” makes any sense. This shift is dramatically increasing the number of flaky tests as well as the costs of such testing relative to the benefits, leaving many organizations wondering if they are worth doing at all. 

      The approach to testing that is “good enough” for any given organization is going to vary by context. In this talk, I’ll cover some different testing options and the advantages and disadvantages to each. We’ll discuss the dangers of mocking and stubbing, the problems with relying on testing journeys, and dealing with bloated test suites that are difficult to maintain.

      Another trend in software development is away from monolithic architectures and toward micro services and service oriented architectures. This approach provides opportunities for decreasing the costs and overhead of UI testing while still maintaining all of the benefits of DOM to Database verification.

    • Liked Oren Rubin
      keyboard_arrow_down

      Oren Rubin - Selenium Wat!

      Oren Rubin
      Oren Rubin
      Founder and CEO of Testim
      Testim.io
      schedule 2 years ago
      Sold Out!
      45 mins
      Case Study
      Intermediate

      Every language and framework which lives as long as Selenium has its fuckups.. and we're here to embrance them and joke about them.

      E.g. JS Wat https://www.youtube.com/watch?v=FqhZZNUyVFM

    • Liked Tanay Nagjee
      keyboard_arrow_down

      Tanay Nagjee - Run your Selenium tests in a fraction of the time

      45 mins
      Talk
      Intermediate

      Comprehensive functional testing is a widely accepted best practice and Selenium is the prevalent tool of choice. But long-running Selenium test suites cannot be fit into short continuous integration (CI) cycles and are often run once a night or less. Running functional tests less frequently means bugs are discovered later than they should. Tanay Nagjee will discuss how Selenium test suites can be parallelized at a very fine granularity and included in CI builds. By leveraging a cluster of compute horsepower (on-premise and/or cloud), large Selenium suites can execute in a fraction of the time by smartly parallelizing the individual tests and running them on individual *cores* (not hosts). Tanay will outline the approach and tools to achieve these results with Selenium, and will present a live demonstration.