Infinite Improbability Testing- Execute All Tests in Parallel

schedule Jun 30th 11:30 AM - 12:15 PM place Kalinga Hall 2 people 8 Attending

100000 tests executed under half an hour - sounds like a myth? Well, we made it, and I'm going to tell you a story how we got there. Through surveys, we discovered that many companies desire to utilize the benefits from parallel tests execution to optimize their software development process. However, they struggle with the process. Lack of available tooling, documentation, tests data arrangement/deletion, handling E2E tests specifics like browsers, emulators, etc. We were one of this companies. Because of that, we created an open-source tool for the job. Throughout the presentation, you will find statistics where, depending on the type of the tests, the tests execution can speed up from 4- 40 times which makes the run of the 100000 tests possible for 27 minutes.

It will be shown how to utilize the tool, its various features and where/when it is appropriate to use it. Also, you will find example solutions to most of the common challenges in executing tests in parallel. We believe that in the near future the parallel tests execution will be a necessity, much like unit tests or continuous integration now. This will be one of the pillars for the companies to improve their competitiveness and effectiveness.

 
6 favorite thumb_down thumb_up 6 comments visibility_off  Remove from Watchlist visibility  Add to Watchlist
 

Outline/structure of the Session

The presentation includes state-of-the-art slides mixed with lots of research such as our surveys and industry ones. Also, there is a balance between theory and practice. Short plan of the talk:

How people usually run their 'heavy' tests?

What is a parallel tests execution?

Horizontal vs Vertical Scaling

Advantages

The Meissa Project

What Is Meissa?

Meissa Features

How Do We Use It?

Why We Built It? Comparison with Existing Tools

How It Works Internally?

Challenges Running Tests in Parallel And Their Sample Solutions

When Is It Appropriate to Use and When Not?

Go Even Further- The Biggest Experiment

Learning Outcome

Key Takeaways:

1. how to optimize/speed up tests execution through parallelization

2. common problems/challenges in running tests in parallel

3. sample solutions to the challenges

4. how to stay in the game of CD

Target Audience

everyone who writes coded tests

Prerequisite

Beginner knowledge of writing coded tests with WebDriver or similar framework. Basic knowledge of CI tools and related processes.

schedule Submitted 3 months ago

Comments Subscribe to Comments

comment Comment on this Submission
  • Andrew Krug
    By Andrew Krug  ~  1 month ago
    reply Reply

    Hi Anton 

    I have some concerns about misleading information that is in your presentation regarding the comparative tables(slides 43-46). Can you correct or respond to the following concern?

    Selenoid and Selenium can run tests in parallel if the test runner is capable of parallel tests. Whereas the table suggests that they don't support it at all. And the CI tools typically can run tests in parallel by selecting a configuration option, so I am not sure by what you mean there is additional effort to do it.

    If there was more clarity on those slides by indicating that some of the parallelizations are dependant on the test runner, not on the tool itself.

    I do not want misleading information to be distributed at a Selenium Conference because that would undermine the integrity of the conference.

    • Anton Angelov
      By Anton Angelov  ~  1 month ago
      reply Reply

      Hi Andrew,

      Thank you for the feedback!

      I am not sure whether you read the speaker notes on the mentioned slides.
      I explain there that it is possible to run the tests in parallel with Selenoid and Selenium, but it is up to the test runner. Their purpose is different. I want to make clear that these tools are not runners. The same is valid for the slide for CI tools. For example, in Jenkins, it is possible to execute the tests in parallel, and I give an example how to do it, but the tests are not executed optimally- like the balancing is manual- using attributes/annotations. There are commercial CI tools that can run tests natively in parallel, but I wanted to focus on free/open-source tools.
      If you don't have time to search into the presentation, I can send you the speaker notes of these slides to check my explanations.
      I am open for suggestions what to change in the tables if you think they are misleading. The last thing that I want is to mislead the audience.

      • Andrew Krug
        By Andrew Krug  ~  1 month ago
        reply Reply

        I know it is what you would be talking about, but as soon as a large graphic shows up the audience will be reading directly from it, not necessarily listening to you.

         

         

         

        • Anton Angelov
          By Anton Angelov  ~  1 month ago
          reply Reply

          This is a valid point.
          I don't think it is correct either to add a check sign.
          What about adding a check sign with *, like the way authors in the books add them when they want to make a note about something subtle?

  • Pooja Shah
    By Pooja Shah  ~  2 months ago
    reply Reply

    Hi Anton,

    "100000 tests executed under half an hour" and open-source, sounds very attractive, thanks for proposing this. Can you please help with a few quick confirmations on:

    1. Are these numbers (in 30 minutes) for UI End-End tests on unit tests? ( the time difference between both is definitely huge and not comparable)

    2. How is The Meissa Project solving more problems at present than the TestNG, JunitUnit runners,  apart from adding the language support in future in Meissa?

     

     


    • Liked Marcus Merrell
      keyboard_arrow_down

      Marcus Merrell / ManojKumar - Selenium Grid

      Marcus Merrell
      Marcus Merrell
      Engineering Manager, CRM
      RetailMeNot, inc
      ManojKumar
      ManojKumar
      Sr. Technical Consultant
      Applitools
      schedule 3 months ago
      Sold Out!
      480 Mins
      Workshop
      Advanced

      Selenium Grid can be a bit daunting to get up and running. Starting it is quite easy, but using it effectively can require pulling in third-party tools. In this workshop we’ll cover how to run your grid effectively, using best practices culled from several large grid installations.

    • Liked Diego Molina
      keyboard_arrow_down

      Diego Molina - The Holy Trinity of UI Testing

      45 Mins
      Talk
      Intermediate

      Sometimes it is hard to know what to test in a web application, and the first step before testing is defining what we want to test. This may sound trivial, but in reality this is often not done properly. We tend to oversee the obvious and we test without knowing what we want to accomplish.

      What do we want to achieve? Validate user behaviour? Check if the page design is responsive on different devices? Or maybe to know that our web application looks like we expect.

      When we know the purpose of our test, we can start planning, coding, executing and improving our tests. But most importantly, we will know what approach we can use to develop the test.

      Functional, layout and visual testing are the three pillars of the UI testing trinity. We can use these approaches to develop focused tests, tests that are asserting a specific aspect of our web application.

      But how can we identify what approach to use? When should we combine them? There is an information overflow that presents a huge variety of tools that can help us to test through any of these approaches. Sadly, this large amount of information is making us focus more on the tools instead of focusing on the testing strategy.

      The intention of this talk is to break in pieces the process of identifying how to develop a focused test, and more importantly, to understand when it makes sense to combine functional testing with layout or visual testing, and what to consider before using layout or visual testing.

      The talk will then go deeper through scenarios and code examples that show how to create layout and visual tests. It will also discuss scenarios where a functional test is not enough, or where a visual test is better than a layout test. This talk’s main goal is to offer a different perspective when testing a web application through the UI testing trinity.

      If you are interested in how to integrate layout or visual testing to your current workflow, you should attend this talk!

      Note: Thanks to the feedback I got after presenting this topic at SauceCon 2018, I have been able to make nice improvements to the content that will be helpful for the attendants.

    • Liked Kushma Thapa
      keyboard_arrow_down

      Kushma Thapa - Running Automation tests twice as faster: How we got rid of Selenium Grid dependency.

      45 Mins
      Talk
      Intermediate

      With the rapid scaling of our health care application under consideration, in terms of number of users and the functionality involved as well as migration from a legacy system, the need to increase the coverage of automated tests for both systems seemed vital. Also, running a large no of test suites on daily basis against multiple browsers was critical.

      As a part of this transition, the existing popular concept of using a continuous integration server with a grid machine and individual node machines poised issues in terms of cost as well as performance. Increase of automated test led to the increase in demand of node machines distributed across different machines on the same/different network. While the network latency for a limited number of node machines could be ignored, it eventually caused a significant reduction in performance.

      Being part of a large enterprise, changing the architecture was not a feasible solution. In this talk we will talk about how selenium grid was removed from the equation to achieve automated tests that ran twice as faster and how it can be integrated on enterprise level requirement.

    • Liked Rajdeep
      keyboard_arrow_down

      Rajdeep - Android Application Backdoor via Appium

      Rajdeep
      Rajdeep
      Sr. Test Automation Engineer
      Badoo
      schedule 3 months ago
      Sold Out!
      45 Mins
      Demonstration
      Advanced

      Application Backdoor via Appium

      There's a shift towards open-source mobile test automation tools happening today among developers and QAs. Whether it be Appium, Calabash or anything else: all are good, with some major limitations.

      While a chosen tool may work well when you first start using it, things can quickly get out of hand with changing business requirements. We started using Calabash at Badoo when there was no Appium. Given the capability of Appium to drive the whole device, we started automation of new apps with Appium. However, we realized a powerful feature was missing in Appium for Android! : The ability to call Application code from automation code like Calabash Backdoors.

      As Appium UiAutomator server is based on instrumentation, we modified it such that we could instrument our app under test. This gave us the power to access context of Application under test and invoke public methods of Activity using reflection APIs. We use these methods to setup app state, seed DB OR even enable/disable some client-side A/B tests. This makes our application more testable and our tests more predictable.

      This talk is going to be about how I achieved the above solution and benefits of backdoors.

      There will be a small demo and code!

    • Liked Gil Tayar
      keyboard_arrow_down

      Gil Tayar - Not Only Cars: “AI, Please Test My App”

      Gil Tayar
      Gil Tayar
      Sr. Architect
      Applitools
      schedule 3 months ago
      Sold Out!
      45 Mins
      Talk
      Beginner

      Autonomous cars were a Scifi dream not 10 years ago. A computer driving a car? No way. But it did happen, and is happening. And if scientists do it for a complicated task such as driving, can they do it for automated regression testing? In this talk we explore what is being done in the field today, but also speculate about the future: we introduce the 6 levels of autonomous testing (that correspond to the 5 levels of autonomous driving), and try and figure out what kind of help current AI techniques can bring to automated testing.

    • Liked Gaurav Tiwari
      keyboard_arrow_down

      Gaurav Tiwari - Including voice command related tests to your automation Suite of Selenium/Appium test

      Gaurav Tiwari
      Gaurav Tiwari
      SDET
      Amazon
      schedule 2 months ago
      Sold Out!
      45 Mins
      Talk
      Beginner

      Almost every app (Web/Mobile/Desktop) is now being powered with voice search, voice typing or any other voice related action. Some great example are Alexa search in Amazon Music app, Google Voice search. We also often have some voice related feature in our web/mobile app. But we usually manually test these scenario.

      Now its time when we start adding these test as our automation sprint Definition of Done. During this talk, I will be going through some sample codes and frameworks and would be demonstrating how you can integrate these tests along with your existing Selenium/Appium test scripts.

      I would also be discussing way forward to integrate Artificial Intelligence to your automation frameworks.

    • Liked NaliniKanth M
      keyboard_arrow_down

      NaliniKanth M / Srinivasan Sekar - New hope in Selenium Docker

      45 Mins
      Demonstration
      Advanced

      Idea behind this talk is to describe how to create a short lived containers for each test and scale to large Selenium Cluster using Selenoid. What is standard Selenium architecture and why it is not suitable for big clusters, Also states issues in maintaining a large Docker Selenium Grid / Selenium Clusters. In this session we will also be talking about the challenges we faced in using official selenium docker images and how we scaled up regression execution time from 3:3 (3 hours to 3 mins). Followed by a demo of how Selenoid server solves the problem with docker. How it works, how to use inside big Selenium cluster, where else it could be used.

    • Liked Syam Sasi
      keyboard_arrow_down

      Syam Sasi / Jerry Zhao - How to build a device lab in your office in 48 hours!

      45 Mins
      Talk
      Intermediate

      Continuous testing is an integral part of continuous delivery pipeline. When it comes to mobile application, the testing become increasingly complex.

      As part of our internal quarter hackathon at Carousell, we developed an automation testing framework and device lab in 48 hours which has both parallel and distributed mode of running.

      We will share about how you can set up a similar device lab in your organization and the best practices to be followed.

    • Liked Raj Thapa
      keyboard_arrow_down

      Raj Thapa - 100% automated, customized and continuously integrated performance test with result analysis and reporting

      Raj Thapa
      Raj Thapa
      Software Engineer
      Verscend Technologies
      schedule 2 months ago
      Sold Out!
      45 Mins
      Talk
      Intermediate

      While performance tests are common among web applications to test their speed, scalability and reliability, a systematic approach to conduct dynamic, automated and customizable automated tests along with fully automated results reporting and display can be regarded as a major challenge.

      In this session, discussions on an approach to conduct about the aforementioned with zero manual intervention will be carried out. The result being a fully automated test strategy that can easily be outlined to Product owners and higher-level management.

      The discussion will be focused on the overall process that initiates with the execution of performance tests using Continuous Integration Server whose results are extracted, logged and used for performance analysis, monitoring and baselining.

      The results are reported in a custom dashboard comprising of features that compare every build with historical data, compare deviations and provide a concrete idea of standard performances regarding the response time of the server over a period. A separate aggregated high-level report is also sent automatically as an email to respective stakeholders.

    • Haritha Hari
      Haritha Hari
      Sr. Quality Analyst
      ThoughtWorks
      schedule 2 months ago
      Sold Out!
      45 Mins
      Demonstration
      Intermediate

      Docker can be used to containerize Selenium tests and can be made to run against applications in another container. These tests can be further run on Selenium grid without the need of physical machines or VMs using Docker Selenium. Multiple and different versions of browsers, headless mode etc can be run inside containers and can be used to run tests in parallel. All these can be achieved with minimal changes to the existing Selenium tests.

    • Liked NaliniKanth M
      keyboard_arrow_down

      NaliniKanth M / Prasad Mudedla - Continuous Security with Selenium tests

      45 Mins
      Demonstration
      Intermediate

      We have come to a place where we run our selenium tests on pipelines every day.
      Now are these tests confined to test only the functionality of the application?
      No, we can use this tests to do much more. Let's test security with those same tests.

      Turn your selenium tests in to Security tests, get security testing in to your CI pipelines just by tweaking a bit of the selenium integration suite. Find and report security issues that exists which can be caught way before paying off for it.
      In this Demo we will introduce a framework that will help your selenium tests turn into security tests and run on pipelines.
      With ZAP API + selenium and CI pipelines, find security issues on the application see a red or green security pipeline. Not just that fetch the reports and take necessary actions on every build.
      The talk covers different ways that ZAP tests the web application for security(automated with selenium tests). We will see what all of them can be integrated with selenium and can make most out the suite.
      As an add we will see how your API can be tested for security using ZAP API :)

      Tools used:
      1. Selenium Web driver framework.
      2. ZAP framework integrated with selenium framework.
      3. GO (for CI/CD)
      4. Maven for building and fetching reports.

    • Liked Michael Palotas
      keyboard_arrow_down

      Michael Palotas - Enterprise Automation with Selenium - and how it (mostly) has little to do with Selenium itself

      45 Mins
      Talk
      Intermediate

      An increasing number of enterprises are moving to Selenium for their GUI automation. The focus for most teams is the creation and authoring of automated tests. When automation projects become difficult to maintain or fail all together, people often point the finger to “Selenium”. From experience, Selenium is usually not the problem in getting an enterprise grade test automation solution off the ground.

      When looking deeper, the absence of basic software development approaches / best practices and the lack of a solid approach to building / buying and managing the cross browser test execution infrastructure are the main reasons for failure.

      This talk showcases:

      • Practical examples of how test automation with Selenium is a software development and infrastructure project, which needs to be treated and staffed as such.
      • The major pitfalls, which can prevent teams from building a scalable and reliable automation solution with the Selenium tool family.
      • How to apply patterns and approaches in making test automation with Selenium a full success.
    • Liked Shashank Chaturvedi
      keyboard_arrow_down

      Shashank Chaturvedi - Data Analytics in QA using PowerBI

      45 Mins
      Demonstration
      Intermediate

      In CI/CD world, we execute thousands of test cases, multiple times in a day. These tests generate tons of data, which can be extremely useful to generate insight on identifying recurring patterns of failures, hotspots, infra issues etc.

      In this talk, I'll share how we can reduce the results analyzing time, but also how we can extract meaningful information from all the historic test runs and provide in-depth insights using PowerBI Visualization and Analytics.

      Few Examples Below:

      Delta Analysis

      Error Analysis

    • Liked Dharmender Kumar
      keyboard_arrow_down

      Dharmender Kumar - Develop Securely with Zaplenium

      Dharmender Kumar
      Dharmender Kumar
      Architect
      JDA
      schedule 3 months ago
      Sold Out!
      45 Mins
      Talk
      Intermediate

      Zaplenium simply means ZAP + Selenium and named after the integration of both to ensure application security while it's being developed. ZAP provide client API's that help to capture URL's by spidering and scanning over the pages those are interacted in user flows. With ZAPlenium the tests can be targeted to certain areas of application where selenium helps in navigating to those areas and target ZAP to spider over those and scan them. Also, Zaplenium will help defining the assertions those can come handy while we integrate this in CI/CD pipelines.

      Participants will be able to define security sanity suite to their build pipelines. With code change the security can be tapped for any High, Medium and Low risks. Session will make use of open source technologies i.e. OWASP ZAP and Selenium to define tests with assertion.

      Security Tool: OWASP ZAP

      Functional Tool: Selenium Web driver

    • Liked Jim Holmes
      keyboard_arrow_down

      Jim Holmes - Five Key Tips To Write Better WebDriver Scripts

      Jim Holmes
      Jim Holmes
      Executive Consultant
      Guidepost Systems
      schedule 3 months ago
      Sold Out!
      90 Mins
      Workshop
      Beginner

      Anyone can learn to write WebDriver scripts to help with automation and testing. Even team members with no coding experience can learn to help contribute value to their teams, and learn in fairly short order!

      This session is targeted to help team members write WebDriver code that's accurate, understandable, and maintainable. Learning these tips will help you and your teams write stable WebDriver scripts that won't require constant tweaking and maintenance.

      Attendees will see practical demonstrations of concepts including the Page Object Pattern, good Locator strategy, dealing with asynchronous actions, and writing custom APIs (also called "backing APIs") to help facilitate better testing.

      Additionally, we'll discuss why it's so important to team testers up with software developers in order to get the best out of testable systems.

      Examples will be in C#, but Ruby and Java will also be discussed—or any other platforms the attendees ask about.

    • Liked Manjyot Singh
      keyboard_arrow_down

      Manjyot Singh - QAOPS - QA Testing in a DevOps World

      Manjyot Singh
      Manjyot Singh
      Lead QA Consultant
      Thought works
      schedule 2 months ago
      Sold Out!
      45 Mins
      Demonstration
      Intermediate

      QAOps is Continuous Testing Strategies when
      frequent software delivery matters.

      It is no surprise that automation and orchestration
      make life very easy, be it a small organization or
      a large-scale industry that houses hundreds of
      servers. IT automation is essentially the ability to
      orchestrate and integrate tools,
      people, and processes through a certain workflow. I will be using Ansible and Docker to showcase the same and writing ansible tests on top of that.

      Ansible - Ansible is an IT automation tool which can configure systems, deploy software, and orchestrate more advanced IT tasks such as continuous deployments. Ansible is actually designed to be a “fail-fast” and ordered system, therefore it makes it easy to embed testing directly in Ansible playbooks.

      Docker - A docker container is a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings. Containers isolate software from its surroundings, for example differences between development and staging environments and help reduce conflicts between teams running different software on the same infrastructure.