Serverless - how to speed up tests over 300 times and achieve continuous feedback?

Automated tests can provide results faster and it’s possible to execute them more frequently than manual ones. That allows to test earlier in the development process, decrease overall time needed for tests and what is probably the most important it’s possible to release and deliver business value faster and more frequently.

But what if we have more and more tests and even automated execution of them takes too much time - 10 minutes... 30 minutes... maybe even hours? Should we consider the ability to execute full tests set just a few times a day as something normal? Is adding more compute resources the only option to reduce the execution time? Or maybe there are too many high-level tests and some of them should be replaced by low-level ones according to tests pyramid? Is the tests pyramid still valid in the cloud world?

During the presentation you will see how the serverless cloud services like AWS Lambda may be used to run tests in the highly parallelized environment that can speed up execution even hundreds of times.

 
 

Outline/Structure of the Talk

- Introduction
- Advantages of Automated Tests
- Problems
- Possible workarounds
- Case study: 500 E2E Selenium tests, 20 seconds each
- Parallel test execution (AWS Lambda) - implementation
- Summary

Learning Outcome

How to speed up test execution even hundreds of times with serverless cloud services like AWS Lambda?

Target Audience

People related to Test Automation

schedule Submitted 7 months ago

Public Feedback

comment Suggest improvements to the Speaker
  • Pallavi ...
    By Pallavi ...  ~  5 months ago
    reply Reply

    hello tomasz

    is this implementation of parallel test execution going to showcase selenium grid usage as well for parallel test execution on cloud? 

    can you please elaborate more on the outline of your talk with proper division in terms of minutes for each topic. 

    the implementation which you will be showing , can it be extrapolated by any one, as in would its code be available in git to be reused 

    please let me know

    • Tomasz Konieczny
      By Tomasz Konieczny  ~  5 months ago
      reply Reply

      Hello,

      I'm not using Selenium Grid in this case - custom scripts are used instead.

      According to the outlines - I can show you the presentation. I don't have final slides yet. However, I presented a similar talk at two conferences (PyCode 2019 and TestWarez 2019). That talk was based on Proof of Concept I implemented. I reduced the execution time of sample Test Set (500 Selenium tests, 20 seconds each) from almost 3 hours to about 30 seconds with AWS Lambda. You can check it here: https://docs.google.com/presentation/d/1ZIwf6lF8TThWaEomf5SH316zIkum2WrrFFSRMw5_Qog/edit?usp=sharing

      This presentation will be updated based on experience I gained during the development of something more complete than just PoC - a tool that is already used on production. I will show the complete architecture (AWS Lambda, DynamoDB, CloudWatch, S3), describe how it works and how to build a complete tool. I will speak about the problems that appeared (both expected and unexpected) and how I solved them. For example how to deploy and run recent Chrome browser versions that exceed Lambda Layer limits or how to run tests longer than 30 seconds (AWS API Gateway max timeout). I will also speak about continuous feedback and how a significant reduction of test execution time finally allowed to achieve it.

      The code from the presentation can be easily reused.

      • Pallavi ...
        By Pallavi ...  ~  5 months ago
        reply Reply

        Okay, could you please help me understand why Selenium Grid was not used in here? Just curious about it. 

        • Tomasz Konieczny
          By Tomasz Konieczny  ~  5 months ago
          reply Reply

          Earlier, when I used Kubernetes cluster for test execution I also used Selenium Grid. In this case architecture is quite different and Selenium Grid wasn't used because it would be just an additional layer. It would require more complex infrastructure and in this case using it won't have any additional benefits. AWS Lambda functions are stateless so it was better to trigger them directly from runner.

          • Pallavi ...
            By Pallavi ...  ~  5 months ago
            reply Reply

            Okay, I understand although, partially. neverthless thank you for your revert. 

  • Robin Gupta
    By Robin Gupta  ~  6 months ago
    reply Reply

    I like these aspects of the submission, and they should be retained:

    • DevOps component of the Automation pipeline

    I think the submission could be improved by:

    • Adding the cost implications of running a huge battery of tests in the AWS cloud
    • Is this talk focused towards a specific language - Python or Java?
    • Limitations of the approach vis a vis horizontal scaling, browser/OS version, potability etc..
    • Tomasz Konieczny
      By Tomasz Konieczny  ~  6 months ago
      reply Reply
      • For this example set (500 tests, 20 seconds each) execution cost will be about $0.18 per run. I think it's very low especially compared to the overall development costs, the cost of the team waiting for test results etc.
      • I implemented it both in Python and JavaScript/Node. Examples will be shown in one ot these languages (probably Python) but my goal is to show the concept that's technology-independent.
      • I'm using latest chrome/chromium versions (haven't tried other browsers), OS - AWS Linux 2 (Lambda is based on it). Portability - Infrastructure as Code (Serverless Framework) so it's possible to easily create/modify/deploy whole stack and dependencies (like AWS DynamoDB, AWS S3 etc.).

  • Liked Benjamin Bischoff
    keyboard_arrow_down

    Benjamin Bischoff - Smoke tests and mirrors - magic principles in test automation

    45 Mins
    Talk
    Beginner

    Having been a hobbyist magician for 34 years, I started noticing that some close-up and stage magic principles and theories don't only apply to magic tricks but also to software testing.

    An example of this is the application of the KISS principle (Keep It Simple, Stupid) in software development. This helps to avoid unnecessary code complexity and make it easy to understandable and follow. In magic, this applies as well. Magicians strive for clarity and simplicity by creating effects that are easy to follow for spectators.

    In this sessions, I will try to connect those two worlds and thus combine two of my passions. Also, I will try my best to illustrate the magic side of things with demonstrations. This is a challenge I set for myself - let’s see how it goes.

  • Liked Tomasz Wojciechowski
    keyboard_arrow_down

    Tomasz Wojciechowski / Tomasz Wojtuń - How to run 2500 selenium tests in an hour.

    45 Mins
    Case Study
    Advanced

    If you are tired of waiting for your selenium tests status and old fasioned ways of reporting tests status, you are in a good place.

    We will show you our test environment setup and how we run over 2500 tests in about an hour. Not cool enough? Take a look how we make reports via slack.

    Do you want to check if we can be even more faster?

  • Liked Tomasz Wojciechowski
    keyboard_arrow_down

    Tomasz Wojciechowski - Quality Assurance is more than testing

    45 Mins
    Talk
    Intermediate

    Have you ever heard "you are just a tester"? Have you ever felt being a tester is just a stop for "better" position? Are you afraid someday test automation will take your job? Do you feel you could bring more value for your team but don't know how?

    Hi, I'm BeardedQA and I have some answers.

    During this talk, I'll share my experience about moving from Manual Tester to Quality Assurance Engineer position, what quality assurance really is and how to give more value for your team.