How to run 2500 selenium tests in an hour.

If you are tired of waiting for your selenium tests status and old fasioned ways of reporting tests status, you are in a good place.

We will show you our test environment setup and how we run over 2500 tests in about an hour. Not cool enough? Take a look how we make reports via slack.

Do you want to check if we can be even more faster?

 
 

Outline/Structure of the Case Study

- Process presentation

- AWS setup

- Jenkins setup

- Slack reporting

- Why not faster?

Learning Outcome

Big number of selenium tests doesn't mean it will take hours to perform.

Target Audience

Everyone who want to run selenium tests faster

Prerequisites for Attendees

Basic understanding of AWS, Jenkins and Hubot.

schedule Submitted 10 months ago

Public Feedback

comment Suggest improvements to the Author
  • Deepti Tomar
    By Deepti Tomar  ~  7 months ago
    reply Reply

    Hello Tomasz,

    Thanks for your proposal!

    Could you please update the Outline/Structure section of your proposal with a time-wise breakup of how you plan to use 45 mins for the topics you've highlighted?

    Also, To help the program committee understand your presentation style, can you provide a link to your past recording or record a small 1-2 mins trailer of your talk and share the link to the same? 
     
    Thanks,

  • Robin Gupta
    By Robin Gupta  ~  7 months ago
    reply Reply

    Outline of the Case study does not cover the exact problem statement, alternative solution paths and a crisp conclusion. Also, can you please add more details on the costing of the setup from an operational perspective, as the bill on AWS platform can rise exponentially, if we dont pay attention to it.

    • Tomasz Wojciechowski
      By Tomasz Wojciechowski  ~  7 months ago
      reply Reply

      Hi Robin, these are fair points and will be covered in the presentation.

      Thank you for your feedback! :) 

      • Anand Bagmar
        By Anand Bagmar  ~  7 months ago
        reply Reply

        Hi Tomasz, Would help the review team if you can share some details on Robin's comment

  • Maaret Pyhajarvi
    By Maaret Pyhajarvi  ~  7 months ago
    reply Reply

    I find that 2 person talks are 10x work to make good and require a special connection between the presenters connected to the topic. Why does this topic require 2 speakers?

    • Tomasz Wojciechowski
      By Tomasz Wojciechowski  ~  7 months ago
      reply Reply

      We work in one team, sit almost next to each other and cooperate for 4 years, I don't expect much connection issues. :)

      Tomasz is our infrastructure master, he set up most of this process and tools from a technical perspective. I'm more of a "direction guy" who works on the vision. We complement each other. 


  • Liked Krishnan Mahadevan
    keyboard_arrow_down

    Krishnan Mahadevan - My experiments with Grid

    45 Mins
    Tutorial
    Intermediate

    Everyone starts off with a simple grid setup which involves a hub and one or more nodes.

    This traditional setup is a good start but the moment one starts to get serious with the selenium grid and decide to house their own selenium grid for their local executions, that is when issues start.

    My experiences with the Selenium grid in the past couple of years has led me to get introduced some of the most prevalent problems with maintaining an in-house selenium grid.

    • Nodes get unhooked randomly due to network glitches.
    • Nodes introduce false failures due to memory leaks.
    • Selenium Grid running out of capacity.
    • Nodes require OS upgrades/patches etc.
    • Needing to deal with auto upgrades by browsers (especially chrome and firefox)

    Some of these issues I managed to fix by building a "Self Healing" Grid wherein the nodes automatically get restarted after they have serviced "n" tests. But that still didn’t solve many of these other problems.

    That was when I felt, what if there was an on-demand selenium grid.

    What if the Grid could do the following ?

    • The Grid auto scales itself in terms of the nodes based on the current load.
    • The Grid does not require a lot of infrastructure to support it.
    • The Grid can plug itself into some of the cloud providers or leverage a solution such as Docker so that the nodes can be spun and shutdown at will.

    That was how the idea of "Just Ask" an on-demand grid was born.

    Just-Ask is an on-demand grid. It has no nodes attached to it.

    It’s designed to spin off nodes on demand, run test against the newly spun off node and after test runs to completion, clean-up the node as well. The node can be backed by anything. It could be Docker (or) it could be a VM running on any of the popular clouds.

    The session aspires to walk the audience through with my experiments with the selenium grid, my learnings on the selenium grid internals and how I used all of that knowledge to build my own On Demand Selenium Grid. What better avenue to share these learnings than a Selenium Conference.

    The session will introduce the audience to the grid internals and their concepts such as

    • What is a Selenium Remote Proxy ? What is it used for? What can you do with it?
    • What is a Hub (or) Node level Servlet ? When would you need one ?
    • All of this followed by a quick demo on "Just Ask", the on-demand grid that I have built and open sourced here: https://github.com/rationaleEmotions/just-ask

  • Liked Tomasz Konieczny
    keyboard_arrow_down

    Tomasz Konieczny - Serverless - how to speed up tests over 300 times and achieve continuous feedback?

    Tomasz Konieczny
    Tomasz Konieczny
    Senior QA Engineer
    Ro
    schedule 8 months ago
    Sold Out!
    45 Mins
    Talk
    Intermediate

    Automated tests can provide results faster and it’s possible to execute them more frequently than manual ones. That allows to test earlier in the development process, decrease overall time needed for tests and what is probably the most important it’s possible to release and deliver business value faster and more frequently.

    But what if we have more and more tests and even automated execution of them takes too much time - 10 minutes... 30 minutes... maybe even hours? Should we consider the ability to execute full tests set just a few times a day as something normal? Is adding more compute resources the only option to reduce the execution time? Or maybe there are too many high-level tests and some of them should be replaced by low-level ones according to tests pyramid? Is the tests pyramid still valid in the cloud world?

    During the presentation you will see how the serverless cloud services like AWS Lambda may be used to run tests in the highly parallelized environment that can speed up execution even hundreds of times.

  • Liked Deepak Koul
    keyboard_arrow_down

    Deepak Koul - Is quality really everyone's responsibility - The Quality Accountability conundrum

    Deepak Koul
    Deepak Koul
    Manager , Quality Engineering
    Red Hat
    schedule 10 months ago
    Sold Out!
    20 Mins
    Talk
    Beginner

    "Quality is everyone's responsibility" has to be one of those phrases which looks great on t-shirts and posters but when put into action, often fails.
    There is a funny leadership story which goes like this " There was an important job to be done and Everybody was sure that Somebody would do it. Anybody could have done it, but Nobody did it"
    Therefore assuming that everyone in your team (developers, designers, QEs or analysts) would be instinctively inclined to incorporating quality in their day to day work is a terrible assumption to make.
    Quality is everyone's responsibility might be true but it cannot work in isolation, it has to be supplemented with an accountability framework.
    In this talk, I am going to present exactly that framework and help people especially leaders - technical and people, implement proactive quality processes in their teams.
    The information presented in this talk does not require audience to have any technical knowledge and applies to all the roles.

  • Liked Christian Bromann
    keyboard_arrow_down

    Christian Bromann - The Nuts and Bolts of WebdriverIO

    Christian Bromann
    Christian Bromann
    Software Engineer
    Sauce Labs
    schedule 8 months ago
    Sold Out!
    90 Mins
    Workshop
    Beginner

    There are thousands ways if not more to setup your automation testing environment. It is often crucial when it comes to stability and maintainability. While you can build a custom framework that fits your needs there are often already sophisticated setups or frameworks available that get you up and running very quickly.

    WebdriverIO is one of these frameworks that is written in Node.js. It gets you up and running within minutes and allows you to scale up your test suite while maintaining your execution time by running tests in parallel. With its huge project community it is an ideal choice for small as well as big projects that use modern frameworks such as React, Angular or Vue.js.

    In this workshop you will learn everything you need to know to run a successful, stable and maintainable WebdriverIO framework. It starts with an introduction to the project and the basic concepts and will end with a sophisticated framework that includes testing strategies like Frontend Performance Testing as well as complex browser interaction with Puppeteer.

  • Liked Srinivasu gangam
    keyboard_arrow_down

    Srinivasu gangam - Zero Touch Automation using NLP (Natural language processing) & AI

    Srinivasu gangam
    Srinivasu gangam
    Sr Manager
    Cotiviti India
    schedule 10 months ago
    Sold Out!
    45 Mins
    Demonstration
    Intermediate

    Problem Statement:

    As part of SDLC process:

    1. Is your product quality impacted due to a smaller number of QA resources available in the team?
    2. Are you waiting for QA resources to certify your code every time when you deploy? Is this impacting your product lead time (Speed to Market)?
    3. Is your Product delivery timelines are impacted due to last minute defects identified?
    4. Do you have your QA resources only in one location, but you want to “follow-the-sun” approach for Software delivery across multiple locations?
    5. Do you have manual testers who are not skilled in programming, but you want them to execute automated test scripts w/o any training efforts and automation setup?
    6. Would you like your team more agile and cross functional with Delivery?
    7. Would you like to increase your QA team’s productivity while they invest more time in script development rather than script execution?

    If answer is ‘Yes’ for above questions, "Zero touch automation" is the solution for above challenges that we have been facing part of SDLC.

    Solution: Zero touch automation with cutting-edge technologies

    In this session, I will cover how we solved this problem using innovative solutions, Cutting-edge technologies like NLP (Natural language processing), AI & Cloud solutions.

    You will learn how AI, NLP integrated with core automation components to achieve Zero touch automation.

    This solution is not just revolutionary, it is paradigm shift in test automation to get results to your email with detailed analysis of failure categorization with recommended actions to users.

    I will also cover how E2E automation will be driven with decisions taken by machines based on what user is looking for . There is no manual intervention in this process. NLP and AI play key role to help machines to take decisions.

    We will also cover how we empowered developer/release manager/any team member/Manager to trigger the scripts from their cell phone and get the detailed execution report without having any automation software installed in their computer or Phone.

    We will be demonstrating how the request will be initiated from User, understand the need from user using NLP & AI , Fetching the code from bitbucket to select appropriate automation scripts , running them on Selenoid/docker server , storing results to MongoDB , receiving email with test results and Failure analysis.

    What is the value of zero touch automation?

    1. Enable speed to market: Now that Developers does not need to wait for QA resource, Changes can be certified quickly and ready to push to production. Lead time will be significantly reduced.
    2. Increase quality: Now that test automation is easy and it can run multiple times in each environment, most of the defects will be uncovered and addressed before code goes to production.
    3. Ease of test execution: Test execution will be very easy, no automation or framework setup required from user side. Test execution can be done 24*7.
    4. Productivity: Increase QA team’s Productivity to focus more on script development rather than focusing on script execution and failure analysis .
  • Liked Naveen Khunteta
    keyboard_arrow_down

    Naveen Khunteta - Best Practices to implement the test automation framework starting from Design - To -> Infrastructure - To -> Execution.

    45 Mins
    Talk
    Intermediate

    Best Practices - How to get the best 'Return ON Investment' (ROI) from your Test Automation.

    This has been observed that, most of the test frameworks wont be able to survive due to lack of expertise, no maintenance, no best practices being followed, and finally your test automation will be dead after few months, and there is no "Return ON Investment" from this. This is the most common problem, most of the companies are struggling and finally back to square to the Manual testing.

    My proposal : HOW to leverage your test automation in terms of best practices, best ROI, and how to adopt best automation culture in your organisation.
    I strongly propose some of the important points/suggestions to achieve this in your Organisation/Team.
    1. Test Automation Practices:
    • Design Patterns (Web/Mobile/API)
    • What to Automate/Not to Automate
    2. Common Automation Frameworks at Org Level:
    • How to design Generic Utilities, Libraries and different Components, which can be suitable for all the teams in the same Org.
    • Best practices to design your Tests (Automation).
      • Common Design Patterns
      • Common application level and Page libraries
      • Best Practices to use Assertions in your Tests (How and What to write for assertions). Most of the people don't write proper assertions and this is making your test unreliable and no defects found during execution.
    3. Inclusion of API/Backend libraries in your UI test automation as an external Maven/Gradle Dependencies to avoid un-necessary tasks, some of the important points to be considered here:
    • User Creation from APIs (No need to automate user creation from web/app for all the test cases)
    • API tests are stabled most of the time
    • API calls takes lesser time as compared to web, hence include API calls in your UI/App framework to save time.
    • Less flaky test

    4. Best Code Review Process (Do not merge your code into Master without proper Code Review)

    • Implement PR (PULL Request) Process
    • Static Code Analysis using SonarQube, Cobertura, JACOCO etc..
    • Get the benefits of Best Test Automation Quality Matrices
    • Sometimes, Manual (Functional Tester) should review your code (Assertion, test steps and use cases) to get the best coverage
    5. Quality is A Team responsibility:
    • Developers, POs, Manual QEs and Automation engineers should be included to get an overview of test automation coverage.
    6. Maintenance of the Frameworks
    • After couple of months, it should not make your life miserable if you don't maintain your libraries and framework properly.
    • Do not use Hard Coded values, make it simple and Generic.
    7. Infrastructure Setup for Test Design and Test Execution:
    • Proper Browser - OS lab setup
    • Proper Mobile Labs setup with different Devices - IOT, iOS, Android, iPad, Tablets
    • Proper CI - CD common configuration using Jenkins, Dev Ops, AWS, Docker and Cloud setup
    • Handling multiple Docker nodes using Kubernates (use of Selenoid, GRID on Cloud)
  • Liked Wim Selles
    keyboard_arrow_down

    Wim Selles - Do you know the dependency of your dependencies dependency?

    Wim Selles
    Wim Selles
    Solution Architect
    Sauce Labs
    schedule 8 months ago
    Sold Out!
    45 Mins
    Case Study
    Beginner

    Nowadays a lot of people are using JavaScript based automation frameworks, and why not? They are the closest language to what your developers are using and it's easy to use. You just

    • Install NodeJS on your local machine
    • npm install this-module
    • npm install that-module,
    • script a test-case
    • npm run test

    And you're done, easy peasy, isn’t it? But when you look at your project you see something called

    • node_modules
    • package.json
    • package-lock.json

    And you hear your developers talk about

    • ES6, ES7
    • TypeScript
    • Polyfills
    • NodeJS LTS, 8, 9, 10, 11, 12, 13 (and counting)
    • Promises, asynchronous behavior
    • ...

    And you are like……………...

    If you recognize yourself in this then this presentation might be useful for you, because during this presentation I will cover NodeJS, the biggest dependency of your dependencies dependency.

  • Liked Benjamin Bischoff
    keyboard_arrow_down

    Benjamin Bischoff - Smoke tests and mirrors - magic principles in test automation

    45 Mins
    Talk
    Beginner

    Having been a hobbyist magician for 34 years, I started noticing that some close-up and stage magic principles and theories don't only apply to magic tricks but also to software testing.

    An example of this is the application of the KISS principle (Keep It Simple, Stupid) in software development. This helps to avoid unnecessary code complexity and make it easy to understandable and follow. In magic, this applies as well. Magicians strive for clarity and simplicity by creating effects that are easy to follow for spectators.

    In this sessions, I will try to connect those two worlds and thus combine two of my passions. Also, I will try my best to illustrate the magic side of things with demonstrations. This is a challenge I set for myself - let’s see how it goes.

  • Liked Tomasz Wojciechowski
    keyboard_arrow_down

    Tomasz Wojciechowski - Quality Assurance is more than testing

    45 Mins
    Talk
    Intermediate

    Have you ever heard "you are just a tester"? Have you ever felt being a tester is just a stop for "better" position? Are you afraid someday test automation will take your job? Do you feel you could bring more value for your team but don't know how?

    Hi, I'm BeardedQA and I have some answers.

    During this talk, I'll share my experience about moving from Manual Tester to Quality Assurance Engineer position, what quality assurance really is and how to give more value for your team.

  • Liked Sameer Arora
    keyboard_arrow_down

    Sameer Arora - Triggering alerts on Slack along with screenshots in case of test case failure

    Sameer Arora
    Sameer Arora
    Senior Software Test Engineer
    Gaana
    schedule 10 months ago
    Sold Out!
    45 Mins
    Demonstration
    Beginner

    One of the biggest limitations with most of the frameworks in selenium is that we need to wait till the end to get the final test case report. Only then we can raise the alert that a particular functionality is broken. By then, it may be too late! In addition to this, in most of the frameworks, the HTML report generated is hosted on our company's private network and we need to connect to the VPN if we are outside the office to open the report and check the related screenshots to see which test case has failed and why.

    So, why not raise an alert as soon as the test case fails? And why not attach a screenshot along with that alert so that all the stakeholders can actually see what has failed in the application?

    In this session, we will learn a simple yet a really useful way of sending an alert along with the failed test case screenshot by integrating slack with selenium which will help us alert all the stakeholders as soon as an automated test case fails so that everyone can react to it immediately and the damage will be minimised.

  • Liked Martin Schneider
    keyboard_arrow_down

    Martin Schneider / Prabhagharan D K - Building and scaling a virtual Android and iOS device lab

    45 Mins
    Case Study
    Intermediate

    Virtual mobile devices (emulators/simulators) are a cost-effective and straightforward alternative to testing on physical devices. We showcase how to set-up and scale an Android emulator farm using Appium, Docker and SQS and how it fits into our larger testing and quality strategy.
    Maintaining physical test devices for mobile automation can be expensive and time-consuming. On top of the initial investment, you need to consider maintenance cost, replacement devices and efforts for manual scaling. On the other side of the spectrum, cloud providers take care of these restrictions, but their services can come at a hefty price tag, especially when your use-case requires a large number of devices. We present a middle path and demonstrate how to use virtual devices to build a reliable and scalable in-house device lab using Docker and Appium.

  • Liked Srinivasu gangam
    keyboard_arrow_down

    Srinivasu gangam - Test case Generator with Optimized coverage - Orthogonal Array Testing Strategy (OATS)

    Srinivasu gangam
    Srinivasu gangam
    Sr Manager
    Cotiviti India
    schedule 8 months ago
    Sold Out!
    45 Mins
    Case Study
    Beginner
    1. Are you Confident about your test coverage?
    2. Do you have millions of permutations and combinations to be covered in testing?
    3. is automation the solution to cover million combinations?
    4. why can not my system decide what test cases to be executed to give complete test coverage?
    5. As part of CI/CD , instead of executing same test cases irrespective changes going in, would you like to execute more appropriate test cases ?

    I will be presenting quick demo on statistical approach and strategy which can help us to generate appropriate number of test cases automatically based on the impacted areas and business metrics . Let me share an example below.

    Let’s say we are testing any eCommerce application Flipcart or amazon. We will have to cover multiple combinations like different categories of items, different payment types, shipping types, Promotions, user types etc.. . I will use little bit of sets theory here.

    Let us take below 5 sets as an example. Assume that there is new payment type is getting introduced as “Paypal” as enhancement. If we need to make sure this new payment type is working with all products, all shipments types , all user types etc… we will have to cover all permutations and combinations which is nothing but Cartesian product of all these sets. The Cartesian product of below 5 sets is - 6*3*2*4*3 =432 test cases.

    Do we really need to execute 432 test cases just for one new payment type ? we just need to execute 24 test case to give confident test coverage .This can be done with OATS tool which is developed with intelligence with Business user Metrics + Orthogonal array testing approach ( OATS) approach . OATS tool will generate required number of test cases with simple input and one click .

    There are 2 key areas of input that we consider before we generate optimized tests.

    1. Business user metrics - Example : if you look at the below payment set , we would like to know what % of customers using each payment type in overall successful transactions. let us assume 60% Credit card payments , 20 % paytm and 20% rest of payment types . These metrics will help tool to give more weightage to Credit card test cases over other payment types. This make sense because we need to focus more on the areas our end users are using.

    2. Impacted areas: let us take an example that there is new payment type (AmazonPay) is getting introduced as en enhancement. We should give more weightage to payment set and make sure all existing payment types are tested with different combinations.

    Set1 - PAYMENT = { Cash, Credit card , Paytm , AmazonPay, Debit card}- # elements - 6

    Set2 – SHIPPING – {Shiptoday, Prime Shipping, ship Tomorrow}-# elements - 3

    Set3 – USER TYPE - { Guest User , Logged in user }-# elements - 2

    Set4 – PRODUCT TYPE – {Electronics, Groceries, Books , Stationary}- # elements - 4

    Set5 – PROMO TYPE – { Promo1 , Promo2 , Promo3}-# elements – 3

    I will be covering how this can be done in detail during presentation along with Tool Demo which was built in with an intelligence using sets theory . you will get an answer on why only 24 test cases are good enough instead of 432 test cases for the change new payment type got introduced.

    Advantages :

    1. Optimized coverage with less test cases

    2. Tester will be confident about test coverage

    3. This tool output can be integrated with Selenium key word driven framework . All this happens during run time w/o manual intervention.

    4. This strategy is good fit for automation testing .

    5. if any combinations are invalid to test, those combinations will be excluded while generating tests.

    6. There are tools in the market to provide statistics about untested code (Ex: JaCoCO ). These statistics can be used as input for OATS tool. We have not done this yet.

    Limitations:

    1. We have seen great value (reduced bug escape) applying this strategy to automated tests. However there is limitation in applying the same strategy to manual exploratory , Adhoc testing.

    2. We have not tried this strategy for performance testing yet.

    3. There are corner scenarios where human can only think will not be covered by this tool . These scenarios to be appended to generated test cases .