100% automated, customized and continuously integrated performance test with result analysis and reporting

While performance tests are common among web applications to test their speed, scalability and reliability, a systematic approach to conduct dynamic, automated and customizable automated tests along with fully automated results reporting and display can be regarded as a major challenge.

In this session, discussions on an approach to conduct about the aforementioned with zero manual intervention will be carried out. The result being a fully automated test strategy that can easily be outlined to Product owners and higher-level management.

The discussion will be focused on the overall process that initiates with the execution of performance tests using Continuous Integration Server whose results are extracted, logged and used for performance analysis, monitoring and baselining.

The results are reported in a custom dashboard comprising of features that compare every build with historical data, compare deviations and provide a concrete idea of standard performances regarding the response time of the server over a period. A separate aggregated high-level report is also sent automatically as an email to respective stakeholders.

 
 

Outline/Structure of the Talk

  • Performance Tests Overview
  • Execution of tests using Continuous Integration
  • Results Extraction and Logging
  • Results Interpretation and Report Generation
  • Automated aggregate high level reporting to stakeholders
  • Performance Results Monitoring and Analysis

Learning Outcome

1. Full fledged automation of performance tests with reporting and result analysis

Target Audience

Enterprise Level Performance and Load Testers

Prerequisites for Attendees

  • Performance and Load Testing
  • JMeter
  • Build Tools
  • Continuous Integration
  • Results Monitoring and Analysis
schedule Submitted 1 year ago

Public Feedback

comment Suggest improvements to the Speaker
  • Pooja Shah
    By Pooja Shah  ~  1 year ago
    reply Reply

    Hi Raj,

    Thanks for proposing this. While the topic is very useful, doing a 90 minutes workshop instead of talk can be practically useful and a big win for you and the attendees. 

    However, if you still plan to do the Talk then 

    1. Would like have a bit more clarity on how do you plan to explain all mentioned to the attendees with a talk. I mean, what is the seqeunce model like - problem followed by solution or how?

    The more real-time examples, the more practical interaction the better the chances attendees gain from it and in return, you gain the joy of sharing it  

    2. The slides included doesn't look related to this talk, a more interactive or practical based slide/short 1-3 minute video record will help us understand more.

    • Raj Thapa
      By Raj Thapa  ~  1 year ago
      reply Reply

      Hello Pooja,

      Thanks for the feedback. A workshop on the topic does sound interesting but as a newbie, I think for now I will limit myself to talk. Once again thanks for the suggestion. You felt the topic can be considered for workshop already means a lot to me.

      I am planning on starting with a slight introduction of performance testing rapidly moving to JMeter and other tools used such as ant and maven following with the bamboo integration and the results interpretation.

      On addition to that I am intending to share some insights on tools we created for results logging, a summary report generator and mailing system following with a dashboard used in monitoring the trends of response times.

      I will be uploading related slides and videos very shortly.

      • Raj Thapa
        By Raj Thapa  ~  1 year ago
        reply Reply

        I have uploaded the video.


  • Liked Simon Stewart
    keyboard_arrow_down

    Simon Stewart - Fix a Bug, Become a Committer

    Simon Stewart
    Simon Stewart
    WebDriver Creator
    Facebook
    schedule 1 year ago
    Sold Out!
    480 Mins
    Workshop
    Beginner

    Have you ever wondered how Selenium works under the covers? Do you get frustrated with locators not locating, pages not loading, or browsers behaving inconsistently from one run to the next? Selenium is an attempt to unify thousands of disparate elements across a wide spectrum of challenges into a single, common interface that works seamlessly with all the major browsers - and yet only a handful of volunteers work to maintain this gigantic effort. If you would like to enhance your own Selenium experience while contributing back to the software that has defined so many of our careers, come to this workshop. In it we'll dissect the different elements of Selenium, dive into its internals, learn how it was built and how to make changes to it, and even write a unit test you can contribute on the same day!

  • Liked Kushma Thapa
    keyboard_arrow_down

    Kushma Thapa - Running Automation tests twice as faster: How we got rid of Selenium Grid dependency.

    45 Mins
    Talk
    Intermediate

    With the rapid scaling of our health care application under consideration, in terms of number of users and the functionality involved as well as migration from a legacy system, the need to increase the coverage of automated tests for both systems seemed vital. Also, running a large no of test suites on daily basis against multiple browsers was critical.

    As a part of this transition, the existing popular concept of using a continuous integration server with a grid machine and individual node machines poised issues in terms of cost as well as performance. Increase of automated test led to the increase in demand of node machines distributed across different machines on the same/different network. While the network latency for a limited number of node machines could be ignored, it eventually caused a significant reduction in performance.

    Being part of a large enterprise, changing the architecture was not a feasible solution. In this talk we will talk about how selenium grid was removed from the equation to achieve automated tests that ran twice as faster and how it can be integrated on enterprise level requirement.

  • Liked Anton Angelov
    keyboard_arrow_down

    Anton Angelov - Infinite Improbability Testing- Execute All Tests in Parallel

    Anton Angelov
    Anton Angelov
    CTO
    Automate The Planet
    schedule 1 year ago
    Sold Out!
    45 Mins
    Talk
    Intermediate

    100000 tests executed under half an hour - sounds like a myth? Well, we made it, and I'm going to tell you a story how we got there. Through surveys, we discovered that many companies desire to utilize the benefits from parallel tests execution to optimize their software development process. However, they struggle with the process. Lack of available tooling, documentation, tests data arrangement/deletion, handling E2E tests specifics like browsers, emulators, etc. We were one of this companies. Because of that, we created an open-source tool for the job. Throughout the presentation, you will find statistics where, depending on the type of the tests, the tests execution can speed up from 4- 40 times which makes the run of the 100000 tests possible for 27 minutes.

    It will be shown how to utilize the tool, its various features and where/when it is appropriate to use it. Also, you will find example solutions to most of the common challenges in executing tests in parallel. We believe that in the near future the parallel tests execution will be a necessity, much like unit tests or continuous integration now. This will be one of the pillars for the companies to improve their competitiveness and effectiveness.

  • Liked Gaurav Tiwari
    keyboard_arrow_down

    Gaurav Tiwari - Including voice command related tests to your automation Suite of Selenium/Appium test

    45 Mins
    Talk
    Beginner

    Almost every app (Web/Mobile/Desktop) is now being powered with voice search, voice typing or any other voice related action. Some great example are Alexa, Google Voice search. We also often have some voice related feature in our web/mobile app. But we usually manually test these scenario.

    Now its time when we start adding these test as our automation sprint Definition of Done. During this talk, I will be going through some sample codes and frameworks and would be demonstrating how you can integrate these tests along with your existing Selenium/Appium test scripts.

    I would also be discussing way forward to integrate Artificial Intelligence to your automation frameworks.

  • Liked Gil Tayar
    keyboard_arrow_down

    Gil Tayar - Not Only Cars: “AI, Please Test My App”

    Gil Tayar
    Gil Tayar
    Sr. Architect
    Applitools
    schedule 1 year ago
    Sold Out!
    45 Mins
    Talk
    Beginner

    Autonomous cars were a Scifi dream not 10 years ago. A computer driving a car? No way. But it did happen, and is happening. And if scientists do it for a complicated task such as driving, can they do it for automated regression testing? In this talk we explore what is being done in the field today, but also speculate about the future: we introduce the 6 levels of autonomous testing (that correspond to the 5 levels of autonomous driving), and try and figure out what kind of help current AI techniques can bring to automated testing.

  • Liked NaliniKanth M
    keyboard_arrow_down

    NaliniKanth M / Srinivasan Sekar - New hope in Selenium Docker

    45 Mins
    Demonstration
    Advanced

    Idea behind this talk is to describe how to create a short lived containers for each test and scale to large Selenium Cluster using Selenoid. What is standard Selenium architecture and why it is not suitable for big clusters, Also states issues in maintaining a large Docker Selenium Grid / Selenium Clusters. In this session we will also be talking about the challenges we faced in using official selenium docker images and how we scaled up regression execution time from 3:3 (3 hours to 3 mins). Followed by a demo of how Selenoid server solves the problem with docker. How it works, how to use inside big Selenium cluster, where else it could be used.

  • Liked Ruchika Rawat
    keyboard_arrow_down

    Ruchika Rawat - Lean Test Approach - in Agile

    Ruchika Rawat
    Ruchika Rawat
    Quality Analyst
    Thoughtworks
    schedule 1 year ago
    Sold Out!
    20 Mins
    Case Study
    Beginner

    Talk Description Automation provides an immense amount of value in preventing regressions and helping to deliver quality software. As an organization’s automation grows and grows, it requires continuous maintenance so that tests remain fast, reliable, and valuable. If not scaled efficiently, an organization’s automation suite will turn into a messy, uncontrollable beast. Having a lean test suite will help to combat this. In this session will present methods to keep automated test suites lean and mean, so they always provide quick and accurate feedback to the software delivery team. Using a few examples, she will discuss a wide range of ideas including evaluating a test's value, parallelizing tests, and producing consistent results. Session attendees will walk away with strategies and practices to scale their test automation over time in a highly efficient and maintainable way.

  • Liked NaliniKanth M
    keyboard_arrow_down

    NaliniKanth M / Prasad Mudedla - Continuous Security with Selenium tests

    45 Mins
    Demonstration
    Intermediate

    We have come to a place where we run our selenium tests on pipelines every day.
    Now are these tests confined to test only the functionality of the application?
    No, we can use this tests to do much more. Let's test security with those same tests.

    Turn your selenium tests in to Security tests, get security testing in to your CI pipelines just by tweaking a bit of the selenium integration suite. Find and report security issues that exists which can be caught way before paying off for it.
    In this Demo we will introduce a framework that will help your selenium tests turn into security tests and run on pipelines.
    With ZAP API + selenium and CI pipelines, find security issues on the application see a red or green security pipeline. Not just that fetch the reports and take necessary actions on every build.
    The talk covers different ways that ZAP tests the web application for security(automated with selenium tests). We will see what all of them can be integrated with selenium and can make most out the suite.
    As an add we will see how your API can be tested for security using ZAP API :)

    Tools used:
    1. Selenium Web driver framework.
    2. ZAP framework integrated with selenium framework.
    3. GO (for CI/CD)
    4. Maven for building and fetching reports.

  • Liked Shashank Chaturvedi
    keyboard_arrow_down

    Shashank Chaturvedi - Data Analytics in QA using PowerBI

    45 Mins
    Demonstration
    Intermediate

    In CI/CD world, we execute thousands of test cases, multiple times in a day. These tests generate tons of data, which can be extremely useful to generate insight on identifying recurring patterns of failures, hotspots, infra issues etc.

    In this talk, I'll share how we can reduce the results analyzing time, but also how we can extract meaningful information from all the historic test runs and provide in-depth insights using PowerBI Visualization and Analytics.

    Few Examples Below:

    Delta Analysis

    Error Analysis

  • Liked Dharmender Kumar
    keyboard_arrow_down

    Dharmender Kumar - Develop Securely with Zaplenium

    Dharmender Kumar
    Dharmender Kumar
    Architect
    JDA
    schedule 1 year ago
    Sold Out!
    45 Mins
    Talk
    Intermediate

    Zaplenium simply means ZAP + Selenium and named after the integration of both to ensure application security while it's being developed. ZAP provide client API's that help to capture URL's by spidering and scanning over the pages those are interacted in user flows. With ZAPlenium the tests can be targeted to certain areas of application where selenium helps in navigating to those areas and target ZAP to spider over those and scan them. Also, Zaplenium will help defining the assertions those can come handy while we integrate this in CI/CD pipelines.

    Participants will be able to define security sanity suite to their build pipelines. With code change the security can be tapped for any High, Medium and Low risks. Session will make use of open source technologies i.e. OWASP ZAP and Selenium to define tests with assertion.

    Security Tool: OWASP ZAP

    Functional Tool: Selenium Web driver

  • Liked Gil Tayar
    keyboard_arrow_down

    Gil Tayar - Frontend Testing: Stepping in and Collaborating with Developers

    Gil Tayar
    Gil Tayar
    Sr. Architect
    Applitools
    schedule 1 year ago
    Sold Out!
    45 Mins
    Talk
    Intermediate

    Testing is shifting left, moving closer to testing the code itself, and not the full product. But while managers dictate a shift to the left, the developers and testers themselves are confused as to how exactly to test the code. While the backend world already has established methodologies for testing, frontend developers and testers are still trying to figure out how to test their code.

    Testers need to step in and work with the frontend developers. Testers need to understand the framework by which frontend code is tested. They need to understand the various kinds of testing that can be performed on frontend code, and which tools can be used for this.

    I will give a recipe that you can follow to ease your fear of the unknown —writing tests for developer code. I will discuss the various kinds of testing and how they fit together in a coherent way, but most importantly - I will include sample code that you can use as a template in your own project, and if time permits—a bit of live coding will be done.

  • Liked Manjyot Singh
    keyboard_arrow_down

    Manjyot Singh - QAOPS - QA Testing in a DevOps World

    Manjyot Singh
    Manjyot Singh
    Lead QA Consultant
    Thought works
    schedule 1 year ago
    Sold Out!
    45 Mins
    Demonstration
    Intermediate

    QAOps is Continuous Testing Strategies when
    frequent software delivery matters.

    It is no surprise that automation and orchestration
    make life very easy, be it a small organization or
    a large-scale industry that houses hundreds of
    servers. IT automation is essentially the ability to
    orchestrate and integrate tools,
    people, and processes through a certain workflow. I will be using Ansible and Docker to showcase the same and writing ansible tests on top of that.

    Ansible - Ansible is an IT automation tool which can configure systems, deploy software, and orchestrate more advanced IT tasks such as continuous deployments. Ansible is actually designed to be a “fail-fast” and ordered system, therefore it makes it easy to embed testing directly in Ansible playbooks.

    Docker - A docker container is a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings. Containers isolate software from its surroundings, for example differences between development and staging environments and help reduce conflicts between teams running different software on the same infrastructure.