Reusing Selenium tests for catching Performance Regressions

Almost all the tests we write today are geared towards verifying the functional correctness of products. Selenium gives us a great way to ensure that our web applications and browser behave correctly and our tests usually do an excellent job running through the happy path.

Most successful websites or hybrid mobile applications are not just functionally correct, but also have a very smooth performance and user experience. Performance, for many, is now a feature. In this talk, we will look at ways to re-use our selenium test cases to also catch any performance regressions. We will measure key performance indicators like frame rates and memory usage as the selenium scripts navigate and perform actions on the website. We will look at logging all these metrics into a dashboard, and integrating this with a continuous integration system like Jenkis or Team City. Finally, we will also look at how such a system can catch any code change that is responsible for making the website slower than a threshold we set.

 

We will look at how we could use existing testing frameworks like Protractor or Jest to add performance metrics. We will extend our test matrix to cover desktop browsers, browsers on popular mobile platforms and even hybrid apps like Apache Cordova, Ionic or Phonegap.

 

 

If Performance is a feature, let us test it like we test features !!

 

 
4 favorite thumb_down thumb_up 1 comment visibility_off  Remove from Watchlist visibility  Add to Watchlist
 

Outline/structure of the Session

  • Start with an extising Selenium test suite and web page to be tested
  • Re-purpose the tests to include performance metrics
  • Check if the performance metrics meet our criteria, if not fail the tests
  • Add this into a Continuous Integration System
  • Monitor the CI system for any performance regressions.

Learning Outcome

At the end of the sessions, the audience will be able to

  • Understand and appreciate the value of adding performance tests to their test suite
  • Look at easy ways to add performance test to existing test matrix
  • Test on desktop browsers, mobile phone and hybrid apps
  • Add this to a continuous integration system

Target Audience

Developers and Testers who are interested in getting more that just functional testing out of their scripts.

schedule Submitted 1 year ago

Comments Subscribe to Comments

comment Comment on this Proposal
  • Anand Bagmar
    By Anand Bagmar  ~  1 year ago
    reply Reply

    seems interesting. A couple of questions:

    - is 90 min too long for this? can you add timings to your structure for better understanding? will you be able to keep the audience engaged for the full duration? have you done similar duration sessions before?

    - will code samples be made available to attendees to take it forward?

    - will you need internet access for your demos? I would suggest having complete local setup to avoid internet dependencies and glitches in the demos.

     


    • Liked Ori Bendet
      keyboard_arrow_down

      Ori Bendet - Tales from the Dark Side: The Growth, Implementation and Influence of Selenium inside Hewlett Packard Enterprise

      45 Mins
      Talk
      Intermediate

      I know what you’re thinking: the creators of WinRunner, QTP/UFT are now embracing Selenium?

      Ten years after Selenium came into existence as an open source alternative to Mercury Interactive, the perception and relationship between QTP and Selenium has morphed from competition to collaboration with complementary test automation frameworks.

      Join Ori Bendet, HPE Inbound Product Manager for Functional Testing to discuss how HPE’s R&D uses Selenium and other open source tools. Understand the new roles and responsibilities of dev/test @HPE and how they fit into current team structure. Discover their lessons learned about how Selenium and open source has contributed to the success and maturity of HPE's own quality assurance and testing tools across the entire portfolio.

    • Liked Sargis Sargsyan
      keyboard_arrow_down

      Sargis Sargsyan - Better Page Object Handling with Loadable Component Pattern

      45 Mins
      Talk
      Advanced

      One of the painful problems in Selenium automated testing is determining whether a HTML page has been loaded. This is especially the case when web application uses a JS  heavy framework such as the popular AngularJS.

      During this talk we will discuss how to handle Selenium Page Object pattern better with Loadable Component.

      The Loadable Component helps test case developers make sure that the page or a component of the page is loaded successfully. I will share my experience about the concept of the Loadable Component and Page Object patterns.

       

       

    • Liked Ankita Gupta
      keyboard_arrow_down

      Ankita Gupta / Jatin Makhija - Web Push Notification Automation Mystery Solved!

      45 Mins
      Demonstration
      Intermediate

      Push Notifications are the latest way of sending updates to our users. More and more Organisations are implementing Web Push Notifications along with emails and other notification systems.

      So the Big Question that arises is "How do we automate them?"

      We have come up with a library in various languages which can be integrated with your Automation suite and provide you everything you need about the notification triggered.

      You can easily then trigger and verify the push notification sent and ship out the product without worries :)

    • Liked Vinay Babu
      keyboard_arrow_down

      Vinay Babu - Web Scrapping with Selenium and Data Analysis using IPython Notebook

      20 Mins
      Talk
      Intermediate

      Data Analysis is one of the upcoming field and as many of the data scientists says that the most of time they spend for analysis is on Data cleaning, So, In this short session we will see how one can pull the data from the web using Selenium Webdriver and will use this data further for the Data Analysis, The entire exercise will be executed on a IPython Notebook, which is a tool used to execute & save your code and perform data analysis using python data analysis libraries, it also provides a platform to massage the data and visualize it in the form of graphs and tables.

      This entire exercise would be helpful for anyone who wants to understand how data can be pulled with the help of Selenium Webdriver from a website and organized using python libraries for the data analysis. During this session we would be using an open source data for analysis and see how we can draw conclusions using this data.

    • Liked Michal Vanek
      keyboard_arrow_down

      Michal Vanek / Filip Braun - Breaking down the barriers: Testing desktop apps with Selenium

      45 Mins
      Talk
      Intermediate

      Selenium was born for web-application testing. But have you ever thought it could be a great tool for testing Windows desktop apps too?

      Today, more and more desktop apps use a web-like approach to implement their UI. The methods vary from basic HTMLayout environment to complex designs in CEF (Chromium Embedded Framework). However traditional GUI automation tools seem to be a step behind or ignoring the trend completely. This situation calls for finding new ways of testing.

      In our talk we shall introduce to you a new way of utilizing Selenium for automated testing of desktop applications. No matter whether the HTML UI content is completely offline or loaded and updated dynamically, Selenium is able to access and navigate it just like in a web page. We’ll also show you how to build a small framework around it and plug it into your Continuous Integration process. All of this will be demonstrated using a real-life instance of Avast Antivirus for Windows.

       

    • Liked Adam Carmi
      keyboard_arrow_down

      Adam Carmi - Advanced Automated Visual Testing With Selenium

      Adam Carmi
      Adam Carmi
      Co-Founder and VP R&D
      Applitools
      schedule 1 year ago
      Sold Out!
      45 Mins
      Talk
      Beginner

      Automated visual testing is a major emerging trend in the dev / test community. In this talk you will learn what visual testing is and why it should be automated. We will take a deep dive into some of the technological challenges involved with visual test automation and show how modern tools address them. We will review available Selenium-based open-source and commercial visual testing tools, demo cutting edge technologies that enable running cross browser and cross device visual tests at large scale, and show how visual test automation fits in the development / deployment lifecycle.

      If you don’t know what visual testing is, if you think that Sikuli is a visual test automation tool, if you are already automating your visual tests and want to learn more on what else is out there, if you are on your way to implement Continuous Deployment or just interested in seeing how cool image processing algorithms can be, this talk is for you!

    • Liked Justin Ison
      keyboard_arrow_down

      Justin Ison - Appium Workshop

      Justin Ison
      Justin Ison
      Senior Software Engineer
      Microsoft
      schedule 1 year ago
      Sold Out!
      480 Mins
      Workshop
      Beginner
      This workshop will feature getting started with automating Android with Appium. We'll learn how to capture elements in Ruby using irb (interactive Ruby), writing a test with rspec. Once we have our tests established, we'll then go into setting up a mobile device grid. Running the tests in parallel, distributed, and capturing critical report data (logs, screenshots, and video). We will also run some examples against a cloud testing service.

       

    • Liked Luke Inman-Semerau
      keyboard_arrow_down

      Luke Inman-Semerau - Grid Workshop

      Luke Inman-Semerau
      Luke Inman-Semerau
      Lead Member Technical Staff
      Salesforce
      schedule 1 year ago
      Sold Out!
      480 Mins
      Workshop
      Advanced

      Selenium Grid can be a bit daunting to get up and running. Starting it is quite easy, but using it effectively requires pulling in third party tools. In this workshop we’ll cover how you would realistically run your grid, using modern tooling to run a grid with docker containers or in a cloud service like AWS or theoretically your own VM provisioning environment.

       

    • Liked Sarvesh Shrivastava
      keyboard_arrow_down

      Sarvesh Shrivastava - Functional and Security Testing - An amalgamated automation approach

      45 Mins
      Demonstration
      Intermediate

      With increasing cyber threats & online attacks, an unavoidable situation for continuous security testing has emerged. Making sure all vulnerabilities are unleashed regularly is highly significant.

      The paper hence proposes a solution where automated security testing could be achieved in conjunction with functional testing carried out using selenium API.

      It introduces a framework that caters to automated security testing along with functional which could provide an integrated testing elucidation.

      The paper, takes in these two premises to offer a solution where functional automation testers can now take on security testing. I propose a framework where automated security testing could be achieved in conjunction with functional testing using existing selenium API scripts.

      The framework covers the top vulnerabilities and provides intuitive results that help a non-security tester interpret and act on the output. At the very core of this framework is the open source tool, OWASP ZAP, which is easy to use and integrates well with Selenium automation frameworks.

      I bring in hands on project experience having implemented this framework for clients, who have been able to get the value of functional and security testing using the same set of scripts – it is this experience I would like to share with the SeleniumConf2016 audience, to help groom functional testers into security testing, with minimal cost and time, also enabling security testing to be performed every time functional automation is taken up.

    • Liked Sargis Sargsyan
      keyboard_arrow_down

      Sargis Sargsyan - Web Application Testing with Selenium

      90 Mins
      Talk
      Intermediate

      Creating automated tests for a web application can be challenging. During this talk we will discuss how to use the popular browser automation framework, Selenium, to create automated tests for web applications. We will examine using Selenium and Java to automate the web browser using Selenium WebDriver.  Also we will talk about the implementation of a simple, maintainable framework for testing a web application using Selenium.

    • Liked Deepshikha Singh
      keyboard_arrow_down

      Deepshikha Singh - Appium: Mobile App Automation Made Awesome.

      45 Mins
      Talk
      Intermediate

       

      This session will help beginners who are interested in mobile testing and help them kick start right away, we will start from ground 0 and then move on to providing better understanding of the tool and writing scripts using the tool. This session is for participants who perceive mobile testing to be difficult and that it requires good coding skill. I will demonstrate by automating (running) all types of application supported by Appium. Appium is an open source project and has made design and tool decisions to encourage a vibrant contributing community.

       

       

       

    • Liked Christina Thalayasingam
      keyboard_arrow_down

      Christina Thalayasingam - Distributed Testing and Test Reporting

      45 Mins
      Demonstration
      Advanced

      As we are moving into the agile world, continuous integration has a major role to play.

      So how do we cater for a complete test on every sprint or every release? We can use Selenium for Test Automation. When we use a continuous integration approach it would be helpful to use Selenium Grid. It allows you to run your tests on different machines against different browsers in parallel. Essentially, Selenium-Grid supports distributed test execution. 

      This helps you to run your automated tests on various different machines, operating systems and browsers at the same time. This saves time and would help  to run your testing in a nightly build.

      Extent Reports will go hand in hand with Selenium Grid as it will help you retrieve all test results including Test Evidences into a comprehendible report.

      This talk would have a quick guide on how to use Selenium. With details on how to create html reports (with latest plug-in) which would give understanding test execution results for both technical and non technical people. The highlight of the talk would be on Selenium Grid which permits to run Selenium test cases on various operating systems and browsers from a specific hub. This would cover quick demonstrations on main browsers used in the industry such as Firefox, Chrome and Internet explorer. This would help for continuous integration.

    • Liked Dharmesh Vaya
      keyboard_arrow_down

      Dharmesh Vaya - Selenium + SoapUI + JMeter : A complete framework

      45 Mins
      Talk
      Beginner

      This talk showcases how you can develop a framework in Java with all kinds of features like WebTesting with Selenium, Service Layer testing with SoapUI and Load Testing with JMeter - all packaged as a single testing solution. Above all, make use of open source libraries and get details HTML reports as well as Summary reports. This solution allows you to seamlessly integrate all your testing requirements under a single framework.

       

    • Liked Rajdeep
      keyboard_arrow_down

      Rajdeep - Enrich Your Automation With Visual Assertions

      Rajdeep
      Rajdeep
      Senior Consultant QA
      ThoughtWorks
      schedule 1 year ago
      Sold Out!
      45 Mins
      Talk
      Intermediate

      This talk is about a ruby gem I wrote for adding visual assertion with selenium as driver.

      While we continuously implement new functionalities in our app, automated functional tests acts as safety net and gives us confidence. Still, there is something missing. Many times we get feedback like:
      "Sign-In button has shifted and user have to scroll down to see it"
      "We wanted to change background color of screen-X, it got changed for Screen-Y screen as well"
      "Why has fonts of this link become so tiny?”
      I am working on customer facing enterprise level Mobile apps of one of the biggest airline in the world. For us user experience is as important as functionality.
      Our regression suit did not have capability to catch such issues. Moreover, we started building another airline app from the same codebase with only change in user interface of this new app. This means, change done for one app, affects another. Hence, it became really critical to enhance our automation with visual assertions.
      To accomplish this, I worte my own gem called 'nakal'.http://rubygems.org/gems/nakal

      This is going to be a Talk + Demonstration

    • Liked Soumyajit Basu
      keyboard_arrow_down

      Soumyajit Basu - Running your tests headless using Java + Selenium + PhantomJs Driver + TestNG

      20 Mins
      Talk
      Beginner

      In this era of continuous software development and continuous delivery automation of a software plays a really big role in reducing time and effort for quality control and visually verifying individual functionalities. Automation plays a vital role in quickly pin pointing any break within the functionalities of the software system and also provides a quick report so that the software development team is aware what functionality in the system needs to be attended.

      The session would contain running an automation test headless by using the capability of phantomjs driver, running Selenium, implemented in Java and triggered by TestNG. People interested in learning automation can utilise the power of phantomjs driver to run tests quickly and get the desired test reports.

    • Liked Jatin Makhija
      keyboard_arrow_down

      Jatin Makhija / Ankita Gupta - Automation testing: An endangered species in Agile development

      45 Mins
      Demonstration
      Beginner

      With the increased pace in adoption of agile methodology, testing is now becoming really complex in relation to the not so simple yet sweet demand of every project manager: "Why don’t you automate test cases while you test?"

      Having worked and counselled in product giants and more than 5 startups in India, we would like to throw some light on what fits the test automation use case. Is UI automation "the Only way to test efficiently?"

      The main issue with startups following Agile is “time crunch.” With multiple deploys a day and a fast paced development cycle, how does one find time to automate the product?

      In this session, we will be sharing some methods/tools/techniques which increase the overall test coverage and have worked well in the past for startup culture.

      A demo of some simple, useful and robust tools used for API, UI and Performance testing will also be given.

    • Liked Dharmesh Vaya
      keyboard_arrow_down

      Dharmesh Vaya - Practical tutorial in Test Framework development

      480 Mins
      Workshop
      Intermediate

      Advanced users in Selenium are given the challenging task of developing a framework from scratch. However, its best to understand the core requirements of how the framework is going to be utilized in order to develop the framework with minimal efforts. This tutorial teaches development of core-building blocks that can be implemented for any Selenium based framework with maximum flexibility and minimum efforts.

    • Liked Sweta Shahi
      keyboard_arrow_down

      Sweta Shahi - Visual Regression Testing - How Selenium can help?

      Sweta Shahi
      Sweta Shahi
      Group Lead
      Srijan Technologies
      schedule 1 year ago
      Sold Out!
      20 Mins
      Demonstration
      Beginner

      In today's software world, web applications need to go though constant change to maintain highest standards of quality of service. With continuous changes, one of the biggest challenges in the testing fraternity is to reduce the effort in regression testing. Often UI testing is cited as an area where automated tools have a limitation.

      In this session, will try to cover how using a simple program and using tools like ImageMagick, we can automate visual regression of the site in no time. Also, this means after every change (front-end or feature) if causes any unexpected behavior can be brought to notice immediately. A set of benchmark images of the stable version will be compared against those from the build's latest version and help reduce the extra effort testers spend in testing the UI on different browsers.