Using genetic algorithm to do exploratory testing with cucumber and selenium.

Writing selenium tests has an inherent problem in which we can only automate what we know the steps for. It is primarily used for regression and can scale very quickly. We generally optimize regression for sprint's relevant features or have smoke tests to get quicker feedback. In addition negative testing is very expensive to write. This entire process is very manual and the maintenance of tests become more costly than the actual writing of tests. 

We can change this. By using genetic algorithm and cucumber steps we can create new feature files to discover where in the application may be be problematic. The goal of each individual would be to find the most defect. The two individuals that find the most defects then create children that follows the sequences of how new individuals are form through genetic algorithms. They will breed the next generation of testers which will focus on problematic areas. 

Let's find out when and if this strategy can be deemed effectively. 

 
 

Outline/Structure of the Experience Report

  • Demonstrate genetic algorithm in traveling salesman problem
  • Demonstrate genetic algorithm in junit test.
  • Demonstrate genetic algorithm in selenium cucumber tests
    • Generation of cucumber feature file
    • Cheap strategy to maintain predictability of selenium and reduction of flakiness. 
    • The report

Learning Outcome

  • Understand the possible use of A.I. algorithms for exploratory testing than for regression testing. 
  • Understand how to maximize coverage in a shorter time using a fully automated process. 

Target Audience

General

schedule Submitted 7 years ago

  • Priyanka Gupta
    keyboard_arrow_down

    Priyanka Gupta / Sarah Eisen - Automation Alchemy On a Mass Scale: Turning Costly Manual Tests Into Automation Gold

    45 Mins
    Talk
    Beginner

    Do you want to hear a story about overcoming obstacles and achieving seemingly unattainable goals at a massive scale? Well, we have one to tell - it’s a true story, and like all good stories, teaches us some valuable lessons. We have gone through the ups and downs of this tale and come out better and smarter. We would love to share those experiences and learning with everyone.

    The story starts with a mission...automate 5000 hours of manual tests for our enterprise product. Like many other product based companies, we had one big monolithic application to test. The mission was to be accomplished with the resources available - no new magical dream team, we had to work with the resources we had - QA analysts with no technical background, a very small automation team, and a huge offshore manual testing group. Go figure! There was another twist - we had to accomplish our mission without dropping the current level of support for testing our enterprise application, including regression and new feature tests. Doesn't it all sound very familiar?

     

    This presentation will cover all aspects of our journey from the beginning to the end. We went through a lot of ups and down, and every single decision we made taught us a great deal. It is those experiences that we want to share with everyone.

    • We created a tool that wrapped the Selenium API in order to make it easy for non developers to write tests. The tests were written in a Domain Specific Language that made Selenium API calls with some application specific logic added in.
    • We needed to build our own execution framework to support our growing automated test base. The framework offered many customized features and was able to sustain 60,000 hours of tests running every single day.
    • We wrote our own best practices and worked closely with the QA team to make sure everyone wrote high quality tests.
    • The results from the tests needed to be displayed in a way that made sense. We created several different dashboards for that purpose and had many different views of the test suite performance, including a heat map to highlight problem areas.
    • Elasticsearch and Kibana were instrumental in helping us parse through the massive volume of test results and make sense of them, giving us metrics in different forms.
    • Daily environment setup for this execution was also massive - 100 or so slaves and several SUTs for every codeline, with support for 3 codelines meant that we needed a big lab setup.


    We successfully completed the mission of automating the manual test behemoth and gained a rich understanding of test automation at scale along the way.

  • James Farrier
    keyboard_arrow_down

    James Farrier / Xiaoxing Hu - Making Your Results Visible - A Test Result Dashboard and Comparison Tool

    45 Mins
    Demonstration
    Intermediate

    If a test fails in the woods and no one is there to see it does anyone care, does anyone even notice. What happens when failing tests become the norm and you can't see the wood from the trees? 

     

    After watching last years Allure Report presentation I was inspired.  Selenium tests (and automation tests in general) are often poorly understood by the team as a whole.  Reports/emails go unread with tests failing becoming an expected outcome rather than a glaring red flag.  We looked at what Allure brought to the table and from that base created a dashboard which was designed to:

    • Display the results of test runs in a way that was useful to managers, testers and the rest of the development team.  Including tools to filter out specific test runs and view the overall trend of the test run results.
    • Make debugging tests easier by grouping errors, displaying history of test results, filtering tests and offering visual comparison of test runs.
    • Help mitigate the problems flaky tests cause with test run result reporting (say that three times fast).
    • Help with our mobile device certification process, by easily providing a view to compare test runs across devices.

    Since it's creation the dashboard has been used and praised by managers through to developers.  With our full suite of tests from unit to integration to selenium and appium being stored on the dashboard.  We've managed to:

    • Decrease the time taken to debug test cases.
    • Increase the visibility of all our test suites, with managers having a better idea of how our selenium test suite is progressing and testers better understanding the coverage of unit tests.
    • Focus the organization on quality.

    We are working with legal at present to have this project open sourced and available to all prior to Selenium Conf 2015.

  • moiz
    moiz
    Software Engineer
    Saucelabs
    schedule 7 years ago
    Sold Out!
    45 Mins
    Talk
    Beginner

    Appium, often dubbed "Selenium for mobile", at heart its a web server written in NodeJs. Its architecture is modular, which means that it is composed of many small, independently maintained and tested modules. Testing Appium is challenging, but clearly very important, since thousands of users depend on it for their testing. Appium also has all the usual challenges of a large open source project, for example, ensuring consistency of JavaScript code style across hundreds of contributors. It's important to have high-quality and readable code.

     
    I will be discussing approaches to and strategies for testing these kinds of large, modular applications. On the Appium team, we use a combination of unit, functional, and integration tests. Modern services like GitHub, Travis CI, and Sauce Labs make it possible for large open source projects to be tested thoroughly, keeping the code and the app at high quality. I will also discuss the use of tools like JSLint and Gulp, which help prevent code style issues.
     
    Testing the tool which is used for testing is clearly very important. This talk aims to showcase how testing should be approached for large, modular projects which has many collaborators.
  • rubytester
    keyboard_arrow_down

    rubytester - Docker Selenium. Getting Started

    45 Mins
    Demonstration
    Beginner
    • `docker-selenium` project is about packaging selenium grid as docker containers (https://github.com/seleniumhq/docker-selenium)
      To me this means I don't have to build any selenium infrastructure machines. I just run the provided images by docker-selenium project (https://hub.docker.com/r/selenium/).
    • I don't have to install selenium jar, java, browsers and other runtime dependencies. They are already built in a docker image and I can just run them as either selenium grid with hub and nodes or as standalone selenium on any docker engine enabled vm. 

    In this talk/demo/case study I will show you how you can use `docker-selenium` project to build several pipelines starting from running on your local dev box to public cloud for quick tests and finally to a stable private cloud for your team.

     

  • Jonathan Lipps
    keyboard_arrow_down

    Jonathan Lipps - The Mobile JSON Wire Protocol

    Jonathan Lipps
    Jonathan Lipps
    Project Lead
    Appium
    schedule 7 years ago
    Sold Out!
    45 Mins
    Talk
    Intermediate

    The JSON Wire Protocol (JSONWP) is the version of the WebDriver spec currently implemented by all the Selenium clients. It defines an HTTP API that models the basic objects of web automation---sessions, elements, etc... The JSON Wire Protocol is the magic that powers Selenium's client/server architecture, enables services like Selenium Grid or Sauce Labs to work, and gives you the ability to write your test scripts in any language.

    The JSONWP has served Selenium faithfully for a number of years, but the future of automated testing lies beyond the borders of the web browser. Mobile automation is an essential ingredient in any build, and tools like Appium or Selendroid have made it possible to run tests against mobile apps using the JSONWP. The JSONWP's current incarnation isn't enough to automate all the new behaviors that mobile apps support, however. Complex gestures, multiple device orientations, airplane mode, and the ability to use both native and web contexts, for example, are all essential to mobile automation.

    For this reason the leaders of the Selenium project, in concert with other Selenium-based projects like Appium and Selendroid, met to discuss the future of the JSONWP. We've been working on its next version, called the "Mobile JSON Wire Protocol" (MJSONWP). Appium and Selendroid already implement much of the MJSONWP spec. In this talk I'll dive into the specifics of the MJSONWP extensions, how they relate to the original JSONWP, and how the Selenium clients have begun to implement them.

    Finally, I will talk about the future of the MJSONWP and how it's related to the current and future versions of the WebDriver spec. I'll share how you can get help with the creation of the MJSONWP, and discuss issues with the authors of the new spec before the API is set in stone. We need the help of everyone who's involved in mobile automation to come up with the best and most future-proof version of the MJSONWP. Ultimately, your understanding of how Selenium works will be improved, and you'll have a much better handle on how projects like Appium and Selenium work together to make sure you have the best automation methods available.

  • Justin Woolley
    keyboard_arrow_down

    Justin Woolley / David Anderson / Marvin Ojwang - Automating for the Second Screen with WebdriverJS

    45 Mins
    Talk
    Intermediate

    There has been a recent explosion in second-screen technologies such as Chromecast, but designing test automation for second-screen applications is far from straightforward. This new paradigm lacks major automated tool support, and coordinating test execution across multiple devices is tricky and error-prone.

    Our automation solution uses WebdriverJS and WebSockets to perform end-to-end test automation that covers our web player controller and second screen application.

    Learn about our approach to second-screen automation which we’ve used to build a reactive, responsive test suite. We’ll describe our solutions to synchronizing test flow between the controller and target device, validation on the device, targeting different integration components, and device management.

  • Ragavan Ambighananthan
    keyboard_arrow_down

    Ragavan Ambighananthan - Distributed Automation Using Selenium Grid / AWS / Autoscaling

    45 Mins
    Talk
    Advanced

    Speed of UI automation has always been an issue when it comes to Continuous Integration / Continuous Delivery. If UI automation suite takes 3 hours to complete, then any commit happens during this time will not be visible in test environment, because the next deployment will happen only after 3 hours. 

    With 2000+ developers and average 250+ checkins per day, the above issues is replicated 250+ times every day. This is not productive and feedback cycle is super slow!

    Another issue is , with 35+ different project teams using 10 or more different jenkins jobs to run their UI automation. So many jobs means (350+), individual teams need to go through the pain of managing their own jenkins job, its a duplicate effort and waste of time. Automation teams need to spend time on writing reliable automation and not managing jenkins jobs.

    Solution is to reduce the UI automation run time from hours to minutes and also use only handful of jobs to run the Distributed Automation!

    Goal: To run all UI automation scenarios within the time take by the longest test case

  • Sveta Kostinsky
    keyboard_arrow_down

    Sveta Kostinsky - Selenium Today vs. Selenium Tomorrow: Digital as the Convergence of Mobile & Web Programs

    45 Mins
    Talk
    Beginner

    Today, mobile is increasingly trumping web as the most important brand engagement point; enterprises are moving away from mobile and web projects independent of each other. The rapid adoption of responsive web encourages teams to discover one approach to measuring software quality regardless of form factors.

     

    Selenium is current market leading solution for web testing, but how does it stand with mobile? The truth is that working with Selenium presents a few challenges, including:

    • Building and maintaining an internal structure to support it
    • Bridging an architectural gap
    • Requirements demand support for unattended test execution
    • Lack of real network conditions for mobile testing

     

    There is a solution to address these challenges!

    Let’s work through a demo and show how to test mobile & web in parallel with Selenium

  • Anurag Singh
    Anurag Singh
    Sr. SDET
    CK-12 Foundation
    schedule 7 years ago
    Sold Out!
    45 Mins
    Experience Report
    Intermediate

    Selenium Conference 2014 in Bangalore was the first time got a chance to attend this event. During the event got exposed to lots of brilliant ideas and experiences shared in various talks.

    In last one year had opportunity to try some of them which resonated within Automation Community of our organisation. As part of this talk would like to share our experiences, which might help participants to get it done right first time.

    A. Getting the right test pyramid: The idea of having Unit Tests, Integration Tests and GUI Tests in right proportion makes perfect sense. In this part of talk I would like to take you through our efforts to beef up the Integration Tests. It was a two pronged strategy. First automated backend api tests using BDD and some python modules.
    Second added integration tests for frontend modules done using ReactJS using Chai, Mocha, Sinon, React TestUtils, PhantomJS etc.

    B. Appium for Mobile Automation: Our test infrastructure didn't have capability to support Mobile Automation. To support the Mobile First approach catching up and requirement of rich user experience on mobile for our service we needed Appium.

    In this part of talk will share our experiences enabling existing tests to run on Mobile devices using Appium and with iOS/Android emulators, real devices, SauceLabs.

    C. Using third party infrastructure service instead of local grid:  Idea of not maintaining the test infrastructure and growing number of tests on web and mobile made us look in this direction. We chose SauceLabs as the preferred infrastructure service. This part of talk will cover how we went about trying SauceLabs, challenges we have faced and some pros and cons of using SauceLabs.

  • David Giffin
    keyboard_arrow_down

    David Giffin - A Large-scale, Data-driven Company's Journey of Going From Manual to Automated Testing In 6 Months

    David Giffin
    David Giffin
    Technologist
    TrueCar
    schedule 7 years ago
    Sold Out!
    45 Mins
    Talk
    Beginner

    Manual Testing.  Depending on how you've been influenced by those two simple words, reactions vary from slight disgust to full-on depression.  Of course, the solution is clear: automate, but how do you get there when your company is continually pushing out the next big feature?  As the set of features to cover increases, the lack of scalability of manual testing becomes more apparent.

     

    This is a problem that we struggled with at our company.  Automation tactics were explored and implemented, but problems persisted as proposed solutions did not cater to the demands of the manual testers.

     

    After years of failure and disappointment, our latest stint resulted in success.  Not only do we have hundreds of automated tests across various platforms (mobile and web) and products, but manual testing has been eliminated with zero casualties.  As we move forward towards Continuous Delivery and improved automation performance, we wanted to take this moment to look back and share stories of failure and success.

  • Anthony Browness
    keyboard_arrow_down

    Anthony Browness - Dependency Injection in the Page Object Model

    45 Mins
    Talk
    Advanced

    The Page Object Model is awesome, but you can make it even better with Dependency Injection.  I'll show you how to get started and some benefits of using Spring IOC in your Page Objects.  Though the concepts of this talk can be extended to any language, the examples will be shown in Java.

  • Bhumika S
    keyboard_arrow_down

    Bhumika S - The 10 Myths of Testing that we need to Break

    Bhumika S
    Bhumika S
    Sr. Quality Analyst
    MYOB
    schedule 7 years ago
    Sold Out!
    45 Mins
    Talk
    Beginner

    This talk aims to break 10 (amongst many) Myths that desperately need to be broken around Software testing and Testers. These so-called myths, have been around for so long that they are almost the norm. Breaking these myths is necessary to help budding as well as experienced Quality Analysts and teams, roll out better quality software products and build better team practices as well.

  • manjyot singh
    keyboard_arrow_down

    manjyot singh - WebDriver - Partners in crime

    manjyot singh
    manjyot singh
    Sr. Consultant
    ThoughtWorks
    schedule 7 years ago
    Sold Out!
    45 Mins
    Talk
    Intermediate

    Talk is about :

    1) How neatly we can design our tests using page object patterns.

    2) How easy it is for users belonging to different roles to understand the object of tests using BDD i.e. cucumber.

    3) How handy it is to use test framework like TestNG along with WebDriver.

    4) How easy it is to maintain dependencies of your test framework using build tools i.e. Maven.

    I don't want to get into serious crime of boring the audience with the details that are not at all required/ necessary but to target the areas which beginners and experts are interested in.

  • 45 Mins
    Case Study
    Advanced

    When tasked with producing a comprehensive HTML5 test automation solution for the enterprise we needed to insure we provided extensive value that went well beyond what most existing toolsets individually provided.  We needed to solve some of the most common problems faced by our web testers and developers:

       * Start/Stop shared Selenium browser sessions across different tools with RemoteWebDriver extensions
       * Persist Selenium browser sessions across different testing sessions (tool invocations) with RemoteWebDriver extensions
       * Remove element identification and resolution out of the tests for simpler test maintenance with Maps
       * Automate various test design-time activities for the test developer
       * Simplify common but complex Selenium automation functions for testers with Libraries
       * Intermix Selenium API, 3rd party API, and SeBuilder scripts as needed with SeInterpreter extensions
       * Automatically blend image-based testing with Selenium support where needed (screen/canvas)
       * Automatically blend Selenium support with Java AWT Robot (keyboard/mouse) support where needed
       * Extend the "Selenium with Java AWT Robot" support to remote servers and grid nodes
       * Automatically launch "local" or "remote" Selenium Server, Hub, or Grid nodes if not running


    We can discuss each of these issues and how we chose to solve them.  The various solutions along with all source are available as opensource.

  • vishnu nallani chekravarthula
    keyboard_arrow_down

    vishnu nallani chekravarthula - Extending Selenium Element Locator Strategies – Element Filtering

    45 Mins
    Talk
    Intermediate

    Element Locator strategies for Selenium WebDriver are highly flexible, and have been later inherited by many commercial tools. Although the locator strategies are flexible, they are also limited in a sense that, Selenium WebDriver does not currently allow its users to identify/filter UI elements with multiple locator strategies(at a time), as many commercial tools do.

    The solution discussed in this article describes a library that allows Selenium WebDriver users to extend the Selenium element locator strategies for Element Filtering and few use cases for the library.

    The solution approach allows users to continue to use the existing UI Element definitions in their tests, and extend them, using the By reference. The library will replace the existing Selenium WebDriver “By” reference.

    Filtering based on multiple locator strategies

    There are various scenarios where to uniquely identify an UI element, a complex XPath has to be written. However, the element can be identified uniquely using multiple locator strategies for the UI Element. The UI Elements can also be filtered, when there are multiple matches in a page. This is the UI Element recognition mechanism used in many commercial test automation tools.

    The algorithm for filtering UI Elements based on multiple locator strategies is based on priority of locator strategies. The priority of locator strategies when filtering is:

    1. ID
    2. Name
    3. TagName
    4. ClassName
    5. XPATH
    6. LinkText and PartialLinkText
    7. CSS

    The By.elementFilter method takes multiple locator strategies, and searches the page for elements matching a particular locator strategy/property, and checks if it is a unique match on the page, if not then it uses the next locator strategy passed to it and so on.

    This method is also very helpful when the application undergoes constant changes and UI Elements might have either of XPATH, ID , NAME, TagName, ClassName etc still unchanged. That way, it helps reduce a lot of maintenance effort in Selenium WebDriver implementations which is due to UI element changes.

    Filtering based on Index

    When there are multiple similar UI Elements in a page, such as cells in a grid/table, it makes sense to identify objects based on their Index based on their appearance on the web page.

    The By.indexFilter method allows users to define an UI Element based on its Index of occurrence of the UI Element. The Index starts from 1.

    Filtering based on relative element

    When a UI element cannot be identified uniquely and reliably by any of its properties, but has some elements in its hierarchy or relative to a particular element, this method can be used to identify the element

    The By.relationFilter method allows users to define an UI Element in relation to another element. The relation can be defined as “Left”, ”Right”, ”Top”, ”Bottom”, ”Child”, ”Parent”

    Filtering for Tables

    When dealing specifically with Tables, which have the

    html tag, the By.tableFilter method allows the user to quickly identify specific cells in the table, without having to write complex XPaths or logics to achieve the same.

    The By.tableFilter method allows users to define a cell in the table with Row,Column numbers. This allows users to directly use the UI Element in their code instead of writing their logic each time. This also increases efficiency and readability of the code.

  • Justin Ison
    keyboard_arrow_down

    Justin Ison - Android Mobile Device Grid & CI - Getting Started

    Justin Ison
    Justin Ison
    Sr. Software Engineer
    Applitools
    schedule 7 years ago
    Sold Out!
    45 Mins
    Talk
    Intermediate

    In the modern era, we have many different cloud testing services to choose from. These cloud services are useful and help reduce the burden of building and maintaining your own Selenium Grid environment. However, there are many scenarios in which you need your tests running locally and quickly, such as you work for the government (or agency), you have sensitive software/data you cannot expose to the cloud, or service costs are too expensive for your organization.

    This presentation will feature getting started with setting up your own mobile device grid, running your tests in parallel, running in CI (Jenkins), and the lessons I have learned along the way.

     

  • Jason Watt
    keyboard_arrow_down

    Jason Watt - Challenges of the Mobile Cloud

    Jason Watt
    Jason Watt
    Sr. Software Engineer
    Salesforce
    schedule 7 years ago
    Sold Out!
    45 Mins
    Talk
    Intermediate

    Creating a mobile app is now the new cross platform problem. The major mobile platforms tend to gear their development tool chain towards individuals and their workstations.  But what if you want to introduce a CI solution to this environment? What if your app is launching on more than one platform and there's a team of 20+ developers working on it? What if your tests are more than just Selenium based?

    This is normally where you can look to the cloud for scale but mobile has a ton of challenges to do so.  Come and learn from some of the challenges and pitfalls I've encountered while working towards this goal.

  • Rex Hoffman
    keyboard_arrow_down

    Rex Hoffman - Stop repeating yourself while testing!

    45 Mins
    Talk
    Intermediate
    Make your selenium test code fully reusable by using models and templated data. 
    Make your tests suite easier to reason about and thread safe! 
     
     
    A walk through of five projects showing how to test applications in a deterministic way, with increasingly less code duplication.   The business value of the application under test is clearly highlighted at every level of abstraction, the standard page object, a model, and test scoped data.  Clean, decoupled use of cucumber-jvm to drive tests will also be demonstrated in the final project.
     
    Based on testing libraries used successfully in anger at medium sized companies (hundreds of devs).
     
     
  • Mike Levin
    keyboard_arrow_down

    Mike Levin - Selenium meta hub – scalable and redundant infrastructure

    45 Mins
    Talk
    Intermediate
     
    Selenium grid is widely used in lots of companies and projects. Unfortunately with the current open source implementation one can not run more than one hub which can cause various problems due to hardware or network instability. Single hub architecture is also hardly scalable, at least it requires hardware upgrades for the hub.
    At the same time many teams implement their own internal solutions which are usually not shared because of the team/organization specific or custom selenium hacks.
    In Yandex we have multi hub solution for more than 5 years. At the same time we are doing our best to avoid making custom internal patches to selenium.
    During these years we were using client side balancing approach: client applications were always obtaining a browser via a special internal library. It knew the configuration of all the hubs and browsers and was performing the search for an available node on request.
    But when it came to the different test frameworks, different languages and different runtimes this approach became difficult to support. As long as test practices move from test engineers to development teams the diversity of frameworks and runtimes increases. So we come up with meta hub solution.
     
    Our meta hub solution has the following basis:
    • Stock versions of selenium and selenium grid
    • Stock web driver interface for the client
    • Virtual infrastructure. We use Openstack for all parts of our infrastructure. It’s not necessary but It makes sense.
    • Fixed load for each hub and node – scalability via adding new hubs with fixed volume.
    • Redundancy and scalability
    • Stateless solution for meta hub. No storage requires to keep state between several meta hub nodes
     
    We made solution that includes:
    • Proxy software between client and multi-hub grid installation
    • Some configuration adjustments for hubs/nodes.
     
    I'll talk about our solution. We going to go open source on the SeleniumConf conference or earlier.

     

  • Sarah Thompson
    keyboard_arrow_down

    Sarah Thompson - DevOps meets QA - Using Puppet to set up and manage your Selenium Grid

    Sarah Thompson
    Sarah Thompson
    QE UI Toolsmith
    Puppet Labs
    schedule 7 years ago
    Sold Out!
    45 Mins
    Talk
    Intermediate

    For testers, setting up and maintaining a Selenium Grid infrastructure can be timely and costly. A lot of the time, we are asked to do this as part of our day to day job when we really want to focus on testing the product!

    There are some great cloud based alternatives out there that allow you to easily run your tests on a wide range of Operating Systems and against multiple browser types (at a price).

    But what if you already have plenty of devices available within your own company (be it physical machines, virtual machines or cloud based resources) and you want to be able to setup and manage your own grid infrastructure:

    • to allow you greater control over the configuration (perhaps you want to have a headless browser like PhantomJS available on your grid)
    • to run your tests faster (the cloud based alternatives are a good bit slower for obvious reasons)
    • or to simply save money.
help