Do you want to hear a story about overcoming obstacles and achieving seemingly unattainable goals at a massive scale? Well, we have one to tell - it’s a true story, and like all good stories, teaches us some valuable lessons. We have gone through the ups and downs of this tale and come out better and smarter. We would love to share those experiences and learning with everyone.

The story starts with a mission...automate 5000 hours of manual tests for our enterprise product. Like many other product based companies, we had one big monolithic application to test. The mission was to be accomplished with the resources available - no new magical dream team, we had to work with the resources we had - QA analysts with no technical background, a very small automation team, and a huge offshore manual testing group. Go figure! There was another twist - we had to accomplish our mission without dropping the current level of support for testing our enterprise application, including regression and new feature tests. Doesn't it all sound very familiar?

This presentation will cover all aspects of our journey from the beginning to the end. We went through a lot of ups and down, and every single decision we made taught us a great deal. It is those experiences that we want to share with everyone.

• We created a tool that wrapped the Selenium API in order to make it easy for non developers to write tests. The tests were written in a Domain Specific Language that made Selenium API calls with some application specific logic added in.
• We needed to build our own execution framework to support our growing automated test base. The framework offered many customized features and was able to sustain 60,000 hours of tests running every single day.
• We wrote our own best practices and worked closely with the QA team to make sure everyone wrote high quality tests.
• The results from the tests needed to be displayed in a way that made sense. We created several different dashboards for that purpose and had many different views of the test suite performance, including a heat map to highlight problem areas.
• Elasticsearch and Kibana were instrumental in helping us parse through the massive volume of test results and make sense of them, giving us metrics in different forms.
• Daily environment setup for this execution was also massive - 100 or so slaves and several SUTs for every codeline, with support for 3 codelines meant that we needed a big lab setup.

We successfully completed the mission of automating the manual test behemoth and gained a rich understanding of test automation at scale along the way.

If a test fails in the woods and no one is there to see it does anyone care, does anyone even notice. What happens when failing tests become the norm and you can't see the wood from the trees? 

After watching last years Allure Report presentation I was inspired.  Selenium tests (and automation tests in general) are often poorly understood by the team as a whole.  Reports/emails go unread with tests failing becoming an expected outcome rather than a glaring red flag.  We looked at what Allure brought to the table and from that base created a dashboard which was designed to:

• Display the results of test runs in a way that was useful to managers, testers and the rest of the development team.  Including tools to filter out specific test runs and view the overall trend of the test run results.
• Make debugging tests easier by grouping errors, displaying history of test results, filtering tests and offering visual comparison of test runs.
• Help mitigate the problems flaky tests cause with test run result reporting (say that three times fast).
• Help with our mobile device certification process, by easily providing a view to compare test runs across devices.

Since it's creation the dashboard has been used and praised by managers through to developers.  With our full suite of tests from unit to integration to selenium and appium being stored on the dashboard.  We've managed to:

• Decrease the time taken to debug test cases.
• Increase the visibility of all our test suites, with managers having a better idea of how our selenium test suite is progressing and testers better understanding the coverage of unit tests.
• Focus the organization on quality.

We are working with legal at present to have this project open sourced and available to all prior to Selenium Conf 2015.

Almost every manual QA started with Selenium IDE or at least tried it at some point.. retention rate is close to 0.

In this talk we will depict the challenges that Test Automation Developers face. We'll categorise the challenges, name them, and see the skillset required to overcome the difficulties.

We'll compare the different Record/Playback tools and see where they excel, and where we must resort to traditional solutions (e.g. generic programming languages, and the Page Object design).

This talk will discuss topics such as Reuse of code, Component Isolation, Separation of Concerns, APIs, where test fragility stem from and how to overcome it, etc..

• `docker-selenium` project is about packaging selenium grid as docker containers (https://github.com/seleniumhq/docker-selenium)
  To me this means I don't have to build any selenium infrastructure machines. I just run the provided images by docker-selenium project (https://hub.docker.com/r/selenium/).
• I don't have to install selenium jar, java, browsers and other runtime dependencies. They are already built in a docker image and I can just run them as either selenium grid with hub and nodes or as standalone selenium on any docker engine enabled vm. 

In this talk/demo/case study I will show you how you can use `docker-selenium` project to build several pipelines starting from running on your local dev box to public cloud for quick tests and finally to a stable private cloud for your team.

Protractor, the testing framework for Angular built upon WebDriverJS, provides additional WebDriver-like functionality to test Angular based websites. But since it is written in Javascript it has been limited to users of WebDriverJS. In this talk I will outline my efforts as well as the efforts of others to bring Protractor like functionality to Python, Java, and other WebDriver ports.

Automated visual testing is a major emerging trend in the dev / test community. In this talk you will learn what visual testing is and why it should be automated. We will take a deep dive into some of the technological challenges involved with visual test automation and show how modern tools address them. We will review available Selenium-based open-source and commercial visual testing tools, demo cutting edge technologies that enable running cross browser and cross device visual tests at large scale, and show how visual test automation fits in the development / deployment lifecycle.

If you don't know what visual testing is, if you think that Sikuli is a visual test automation tool, if you are already automating your visual tests and want to learn more on what else is out there, if you are on your way to implement Continuous Deployment or just interested in seeing how cool image processing algorithms can be, this talk is for you!

The JSON Wire Protocol (JSONWP) is the version of the WebDriver spec currently implemented by all the Selenium clients. It defines an HTTP API that models the basic objects of web automation---sessions, elements, etc... The JSON Wire Protocol is the magic that powers Selenium's client/server architecture, enables services like Selenium Grid or Sauce Labs to work, and gives you the ability to write your test scripts in any language.

The JSONWP has served Selenium faithfully for a number of years, but the future of automated testing lies beyond the borders of the web browser. Mobile automation is an essential ingredient in any build, and tools like Appium or Selendroid have made it possible to run tests against mobile apps using the JSONWP. The JSONWP's current incarnation isn't enough to automate all the new behaviors that mobile apps support, however. Complex gestures, multiple device orientations, airplane mode, and the ability to use both native and web contexts, for example, are all essential to mobile automation.

For this reason the leaders of the Selenium project, in concert with other Selenium-based projects like Appium and Selendroid, met to discuss the future of the JSONWP. We've been working on its next version, called the "Mobile JSON Wire Protocol" (MJSONWP). Appium and Selendroid already implement much of the MJSONWP spec. In this talk I'll dive into the specifics of the MJSONWP extensions, how they relate to the original JSONWP, and how the Selenium clients have begun to implement them.

Finally, I will talk about the future of the MJSONWP and how it's related to the current and future versions of the WebDriver spec. I'll share how you can get help with the creation of the MJSONWP, and discuss issues with the authors of the new spec before the API is set in stone. We need the help of everyone who's involved in mobile automation to come up with the best and most future-proof version of the MJSONWP. Ultimately, your understanding of how Selenium works will be improved, and you'll have a much better handle on how projects like Appium and Selenium work together to make sure you have the best automation methods available.

There has been a recent explosion in second-screen technologies such as Chromecast, but designing test automation for second-screen applications is far from straightforward. This new paradigm lacks major automated tool support, and coordinating test execution across multiple devices is tricky and error-prone.

Our automation solution uses WebdriverJS and WebSockets to perform end-to-end test automation that covers our web player controller and second screen application.

Learn about our approach to second-screen automation which we’ve used to build a reactive, responsive test suite. We’ll describe our solutions to synchronizing test flow between the controller and target device, validation on the device, targeting different integration components, and device management.

Speed of UI automation has always been an issue when it comes to Continuous Integration / Continuous Delivery. If UI automation suite takes 3 hours to complete, then any commit happens during this time will not be visible in test environment, because the next deployment will happen only after 3 hours. 

With 2000+ developers and average 250+ checkins per day, the above issues is replicated 250+ times every day. This is not productive and feedback cycle is super slow!

Another issue is , with 35+ different project teams using 10 or more different jenkins jobs to run their UI automation. So many jobs means (350+), individual teams need to go through the pain of managing their own jenkins job, its a duplicate effort and waste of time. Automation teams need to spend time on writing reliable automation and not managing jenkins jobs.

Solution is to reduce the UI automation run time from hours to minutes and also use only handful of jobs to run the Distributed Automation!

Goal: To run all UI automation scenarios within the time take by the longest test case

Today, mobile is increasingly trumping web as the most important brand engagement point; enterprises are moving away from mobile and web projects independent of each other. The rapid adoption of responsive web encourages teams to discover one approach to measuring software quality regardless of form factors.

Selenium is current market leading solution for web testing, but how does it stand with mobile? The truth is that working with Selenium presents a few challenges, including:

• Building and maintaining an internal structure to support it
• Bridging an architectural gap
• Requirements demand support for unattended test execution
• Lack of real network conditions for mobile testing

There is a solution to address these challenges!

Let’s work through a demo and show how to test mobile & web in parallel with Selenium

A big blocker for putting a website on truly continuous production delivery is the amount of time it take to validate that the site works correctly.  Tests themselves take time to run, and test results are unreliable to the point where it takes a human to investigate and interpret them.  When counting the time that it takes to both run and interpret results, test runs for an enterprise web site can take an entire day from inception to useful result.

This session describes common points of failure in test execution that add both latency and unreliability and what can be done to overcome them while still preserving the value of UI validation.  We'll discuss why, after addressing these concerns, UI can be unblocked to reliably field thousands of validation scenarios on a local machine in a matter of minutes. 

Manual Testing.  Depending on how you've been influenced by those two simple words, reactions vary from slight disgust to full-on depression.  Of course, the solution is clear: automate, but how do you get there when your company is continually pushing out the next big feature?  As the set of features to cover increases, the lack of scalability of manual testing becomes more apparent.

This is a problem that we struggled with at our company.  Automation tactics were explored and implemented, but problems persisted as proposed solutions did not cater to the demands of the manual testers.

After years of failure and disappointment, our latest stint resulted in success.  Not only do we have hundreds of automated tests across various platforms (mobile and web) and products, but manual testing has been eliminated with zero casualties.  As we move forward towards Continuous Delivery and improved automation performance, we wanted to take this moment to look back and share stories of failure and success.

The Page Object Model is awesome, but you can make it even better with Dependency Injection.  I'll show you how to get started and some benefits of using Spring IOC in your Page Objects.  Though the concepts of this talk can be extended to any language, the examples will be shown in Java.

Every language and framework which lives as long as Selenium has its fuckups.. and we're here to embrance them and joke about them.

E.g. JS Wat https://www.youtube.com/watch?v=FqhZZNUyVFM

Comprehensive functional testing is a widely accepted best practice and Selenium is the prevalent tool of choice. But long-running Selenium test suites cannot be fit into short continuous integration (CI) cycles and are often run once a night or less. Running functional tests less frequently means bugs are discovered later than they should. Tanay Nagjee will discuss how Selenium test suites can be parallelized at a very fine granularity and included in CI builds. By leveraging a cluster of compute horsepower (on-premise and/or cloud), large Selenium suites can execute in a fraction of the time by smartly parallelizing the individual tests and running them on individual *cores* (not hosts). Tanay will outline the approach and tools to achieve these results with Selenium, and will present a live demonstration.

Oliver Wendell Holmes once said: I would not give a fig for the simplicity this side of complexity, but I would give my life for the simplicity on the other side of complexity.

Tool evangelists around the world have been using this phrase for selling their products. They make it a point to look into customer’s eyes and say “Scalability, you know, that’s the main problem with Selenium. What about 3 years down the line, when you have multiple applications in your landscape and Selenium does not support it?”

But they simply ignore the fact, knowingly off course, that this ever belting technology world is going where Selenium is right now. Applications are shrinking into browser windows and changing tracks to align itself to this mobile era.

Even commercial tool like TestComplete from SmartBear has started supporting Selenium and many will follow soon. Reason for this change is not only that most of the organizations are preferring open source tools like Selenium for starting point of their automation activities but also the fact that Selenium by far has proven itself to be one of the best automation tools when it comes to mobile or browser based desktop automation.

Here our aim is to display how seamlessly Selenium integrates with TestComplete and QAComplete and it is for us to understand that it is not Selenium which needs other tools to extend it but it is the other way round.

Element Locator strategies for Selenium WebDriver are highly flexible, and have been later inherited by many commercial tools. Although the locator strategies are flexible, they are also limited in a sense that, Selenium WebDriver does not currently allow its users to identify/filter UI elements with multiple locator strategies(at a time), as many commercial tools do.

The solution discussed in this article describes a library that allows Selenium WebDriver users to extend the Selenium element locator strategies for Element Filtering and few use cases for the library.

The solution approach allows users to continue to use the existing UI Element definitions in their tests, and extend them, using the By reference. The library will replace the existing Selenium WebDriver “By” reference.

Filtering based on multiple locator strategies

There are various scenarios where to uniquely identify an UI element, a complex XPath has to be written. However, the element can be identified uniquely using multiple locator strategies for the UI Element. The UI Elements can also be filtered, when there are multiple matches in a page. This is the UI Element recognition mechanism used in many commercial test automation tools.

The algorithm for filtering UI Elements based on multiple locator strategies is based on priority of locator strategies. The priority of locator strategies when filtering is:

1. ID
2. Name
3. TagName
4. ClassName
5. XPATH
6. LinkText and PartialLinkText
7. CSS

The By.elementFilter method takes multiple locator strategies, and searches the page for elements matching a particular locator strategy/property, and checks if it is a unique match on the page, if not then it uses the next locator strategy passed to it and so on.

This method is also very helpful when the application undergoes constant changes and UI Elements might have either of XPATH, ID , NAME, TagName, ClassName etc still unchanged. That way, it helps reduce a lot of maintenance effort in Selenium WebDriver implementations which is due to UI element changes.

Filtering based on Index

When there are multiple similar UI Elements in a page, such as cells in a grid/table, it makes sense to identify objects based on their Index based on their appearance on the web page.

The By.indexFilter method allows users to define an UI Element based on its Index of occurrence of the UI Element. The Index starts from 1.

Filtering based on relative element

When a UI element cannot be identified uniquely and reliably by any of its properties, but has some elements in its hierarchy or relative to a particular element, this method can be used to identify the element

The By.relationFilter method allows users to define an UI Element in relation to another element. The relation can be defined as “Left”, ”Right”, ”Top”, ”Bottom”, ”Child”, ”Parent”

Filtering for Tables

When dealing specifically with Tables, which have the

html tag, the By.tableFilter method allows the user to quickly identify specific cells in the table, without having to write complex XPaths or logics to achieve the same.

The By.tableFilter method allows users to define a cell in the table with Row,Column numbers. This allows users to directly use the UI Element in their code instead of writing their logic each time. This also increases efficiency and readability of the code.

The evolution of Test Automation started with automation tools that had record and playback features.  This allowed Automation Testers to record and structure the script in such a way that it could be reused. Tools like Selenium, which provided APIs, could interact with different browsers. The Automation Testers could use these APIs to interact with web applications. Additionally, it was possible to develop frameworks for reusing each of the components of the framework. Currently, the focus has shifted more and more towards the designing of frameworks rather than just the tools, so that the testing framework could be integrated with test management applications and continuous integration tools to aid test-driven development.

With that background, we have come up with certain Value Added Services (VAS), a step ahead of developing functional automation scripts. Imagine creating an Automation Framework which will not just check for the functionality of the application, but also check for security, page performance, page layout, accessibility and have an output that can be a trigger to other aspects of testing.

This paper presents three of the Value Added Services that we offer. We are working on creating many more such services on top of the Automation Framework.