The key objectives of organizations is to provide / derive value from the products / services they offer. To achieve this, they need to be able to deliver their offerings in the quickest time possible, and of good quality!

In order for these organizations to to understand the quality / health of their products at a quick glance, typically a team of people scramble to collate and collect the information manually needed to get a sense of quality about the products they support. All this is done manually.

So in the fast moving environment, where CI (Continuous Integration) and CD (Continuous Delivery) are now a necessity and not a luxury, how can teams take decisions if the product is ready to be deployed to the next environment or not?

Test Automation across all layers of the Test Pyramid (be it Selenium-based UI tests, or, xUnit based unit tests, or, Performance Tests, etc.) is one of the first building blocks to ensure the team gets quick feedback into the health of the product-under-test. 

The next set of questions are:
    •    How can you collate this information in a meaningful fashion to determine - yes, my code is ready to be promoted from one environment to the next?
    •    How can you know if the product is ready to go 'live'?
    •    What is the health of you product portfolio at any point in time?
    •    Can you identify patterns and do quick analysis of the test results to help in root-cause-analysis for issues that have happened over a period of time in making better decisions to better the quality of your product(s)?

The current set of tools are limited and fail to give the holistic picture of quality and health, across the life-cycle of the products.

The solution - TTA - Test Trend Analyzer

TTA is an open source product that becomes the source of information to give you real-time and visual insights into the health of the product portfolio using the Test Automation results, in form of Trends, Comparative Analysis, Failure Analysis and Functional Performance Benchmarking. This allows teams to take decisions on the product deployment to the next level using actual data points, instead of 'gut-feel' based decisions.

Typically in organizations, there are multiple projects / products. These products may be of implemented using tech-stacks over many years. Yet - they interact with each other in some way. To manage the complexity around Test Automation, many organizations prefer to have a common Test Automation solution across these products in an effort to build, standardize and maintain the framework.

However, this is not a good idea! With this approach one potentially ends up having to compromise on the quality of automation that can be done for each product, limited by the toolset.

The better approach would be to use the tools and technologies that are "right" for each product. This does have other disadvantages, but you would ensure each product is well tested! The only missing piece which remains is that these different products talk with each other. You need to test the integration between them in an automated way to verify all is well.

"TaaS" is an open-source product solution that allows you do achieve the "correct" way of doing integration testing across a variety of products via Test Automation.

Example: 

For one set of products, Selenium-based toolset may be the right choice, where as for legacy reasons, QTP may be used for some other product. With TaaS - you will be able to automate the Integration Testing between these products, by re-using the tests already implemented in the individual product suites.

If a test fails in the woods and no one is there to see it does anyone care, does anyone even notice. What happens when failing tests become the norm and you can't see the wood from the trees? 

After watching last years Allure Report presentation I was inspired.  Selenium tests (and automation tests in general) are often poorly understood by the team as a whole.  Reports/emails go unread with tests failing becoming an expected outcome rather than a glaring red flag.  We looked at what Allure brought to the table and from that base created a dashboard which was designed to:

• Display the results of test runs in a way that was useful to managers, testers and the rest of the development team.  Including tools to filter out specific test runs and view the overall trend of the test run results.
• Make debugging tests easier by grouping errors, displaying history of test results, filtering tests and offering visual comparison of test runs.
• Help mitigate the problems flaky tests cause with test run result reporting (say that three times fast).
• Help with our mobile device certification process, by easily providing a view to compare test runs across devices.

Since it's creation the dashboard has been used and praised by managers through to developers.  With our full suite of tests from unit to integration to selenium and appium being stored on the dashboard.  We've managed to:

• Decrease the time taken to debug test cases.
• Increase the visibility of all our test suites, with managers having a better idea of how our selenium test suite is progressing and testers better understanding the coverage of unit tests.
• Focus the organization on quality.

We are working with legal at present to have this project open sourced and available to all prior to Selenium Conf 2015.

Selenium WebDriver is a great tool, but it's not a testing library. It's a browser manipulation tool. There is a gap. 

Selenide adds a possibility for easy and stable testing.

Why yet another Selenium wrapper?

There are several testing libraries around Selenium webdriver. But it seems that they do not resolve the main problems of UI tests. Namely, instability of tests caused by dynamic content, JavaScript, Ajax, timeouts etc. Selenide was created to resolve these problems. With Selenide, you can forget all these common timing issues and concentrate on business logic.

Speed of UI automation has always been an issue when it comes to Continuous Integration / Continuous Delivery. If UI automation suite takes 3 hours to complete, then any commit happens during this time will not be visible in test environment, because the next deployment will happen only after 3 hours. 

With 2000+ developers and average 250+ checkins per day, the above issues is replicated 250+ times every day. This is not productive and feedback cycle is super slow!

Another issue is , with 35+ different project teams using 10 or more different jenkins jobs to run their UI automation. So many jobs means (350+), individual teams need to go through the pain of managing their own jenkins job, its a duplicate effort and waste of time. Automation teams need to spend time on writing reliable automation and not managing jenkins jobs.

Solution is to reduce the UI automation run time from hours to minutes and also use only handful of jobs to run the Distributed Automation!

Goal: To run all UI automation scenarios within the time take by the longest test case

Traditional approach in performance testing does not include client side processing time (i.e. DOM Content Load, Page Render, JavaScript Execution, etc.) as part of response times, performance tests has always been conducted to stress the server so tools like Jmeter have been very popular to execute tests. With increasing complexity of architectures (Web, Browser, Mobile) on the client side it has been important to understand the real user experience.   Commercial tools have started to provide features that can provide insights into real user experience after the bytes are transferred to the client end.  With the ability to call Selenium scripts via Jmeter the ability to conduct real user experience tests using open source stack has opened up new avenues to comment on real user experience.   This enables us to comment on

• Provides Page load times similar to On Load time of real browsers
• Generates HAR file with following statistics
• Details of summary of request times and content types
• Waterfall chart with page download time breakdown statistics such as  DNS resolution time, Connection time, SSL handshaking time, Request send time, wait time and receive time.

By integrating the open source stack tools it enables us to provide the same insights which a commercial of the shelf tools would offer.   At Gallop we have implemented this at multiple clients providing them insights into various bottlenecks at the client side which helped us to provide greater value proposition

A big blocker for putting a website on truly continuous production delivery is the amount of time it take to validate that the site works correctly.  Tests themselves take time to run, and test results are unreliable to the point where it takes a human to investigate and interpret them.  When counting the time that it takes to both run and interpret results, test runs for an enterprise web site can take an entire day from inception to useful result.

This session describes common points of failure in test execution that add both latency and unreliability and what can be done to overcome them while still preserving the value of UI validation.  We'll discuss why, after addressing these concerns, UI can be unblocked to reliably field thousands of validation scenarios on a local machine in a matter of minutes. 

Manual Testing.  Depending on how you've been influenced by those two simple words, reactions vary from slight disgust to full-on depression.  Of course, the solution is clear: automate, but how do you get there when your company is continually pushing out the next big feature?  As the set of features to cover increases, the lack of scalability of manual testing becomes more apparent.

This is a problem that we struggled with at our company.  Automation tactics were explored and implemented, but problems persisted as proposed solutions did not cater to the demands of the manual testers.

After years of failure and disappointment, our latest stint resulted in success.  Not only do we have hundreds of automated tests across various platforms (mobile and web) and products, but manual testing has been eliminated with zero casualties.  As we move forward towards Continuous Delivery and improved automation performance, we wanted to take this moment to look back and share stories of failure and success.

Every language and framework which lives as long as Selenium has its fuckups.. and we're here to embrance them and joke about them.

E.g. JS Wat https://www.youtube.com/watch?v=FqhZZNUyVFM

This talk aims to break 10 (amongst many) Myths that desperately need to be broken around Software testing and Testers. These so-called myths, have been around for so long that they are almost the norm. Breaking these myths is necessary to help budding as well as experienced Quality Analysts and teams, roll out better quality software products and build better team practices as well.

Visual Test Automation using Selenium

Visual Testing is the method of verifying that the application’s GUI appears correctly to its users. Most of the people say visual testing is hard to automate. Given the number of web browsers, operating systems, screen resolutions, responsive design, internationalization, etc.) the nature of visual testing can be complex. But with existing open source and commercial solutions, this complexity is manageable, making it easier to automate than it once was, since verification with traditional automated functional testing tools can be very challenging.

It can be easily achieved by integrating Selenium with Applitools. This talk mainly focuses on verifying the application’s graphical user interfaces (GUI) and finding the visual bugs using Applitools. It is very helpful for all sites having graphical functionalities like (charts, graph, dashboards etc).  Verify that the GUI appears correctly across all devices & browsers. The nature of visual testing can be complex. But with existing open source and commercial solutions, this complexity is manageable, making it easier to automate than it once was. And the payoff is well worth the effort.

Take pressure off manual QA: increase coverage, test faster & more accurately.  Reduce maintenance efforts: automatically propagate changes across execution environments. Release faster, with confidence & flawless.

Applitools Eyes Express captures the screen you want to test, and compares it to a baseline image – instantly, with a single click. No extra testing code necessary, no boring error logs.

For example, a single automated visual test will look at a page and assert that every element on it has rendered correctly. Effectively checking hundreds of things and telling you if any of them are out of place. This will occur every time the test is run, and it can be scaled to each browser, operating system, and screen resolution you care about.

Put another way, one automated visual test is worth hundreds of assertions. And if done in the service of an iterative development workflow, then you’re one giant leap closer.

Each of these tools follows some variation of the following work flow:

1. Drive the application under test (AUT) and take a screenshot
2. Compare the screenshot with an initial “baseline” image
3. Report the differences
4. Update the baseline as needed

Comprehensive functional testing is a widely accepted best practice and Selenium is the prevalent tool of choice. But long-running Selenium test suites cannot be fit into short continuous integration (CI) cycles and are often run once a night or less. Running functional tests less frequently means bugs are discovered later than they should. Tanay Nagjee will discuss how Selenium test suites can be parallelized at a very fine granularity and included in CI builds. By leveraging a cluster of compute horsepower (on-premise and/or cloud), large Selenium suites can execute in a fraction of the time by smartly parallelizing the individual tests and running them on individual *cores* (not hosts). Tanay will outline the approach and tools to achieve these results with Selenium, and will present a live demonstration.

Element Locator strategies for Selenium WebDriver are highly flexible, and have been later inherited by many commercial tools. Although the locator strategies are flexible, they are also limited in a sense that, Selenium WebDriver does not currently allow its users to identify/filter UI elements with multiple locator strategies(at a time), as many commercial tools do.

The solution discussed in this article describes a library that allows Selenium WebDriver users to extend the Selenium element locator strategies for Element Filtering and few use cases for the library.

The solution approach allows users to continue to use the existing UI Element definitions in their tests, and extend them, using the By reference. The library will replace the existing Selenium WebDriver “By” reference.

Filtering based on multiple locator strategies

There are various scenarios where to uniquely identify an UI element, a complex XPath has to be written. However, the element can be identified uniquely using multiple locator strategies for the UI Element. The UI Elements can also be filtered, when there are multiple matches in a page. This is the UI Element recognition mechanism used in many commercial test automation tools.

The algorithm for filtering UI Elements based on multiple locator strategies is based on priority of locator strategies. The priority of locator strategies when filtering is:

1. ID
2. Name
3. TagName
4. ClassName
5. XPATH
6. LinkText and PartialLinkText
7. CSS

The By.elementFilter method takes multiple locator strategies, and searches the page for elements matching a particular locator strategy/property, and checks if it is a unique match on the page, if not then it uses the next locator strategy passed to it and so on.

This method is also very helpful when the application undergoes constant changes and UI Elements might have either of XPATH, ID , NAME, TagName, ClassName etc still unchanged. That way, it helps reduce a lot of maintenance effort in Selenium WebDriver implementations which is due to UI element changes.

Filtering based on Index

When there are multiple similar UI Elements in a page, such as cells in a grid/table, it makes sense to identify objects based on their Index based on their appearance on the web page.

The By.indexFilter method allows users to define an UI Element based on its Index of occurrence of the UI Element. The Index starts from 1.

Filtering based on relative element

When a UI element cannot be identified uniquely and reliably by any of its properties, but has some elements in its hierarchy or relative to a particular element, this method can be used to identify the element

The By.relationFilter method allows users to define an UI Element in relation to another element. The relation can be defined as “Left”, ”Right”, ”Top”, ”Bottom”, ”Child”, ”Parent”

Filtering for Tables

When dealing specifically with Tables, which have the

html tag, the By.tableFilter method allows the user to quickly identify specific cells in the table, without having to write complex XPaths or logics to achieve the same.

The By.tableFilter method allows users to define a cell in the table with Row,Column numbers. This allows users to directly use the UI Element in their code instead of writing their logic each time. This also increases efficiency and readability of the code.

In the modern era, we have many different cloud testing services to choose from. These cloud services are useful and help reduce the burden of building and maintaining your own Selenium Grid environment. However, there are many scenarios in which you need your tests running locally and quickly, such as you work for the government (or agency), you have sensitive software/data you cannot expose to the cloud, or service costs are too expensive for your organization.

This presentation will feature getting started with setting up your own mobile device grid, running your tests in parallel, running in CI (Jenkins), and the lessons I have learned along the way.

With all the focus on Test practices and tools getting better for the present, it is easy to forget about the Future of Testing, especially in terms of Tools and Infrastructure? A question that always comes up in my mind - “Are we so caught up in the past and present, that we will not be as effective in the future?”

In this talk, we will go on a journey to figure out what new challenges are coming up in the future, and more importantly, what do we need to do next to prepare for it. Also, just preparing for the future is not sufficient. We have an opportunity to stretch beyond our current set of skills, capabilities and boundaries to influence out future! The question is - will we make use of this opportunity? Will Selenium help us take the step in the right direction? Or, will it hold us back?

When we talk about enterprise level automation where in there are multiple technology stack involved it becomes exceedingly difficult to automate every application using the same tool. Selenium web driver has become a defacto standard when it comes to web application automation and no body would prefer any other tool over it when it comes to web. But what about the other applications, We may have some SAP/ some .NET applications that are linked to the web application and needs to be automated as well. Lets go for some other tool here, but oops this tool does not talk java (which is what the automation team uses here). Basically we need to maintain two different automation teams with two different automation framework which is defenitely and overhead.

The talk is about

1. How to over come this scenario

2. Rabbit MQ

3. Integrating the MQ server with Grid

4. Other tool (other language binding) integrated to selenium

5. Building java COM bridge

6. Developing custom bindings to your tools which by default does not have them

Page-Object pattern is very commonly used when implementing Automation frameworks. However, as the scale of the framework grows, there is a limitation on how much reusability really happens. It inherently becomes very difficult to separate the test intent from the business domain.

Learn more about this problem, and the solution (by means of real-project examples) - Business Layer - Page - Object pattern, which has helped me keep my code DRY.