Q & A with the Selenium Committee moderated by Anand Bagmar

It’s 2020 and a new decade awaits us. 100 years ago, our world was famous for the Russian Revolution and World War 1.

In our century, we are also facing tremendous upheaval. The bushfires in Australia, floods in Indonesia. We are seeing the impact of climate change before our eyes.

Technology-wise, agile has become mainstream. Hardware is cheap, CI and CD are commonplace. Robotics and Machine Learning is a reality. This is the age we live in. This is the now.

How do we begin to plan for such a reality? How do we test software in such an ecosystem? Should we even try? How do we maintain skills, when frameworks change as rapidly as nail varnish on a teenager?

As leaders in software testing, these are the questions we need to be able to answer. We need leadership now more than ever. We can’t predict the future, but we can prepare ourselves to be able to deal with change, to develop strategies that facilitate rapid learning and rapid change.

We can upskill our people to be able to handle inconsistency and complexity. To know when technology is beneficial and when to rely on our ability to think critically.

We must create environments where talent can grow and thrive. We learning (and failure) are embraced. Today is the day, we become the test leaders of the future. Are you with me?

Selenium and Sahi started as open source tools around the same time and from the same company. Both tried to solve the problem of UI automation and both have been successful in their own ways. But the resemblance ends there. Selenium became Webdriver and became one of the most popular free tools in testing. Sahi became commercial Sahi Pro and is successfully used in various organizations to automate complex UI's spanning web, desktop and mobile applications. Right from who the user of the tool is to whether the tool should be open source or commercial, there are various philosophical, technical and commercial junctures where Selenium and Sahi have diverged. Based on our journey of over 10 years, this talk reflects on those divergent decision points, the philosophy behind those and what their outcomes have been.

Since nearly the very beginning, the software industry has prized the concept of the visionary, the extraordinary creator, the lone person who relentlessly pursues their goal against impossible odds, all to help make the world a better place through technology. The industry also reveres the concept of dedication to the software above all else in the lives of those visionaries. Even those who have chosen to shepherd large portions of popular open-source projects are often seen as people who have a special quality about them that places them above other developers. These concepts are myths and oftentimes dangerous ones.

Anyone can make a pretty bar graph, but can you make sound decisions based on that graph? Is it actionable, or is it only a concept? How do you turn flashy concepts into actionable visualizations? Can you see the end result of those concepts; will they ever become reality? Do you have the vision and drive to figure out how to get from today to tomorrow before it becomes yesterday?

American mathematician John Tukey once said, "The greatest value of a picture is when it forces us to notice what we never expected to see." What value do you see in your data? And what ideas do you have when you see it? Learn how you can capitalize on your ideas and turn them into reality by blending internal with external, leveraging them into a cohesive strategy for both the short term AND the long term. See the five "Stages of the Spectrum" in action while discovering the difference between impact and influence, and how that difference plays into making data actionable.

Selenium has done a pretty good job in keeping the API surface friendly and usable, but what actually happens when you call some of the commands? In this talk, David will walk you through what happens when you make a call in your test, how it gets to the browser, what happens in the browser, and how it returns all the way back to your test.

Selenium is designed so that each of the commands works synchronously, so you know that a command has finished before it moves onto the next. This creates some interesting problems in browsers since they are mostly designed around asynchronicity.

We will start with how each of the bindings communicates with the browser and then move on to how navigation works. David will show all the different aspects that we need to figure out to tell if a page is “loaded”. He will also show where it goes horribly wrong and how you can write code in your tests to stabilise around these “anomalies”.

From there we will have a look at how clicks work from making sure they are trusted to what if they cause a navigation.

Problem Statement

Though Selenium taking most of the stake in the UI test automation tool market comfortably, there were always challenges that were for selenium test automation engineers are handicapped with especially when dealing with modern JS technologies. For example,

• dealing with DOM elements to solve the stale / loading / non-interactable elements,
• handling full screenshots to know how the elements at the left, bottom, etc,
• measuring the performance of request and response resources at different speeds,
• monitoring the memory of the pages, controls, etc,
• attaching to an existing browser for debugging the failed scripts and many more.

These problems were largely resolved with the integration of selenium with devtool protocols. And that makes the selenium engineer's life merrier than before.

Why this proposal can be different from others?

1. Our solution can be executed as independent with chrome dev tools or with selenium. That gives the power back to the automation engineer to choose what and how to debug/run their tests.
2. We used this solution for one of our largest enterprise customers and moved this solution to public repository this week (for this conference and beyond). With that said, we tested reasonably with more than 10,000+ test scripts and more than 1M tests.
3. The present solution is completely (100%) packed with all Chrome Dev Tools API in Java and with that said, any Java Selenium automation engineer can bind in minutes for their existing code base with no additional dependencies.
4. Finally, we love to present at the local home to start our selenium conference campaign. Where else?

Solution:

The present proposal largely connected with Chrome and Selenium in Java language. However, there is no limitation to expand the boundaries for other language bindings and browsers.

Google Chrome, the most picked browser for browsing, which makes it the primary concentration for developers and testers. DevTools is one such boon for developers, testers especially the new aged test automation engineers. With that said, we built the following design pattern to allow chrome dev tools API to marry Selenium using debugger address / remote targets.

Selenium Devtools

"This world is such a beautiful place to live in." If you can read the first sentence without any screen readers or assistance, you're privileged. As technologists, shouldn’t we be more empathetic towards differently-abled people and make all parts of our website accessible to them? In my humble opinion, the true power of technology can be identified when it reaches out to people of all kinds having different physical or psychological challenges. We not only legally bound to provide Accessibility but also it should be considered as our moral responsibility.

As testers, we have a wonderful opportunity to contribute to Accessibility by ensuring that the site is accessible in many different ways. Although it is impossible to identify all the issues that exist around Accessibility in the world, we are lucky enough to still understand a majority of them. With this understanding, there have been many measures taken in order to make your site accessible. But, don’t forget - we are QA engineers. We got to ensure that the site is accessible as per the standards set by WCAG 2.0 (AA) by testing for accessibility using various tools and techniques.

"All tests in today's automated regression run have been marked as Untested. What happened?"

"No notifications are being sent for test runs on the channel"

"I pulled latest code, and the framework dependency shows compilation error"

"What does this new method in the framework do?"

How often do you hear such things within your team?

As Quality champions, we need to walk the talk. When we expect our developers to write quality code, write unit tests, build features without introducing bugs, the onus lies on us (as test engineers) to do the same. With almost every test engineering team writing automated tests to check functionality of their products and services, it becomes very important to ensure that the test automation framework and the test scripts are bug-free and follow good standards of software engineering.

It cannot be stressed enough that test automation code should be as good as production code. In order to build production-quality test automation framework and scripts, a number of steps can be taken at:

1. Code & System Level

2. Process & People Level

Our test engineering team went through a transition from having random & unexpected failing test runs to having greater confidence in the quality of the tests. Learn from this case study of our journey to ensure that end-end UI automated tests are built with quality in mind. We will also see demonstration of some of the use cases.

With the seismic shift in industry and development of new technologies emerging, QA’s testing approaches are also changing, we must know the right strategies and algorithms to test. One of the latest technology emerging is Artificial Intelligence and Machine Learning. And its applications like Self driving cars, Virtual Assistants are everywhere. They have great impact in our life and most of our decisions, behaviour & destinations depend on them.

So in this presentation/Workshop i would like to present all the ways/strategies/ challenges faced while testing AI/ML applications. Join me in creating a Machine Learning application from scratch and then take it to testing stage, creating edge case scenarios and validations.

Time Management: To make sure that all people are upto date with with setup for hands-on, i will be sharing this document with the participants 12 days before in a temp slack channel, where they can share the progress and ask queries to resolve them quickly.
*No internet is required for participants if they follow the setup doc.

Selenium has become the de-facto standard framework for end-to-end web testing nowadays. JUnit 5 is the latest version of this popular testing framework and implements a brand-new programming and extension model named Jupiter. This talk presents Selenium-Jupiter (https://bonigarcia.github.io/selenium-jupiter/), an open-source JUnit 5 extension aimed to provide seamless integration with Selenium. At first glance, it allows creating Selenium tests using local or remote browsers, reducing the required boilerplate thanks to the dependency injection mechanism supported in JUnit 5. Moreover, it will enable us to use different types of web browsers and Android devices in Docker containers out of the box. All in all, Selenium-Jupiter can be used to carry out different types of tests for web and mobile applications, including functional, compatibility (using test templates to define different devices or browsers -types and versions-) or performance tests.

With an ever increasing no of businesses being conducted on web the testing need to write automated tests for the app's UI is something that can never be ignored. As you all know Selenium provides an API that enables us to do this quite effectively.

However, when tasked with setting up the automation framework, there are a lot of questions that arise in the minds of aspiring test developers regardless of what level they are in their career.

Some of such questions are:

1. How does one actually go about the business of building a robust and effective automation framework on top of selenium?
2. What are the elementary building blocks to include in the framework that an aspiring automation developer should know of?
3. How should we model our tests? XUnit style vs BDD?
4. Are there good practices, sensible design patterns and abstractions that we can follow in our code?
5. What are some of the anti patterns/common mistakes we should avoid

A lot of literature, documentation and blogs exists on these topics on the web already.

However In this talk,

I would combine this existing knowledge and my years of experience in building automation frameworks and breakdown these elements and walk you through exactly the sort of decisions/considerations and practices that you can take while starting to implement or improve the UI automation for your team.

Hope to see you there!

Shadow DOM is kind of web standard that developers use to encapsulate their custom HTML code and style components so that no other global style components can override their code. It ensures that a component will work in any environment, even if other CSS or JavaScript is run on the page i.e. Shadow DOM hides away the complexity of the components from the rest of the page because it renders separately from main DOM.

Selenium legacy selectors are not able to interact these custom elements under the shadow DOM so we need java script to interact with shadow DOM elements and we also require JavascriptExecutor interface for their execution.

We will also talk about the WebdriverIO tool v5.5.0, which implicitly support Shadow DOM elements.

So you have got a few UI tests and they are running in parallel, great! However, life will not be so sweet once these 'a few' turns into 'a lot'. We grew from a few to 1500 UI tests (although not particularly proud of this number, there are situations and reasons)

We started with a simple parallel distribution of tests 3 years ago. As test count increased failure count run time increased along with increased flaky tests. Mobile tests had their own challenges (eg. device dropping-off, random wi-fi issues, etc) To keep up with this, we created a queue and workers based solution which could distribute the tests more efficiently (https://github.com/badoo/parallel_cucumber). Over time, we made more improvements, in particular:

• Segregation of failures based on infrastructure issues and re-queue the tests
• If a device/emulator malfunction, rescue the tests to another device
• Repeating a single test on 100s of the worker in parallel to detect flakiness
• Repeat a test if a known network issue
• Terminating the build early if more than a certain number of tests have failed
• Health check of each device, before each test to ensure reliability
• Muting a test if failure is known, and highlight outdated mutes if the related task is fixed

In this talk, I will talk about the initial challenges with running UI tests in parallel (Selenium and Appium), how we approached the queue based solution and continuous improvement of this solution; finally, how attendees can use it at their workplace or create their own solution based on our learnings.

Browser tests are known to be the flakiest ones. This is partly because browser infrastructure is complicated to maintain. But the second reason is – mainstream browser automation tools such as Selenium server are far from being efficient.

During my previous talks I was speaking about Selenoid - a truly efficient replacement of the standard Selenium server. This year I would like to do a live demonstration how to organize a fault-tolerant and easily scalable Selenium cluster using virtual machines in the cloud. I will start by setting up Selenoid and show its powerful features like video recording, live tests debugging, manual testing and many more. Then I will configure Selenoid to send logs and recorded videos to S3-compatible storage. Finally we will run a Ggr load balancer instance allowing to use all running Selenoid nodes and organize a single entry point to the cluster.

The intelligent mesh of devices is nothing but connecting millions of users, use cases, apps, and devices together to support application developed based on the internet of things. Currently, we see around the world that there are thousands of use cases, millions of apps, billions of users and trillions of things and if you think QA and testing in the interconnected intelligent devices then the scope would be very wide as verification and validation will be applicable in each interface and as and when grows.
I will talk about the challenges faced during IoT application testing and how it can go wrong. To thorough test all the area with the given challenges and IoT test lab is setup. What all are the solution to overcome the challenges. Some important aspects like continuous integration of hybrid environment and testing with multiple devices and with millions of use cases, to improve the existing conventional method with intelligent automation, and most important is scalability and security.

Although testing emerging technology and applications always is exciting, seeing their own strategies and tools fail in IoT testing can be frustrating even for well-seasoned testers. I aim to provide testers with a better understanding of connected systems like smart cities, connected enterprise and help them to apply their critical thinking to deal with uncertainty in their test objects.

Onestic QA department made and maintained test suites, which was a huge bottleneck in our development process.

In order to solve it, management bought us an idea: our developers must be able to build their tests using a DSL framework without friction. QA maintain a big library that provides resources to developers. From there, they were free to extend this library in each project. With all of this we have a CI process, a 1 hour execution to monitor our results and of course our bot, SpongeBot.

SpongeBot can check 40+ e-commerce sites, with 4+ environments, for desktop and mobile platforms always available for developers.

With this solution we achieve to decentralize the work, add value testing production and the confidence in SpongeBot notifications if something goes wrong.

There are dozens of matured docker based solutions available in the market for Android automation testing,

but

what about iOS testing?

It’s hard to create and maintain the docker based solution for iOS testing since it demands the Xcode and for the optimal performance the system needs the Apple certified hardware.

Combining Ansible with Selenium Grid yields a powerful combination because it allows us to set-up our grid and nodes in just a few seconds. In this talk I will demonstrate how to use simulators to build a reliable and scalable in-house iOS simulator lab using Ansible, Selenium Grid and Appium.