Test Automation is considered the silver bullet that will enable teams to shift left and accomplish CI/CD. However, how confident are you with the value added by the automated tests in your organization and teams? Is your automation effective? Is it giving you the ROI you expected from it?

Likewise, how comfortable are you with the testing approach of your team, and how is it helping improve your product quality?

In the context of Quality, Testing, and Test Automation, for each role on the team, we will discuss the following:

• What are effectiveness and ROI?
• What are the commonly used metrics to measure effectiveness and ROI from your Testing & Test Automation activities? (Ex: code coverage, defect leakage, test execution status, test execution time, # of tests, etc.)
  • Do these metrics help in identifying blockers and root causes of issues?
  • How do these metrics help to make decisions to better product quality?

• Discuss other metrics that can help each role make meaningful decisions to improve product quality!
  • Ex: Cycle Time, CLT, defect analysis with RCA, MTTR, Functional coverage, etc.

Simulating real-user scenarios as part of your automation is a solved problem. You need to understand the domain, the product, the user, and then define and implement your scenario.

But there are some types of scenarios that are complex to implement. These are the real-world scenarios having multiple personas (users) interacting with each other to use some business functionalities. These personas may be on the same platform or different (web / mobile-web / native apps / desktop applications).

Example scenarios:

• How do you check if more than 1 person is able to join a zoom / teams meeting? And that they can interact with each other?
• How do you check if the end-2-end scenario that involves multiple users, across multiple apps works as expected?
  • Given user places order on Amazon (app / browser)
  • When delivery agent delivers the order (using Delivery app)
  • Then user can see the order status as "Delivered"

Even though we will automate and test each application in such interactions independently, or test each persona scenarios independently, we need a way to build confidence that these multiple personas and applications can work together. These scenarios are critical to automate!

In this session, I will demonstrate teswiz, an open-source framework that can easily automate these multi-user, multi-app, multi-device scenarios. I will also mention how to run these tests on local and in CI pipelines.

Example: Multi-user, Multi-device test scenario

Example: Multi-user, Multi-app, Multi-device test scenario

Teswiz enables, and guides you to implement your automated tests while adhering to the principles of test automation, like - independent tests that run in parallel, against multiple environments using environment-specific-test-data and generate rich and contextual reports (and test execution trends) in reportportal. 

Test coverage is increased by using Applitools Visual AI, along with Applitools Ultrafast Test Cloud. 

In addition, teswiz takes away the pain of managing your browsers, and android / ios / windows devices for automation. The automated tests can run on local browsers / devices, or against any cloud provider, such as HeadSpin, BrowserStack, SauceLabs, pCloudy.

The following features makes teswiz unique:

The ability to do:

• Multi-user scenario automation between all platforms (android, iOS, Web, Windows desktop applications)
• Managing browsers / devices and parallel execution automatically
• Completely configurable options - means no code change required to run different combinations of tests, on demand
• Integration with Applitools Visual AI and Applitools Ultrafast Test Cloud
• Rich contextual reports (including screenshots, browser / device logs) and trend analysis via ReportPortal.io

The following features are relatively new in teswiz:

• Ability to run test orchestration across different platforms, using different app(s) and different users
• Integration of Applitools Ultrafast Grid for scaling
• Support for Firefox & Safari (earlier it was only Chrome)
• Better driver management & cleanup

Are ALL of the below statements true regarding your functional automated tests?

• As a result of your functional automation, manual testing efforts are reduced.
• Automated tests run fast.
• Automated tests are triggered automatically on any product change.
• Automated tests can run, on-demand, on different (supported) browsers/devices without making any changes in code or configuration.
• The automated test execution needs no manual intervention to run/complete successfully.
• Your automated functional tests give deterministic feedback - i.e. no flaky/intermittent test failures.
• You have all the relevant information to do RCA of failed tests readily available.
• Consolidated reports and trends from test runs are available in a central reporting server for the team.
• The automated test code is clean and easy to maintain, reuse and scale.
• Your automated tests can run in any order, and in parallel.

If you have answered yes to all the above statements, you would be in the very small percentage of teams doing functional automation right!

In my experience, I have seen teams struggle with more than one, and in many cases, all of the above statements. 

On retrospection, I have realized, that most of these challenges are because of a combination of the following issues:

• Lack of holistic vision, understanding, skills, and capabilities for taking on this seemingly easy activity of functional test automation
• Lack of time for design, implementation, and maintenance of the automated test code

In this session, I will share, with examples, the features & capabilities that are not used right in Test Automation implementation resulting in you answering “NO” to the above statements. These are anti-patterns of functional automation implementation and have a negative impact on the quality of feedback from your automated tests.

More importantly, I will share approaches and solutions of how you can avoid these anti-patterns and evolve to answer “YES” to each of the above statements.

Have you heard of “flaky tests”?

There are a lot of articles, blog posts, podcasts, conference talks about what are “Flaky tests” and how to avoid them. 

Some of the ideas proposed are:

• Automatically rerun failed tests a couple of times, and hope they pass
• Automatically retry certain operations in the test (ex: retry click / checking for visibility of elements, etc.) and hope the test can proceed

Unfortunately, I do not agree with the above ideas, and I would term these as anti-patterns for fixing flaky / intermittent tests. 

We need to understand the reasons for flaky / intermittent tests. Some of these reasons could be because of issues like: 

• timing issues (i.e. page loading taking time)
• network issues
• browser-timing issues (different for different browsers / devices)
• data related (dynamic, changing, validity, etc.)
• poor locator strategy (ex: weird & hard-wired xpaths / locators)
• environment issue
• actual issue in the product-under-test

In this talk we will explore the anti-patterns and its impact. We will also look at ideas how you can actually reduce / eliminate the flakiness of your test execution.

Simulating real-user scenarios as part of your automation is a solved problem. You need to understand the domain, the product, the user, and then define and implement your scenario.

But there are some types of scenarios that are complex to implement. These are the real-world scenarios having multiple personas (users) interacting with each other to use some business functionalities. These personas may be on the same platform or different (web / mobile-web / native apps / desktop applications).

Example scenarios:

• How do you check if more than 1 person is able to join a zoom / teams meeting? And that they can interact with each other?
• How do you check if the end-2-end scenario that involves multiple users, across multiple apps works as expected?
  • Given user places order on Amazon (app / browser)
  • When delivery agent delivers the order (using Delivery app)
  • Then user can see the order status as "Delivered"

Even though we will automate and test each application in such interactions independently, or test each persona scenarios independently, we need a way to build confidence that these multiple personas and applications can work together. These scenarios are critical to automate!

In this session, I will demonstrate teswiz, an open-source framework can easily automate these multi-user, multi-app, multi-device scenarios. I will also mention how to run these tests on local and in CI pipelines.

Example: Multi-user, Multi-device test scenario

Example: Multi-user, Multi-app, Multi-device test scenario

Teswiz enables, and guides you to implement your automated tests while adhering to the principles of test automation, like - independent tests that run in parallel, against multiple environments using environment-specific-test-data and generate rich and contextual reports (and test execution trends) in reportportal. 

Test coverage is increased by using Applitools Visual AI, along with Applitools Ultrafast Test Cloud. 

In addition, teswiz takes away the pain of managing your browsers, and android / ios / windows devices for automation. The automated tests can run on local browsers / devices, or against any cloud provider, such as HeadSpin, BrowserStack, SauceLabs, pCloudy.

The following features makes teswiz unique:

The ability to do:

• Multi-user scenario automation between all platforms (android, iOS, Web, Windows desktop applications)
• Managing browsers / devices and parallel execution automatically
• Completely configurable options - means no code change required to run different combinations of tests, on demand
• Integration with Applitools Visual AI and Applitools Ultrafast Test Cloud
• Rich contextual reports (including screenshots, browser / device logs) and trend analysis via ReportPortal.io

The following features are relatively new in teswiz:

• Ability to run test orchestration across different platforms, using different app(s) and different users
• Integration of Applitools Ultrafast Grid for scaling
• Support for Firefox & Safari (earlier it was only Chrome)
• Better driver management & cleanup

Mobile test automation has come a long way in the past several years. Both Apple and Google have matured their respective frameworks. We have set industry standards in mobile development now that didn't exist just a few years ago. So do we really need to still scale our test execution across many devices/simulators/emulators, OS's, and resolutions our application supports? Doing so also raises the possibility for more points of failure across devices, cloud services, or internal grids.

For example, devices may lose internet connectivity, connection to the host machine/grid, lose power, reboot randomly, connectivity to cloud service may be interrupted, or our machine resources may not be ideal, thus all leading to flakey test behavior. Also, not to name all the possible conditionals we may need to add (and maintain) in our test code to handle these different devices. At the end of the day, all we really want is to know our application and its various UI views function and visually look accurate across the devices, OS's, and resolutions we support.

In this talk, I'll go over where we've come from and the pain points of mobile UI automation, and where we're heading. I will demo some of the new tools available on the market to help us get there.

Appium released its first major version almost 7 years ago. Since then, Appium has rolled out a lot of new features and its automation backend architecture has evolved quite a lot. 

Appium 2.0, the next major release, is almost here! It is currently available for testing in beta (v12).

Attend this this workshop if the answer to any of the below questions is "YES":

• Do you want to start Mobile app Automation using Appium but do not know where and what to start with?
• Did you start with Mobile app Automation using Appium, but got stuck among understanding capabilities, locating elements, finding whats wrong?
• Are you curious to know the changes in Appium 2.0 and want to try it out?

If you have answered Yes to one or more of the above questions, then this workshop, precisely designed for beginners to quickly get started with Appium 2.0, is for you!

At this workshop, you will learn everything from scratch, step by step. We will together understand the key concepts, apply them right there and write our “first test” in appium before we learn more nitty-gritties as takeaway for you to go and implement at your actual work directly.

This is a one-day crash course in Appium. The course will be prepared in Java.

Have you heard of “flaky tests”?

There are a lot of articles, blog posts, podcasts, conference talks about what are “Flaky tests” and how to avoid them.

Some of the ideas proposed are:

• Automatically rerun failed tests a couple of times, and hope they pass
• Automatically retry certain operations in the test (ex: retry click / checking for visibility of elements, etc.) and hope the test can proceed

Unfortunately, I do not agree with the above ideas, and I would term these as anti-patterns for fixing flaky / intermittent tests.

We need to understand the reasons for flaky / intermittent tests. Some of these reasons could be because of issues like: 

• timing issues (i.e. page loading taking time)
• network issues
• browser-timing issues (different for different browsers / devices)
• data related (dynamic, changing, validity, etc.)
• poor locator strategy (ex: weird & hard-wired xpaths / locators)
• environment issue
• actual issue in the product-under-test

In this session, with the help of demos, we will look at the following techniques you can use to reduce / eliminate the flakiness of your test execution:

• Reduce number of UI tests
• Use Visual Assertions instead of Functional Assertions
• Remove external dependencies via Intelligent Virtualization

Demo will be done using the following and sample code will be shared with the participants

• Specmatic
• teswiz
• Applitools Eyes

The Test Automation Pyramid is not a new concept.

The top of the pyramid is our UI / end-2-end functional tests - which simulate end-user behavior and interactions with the product-under test.

While Automation helps validate functionality of your product, aspects of UX validations can only be seen and captured by the human eye and is hence mostly a manual activity. This is an area where AI & ML can truly help.

With everyone wanting to be Agile, make quick releases, the look & feel / UX validation, which is typically a a manual, slow, and error-prone activity, quickly becomes a huge bottleneck.

In addition, with any UX related issues propping up cause huge brand-value and revenue loss, may lead to social-trolling and worse - dilute your user-base.

In this hands-on workshop, using numerous examples, we will explore:

In this workshop, using numerous examples, we will explore:

• Why Automated Visual Validation is essential to be part of your Test Strategy
• How Visual AI increases the coverage of your functional testing, while reducing the code, and increasing stability of your automated tests
• Potential solutions / options for Automated Visual Testing, with pros & cons of each
• How an AI-powered tool, Applitools Eyes, can solve this problem.

The Test Automation Pyramid is not a new concept.

The top of the pyramid is our UI / end-2-end functional tests - which simulate end-user behavior and interactions with the product-under test.

While Automation helps validate functionality of your product, aspects of UX validations can only be seen and captured by the human eye and is hence mostly a manual activity. This is an area where AI & ML can truly help.

With everyone wanting to be Agile, make quick releases, the look & feel / UX validation, which is typically a a manual, slow, and error-prone activity, quickly becomes a huge bottleneck.

In addition, with any UX related issues propping up cause huge brand-value and revenue loss, may lead to social-trolling and worse - dilute your user-base.

In this hands-on workshop, using numerous examples, we will explore:

• Why Automated Visual Validation is essential to be part of your Test Strategy
• How Visual AI increases the coverage of your functional testing, while reducing the code, and increasing stability of your automated tests
• Potential solutions / options for Automated Visual Testing, with pros & cons of each
• How an AI-powered tool, Applitools Eyes, can solve this problem
• Hands-on look at Applitools Visual AI and how to get started using it

Very often we work on a code-base that has been written by others, and some time ago. This code-base could be for the product code, or Test Automation code. 

As the product life increases, evolution of the code-base is a natural process. However, there are various catalysts to speed up this evolution process:

• More features / tests to be added, including increased complexity
• People writing the code evolve - their learning, skillset
• Delivery pressure means it is quite possible that correct decisions for implementation may not be taken. In other words, it is possible that short-cuts were taken in the implementation leading to spaghetti code / architecture

People move on to different roles, new people join the team. Each has different opinions, perspectives and experiences.

Am sure there are more reasons you can think of.

Regardless, the challenge for a new person who starts working on such a complex code-base is enormous - as the person needs to start delivering "value".

In this session, I will share various examples and experiences and as a result of being in such situations, the factors I looked at when enhancing the code-base to decide - should I refactor or rewrite the code-under-consideration to be able to move forward faster, while moving towards the long-term vision.

Though I will focus on various examples of Test Automation, this session is applicable for any role that writes / maintains code of any nature.

Have you heard of “flaky tests”?

There are a lot of articles, blog posts, podcasts, conference talks about what are “Flaky tests” and how to avoid them.

Some of the ideas proposed are:

• Automatically rerun failed tests a couple of times, and hope they pass
• Automatically retry certain operations in the test (ex: retry click / checking for visibility of elements, etc.) and hope the test can proceed

Unfortunately, I do not agree with the above ideas, and I would term these as anti-patterns for fixing flaky / intermittent tests.

We need to understand the reasons for flaky / intermittent tests. Some of these reasons could be because of issues like: 

• timing issues (i.e. page loading taking time)
• network issues
• browser-timing issues (different for different browsers / devices)
• data related (dynamic, changing, validity, etc.)
• poor locator strategy (ex: weird & hard-wired xpaths / locators)
• environment issue
• actual issue in the product-under-test

In this session, with the help of demos, we will look at the following techniques you can use to reduce / eliminate the flakiness of your test execution:

• Reduce number of UI tests
• Use Visual Assertions instead of Functional Assertions
• Remove external dependencies via Intelligent Virtualization

Demo will be done using the following and sample code will be shared with the participants

• Specmatic
• teswiz
• Applitools Eyes

Q & A with the Selenium Committee moderated by Anand Bagmar

Challenges in Testing & Automating Games

Testing real-time, multi-user games built for native apps and / or browser-based on phones / mobile devices / tablets / desktop browsers makes testing of regular products as apps, or websites appear like a piece of cake. Testing such real-time and multi-user games becomes even more challenging when you think about automating the same.

I got an opportunity to build Functional Test Automation for a suite of games – and what an exciting time it turned out to be!

These games are built either using Cocos2d-x, or Unity (cross platform game engines for mobile games, apps and other cross platform interactive GUI and are known for their speed, stability, and ease-of-use).

The key challenges I encountered here were:

• Millions of users, playing games on a huge variety of devices (Android & iOS native apps, Mobile-Web, and Desktop Web)
• Limited unit testing
• API testing & Functional Testing done in isolation (mini-waterfall approach)
• Usage of Cocos2d-x & Unity for game rendering – which cannot be automated via Appium
• Limited Functional Automation (for native apps)

An approach to Functional Automation of Real-Time, Multi-User Game scenarios

I overcame the above mentioned challenges by doing the following:

• A better way of working (you can call it ‘Agile’)
• Break down the walls by fostering a mindset of “build quality-in, as opposed to test for quality”
• Built a new functional automation framework using java / testing / appium-test-distribution / reportportal / jenkins with focus on – specify test intent once and run on all supported channels (ex: Android, iOS, Mobile-Web, and Web)
• Built a solution for Cocos2d-x layer automation
• Created a vision of CI-CD for the organization, and setup code-based CI pipelines to enable end-2-end visibility
• Made the framework extensible by providing ability to use same framework for multiple games

The focus of this talk will be to:

• Share an example of a particular use case

• Share the solutions, including code snippets, implemented for:
  • Functional Test Automation Framework Architecture & Design
  • Ease of Test Implementation, while maintaining code quality and promoting reuse
  • Test Execution on local Vs CI, on-demand as well as on every new build

Want to get into WebDriver, but don’t know where to start? Already using WebDriver and want to get more out of your framework? This practical, hands-on, course is designed to help automation engineers with all levels of experience. All attendees will leave with working WebDriver code on their machines and a long list of techniques on how they can successfully use the WebDriver API in their own companies.

This practical, hands-on, course will benefit all experience levels of programming, all attendees will leave with working WebDriver code on their machines and a long list on how they can successfully use the WebDriver API in their context.

The Test Automation Pyramid is not a new concept.

The top of the pyramid is our UI / end-2-end functional tests - which simulate end-user behavior and interactions with the product-under test.

While Automation helps validate functionality of your product, aspects of UX validations can only be seen and captured by the human eye and is hence mostly a manual activity. This is an area where AI & ML can truly help.

With everyone wanting to be Agile, make quick releases, the look & feel / UX validation, which is a slow, and error-prone activity, quickly becomes a huge bottleneck. In addition, with any UX related issues propping up cause huge brand-value and revenue loss, may lead to social-trolling and worse - dilute your user-base.

In this session, using numerous examples, we will explore:

• Why Automated Visual Validation is essential to be part of your Test Strategy
• Potential solutions / options for Automated Visual Testing, with pros & cons of each
• How an AI-powered tool, Applitools Eyes, can solve this problem.

Testing games as native apps on phones / mobile devices / tablets makes testing of regular apps, or websites appear like a piece of cake. Testing games becomes even more challenging when you think about automating the same.

I got an opportunity to build Functional Test Automation for a suite of games – and what an exciting time it turned out to be!

These games are built either using Cocos2d-x, or Unity (cross platform game engines for mobile games, apps and other cross platform interactive GUI and are known for their speed, stability, and easy of use).

The key challenges I encountered were:

• Millions on users, playing the games on a huge variety of devices
• Limited unit testing
• API testing & Functional Testing done in isolation (mini-waterfall approach)
• Usage of Cocos2d-x & Unity for game rendering – which cannot be automated via Appium
• Limited Functional Automation (for native apps)

Come to this session to understand how I overcame the challenges and helped build:

• A better way of working (you can call it ‘Agile’)
• Break down the walls by fostering a mindset of “build quality-in, as opposed to test for quality”
• Setup vision of CI-CD – and how to setup CI pipelines to enable end-2-end visibility
• Build a new functional automation framework using java/testing/appium-test-distribution/reportportal/jenkins with focus on – specify test intent once and run on all supported channels (ex: Andriod, iOS, …)
• Build a solution for Cocos2d-x layer automation
• Ability to use same framework for multiple games

You always wanted to start Mobile app Automation using Appium but did not know where and what to start with? Or started but got stuck among understanding capabilities, locating elements, finding whats wrong and your daily work deadlines? Yeah, we all have been there. Hence here is this workshop, precisely designed for beginners to quickly get started with Appium in NodeJS.

At this workshop, you will learn everything from the scratch step by step. We will together understand the key concepts, apply them right there and write our “first test” in appium before we learn more nitty-gritties as takeaway for you to go and implement at your actual work directly.
Think of this as a one-day crash course in Appium. The course will be prepared in NodeJS but there will be help for those who want to get started in Java.

In this age of a variety of cloud-based-services for virtual Mobile Test Labs, building a real-(mobile)-device lab for Test Automation is NOT a common thing – it is difficult, high maintenance, expensive! Yet, I had to do it!

Attend this talk to understand the Why, What and How I built my own MAD-LAB (Mobile Automation Devices LAB). The discussion will include –

• Understanding the context,
• Why no cloud-based solution worked for me,
• The automation strategy for Android / iOS / Web platform,
• The tech-stack (cucumber-jvm / Appium), and,
• The core implementation to build your own MAD-LAB! (already open-sourced)
• How this results in Continuous Delivery (CD)