The Test Automation Pyramid is not a new concept.

The top of the pyramid is our UI / end-2-end functional tests - which simulate end-user behavior and interactions with the product-under test.

While Automation helps validate functionality of your product, aspects of UX validations can only be seen and captured by the human eye and is hence mostly a manual activity. This is an area where AI & ML can truly help.

With everyone wanting to be Agile, make quick releases, the look & feel / UX validation, which is a slow, and error-prone activity, quickly becomes a huge bottleneck. In addition, with any UX related issues propping up cause huge brand-value and revenue loss, may lead to social-trolling and worse - dilute your user-base.

In this session, we will explore why Automated Visual Validation is now essential in your Automation Strategy and also look at some popular tools that can help solve this problem.

 
 

Outline/Structure of the Demonstration

  • Recap of Test Automation Pyramid
  • What is Visual / UX Testing?
  • Challenges of Visual / UX Testing
  • Solutions / Options for Automated Visual Testing with pros & cons

Learning Outcome

  • Importance of Visual Testing
  • Challenges of Automated Visual Testing
  • Techniques to include Visual Testing part of your Testing Strategy
  • Different options with examples and demos, for automating Visual Testing

Target Audience

QA, Dev, Product Owners, Product Managers, Managers, Automation, SDET, DevOps, Manual testers

Prerequisites for Attendees

None

schedule Submitted 9 months ago

Public Feedback

comment Suggest improvements to the Author
  • Maaret Pyhajarvi
    By Maaret Pyhajarvi  ~  7 months ago
    reply Reply

    I am unsure what the focus of this talk is. The title talks about visual validation whereas the description and outline seem to take the talk more towards the whole test automation pyramid and understanding what kinds of tests can find what kinds of problems. Is your intent here addressing more on how the introduction of new kinds of tools and techniques moves the whole pyramid (like adding visual verification tools has moved people's focus in what they test in 'acceptance tests' and how adding contract tests has done the same but for a different layer of the pyramid) or what would you want to see the audience take out of it and do differently? Use visual testing in the first place? Avoid using it when it is not the most important aspect?

    • Anand Bagmar
      By Anand Bagmar  ~  7 months ago
      reply Reply

      Good way to look at this Maaret. I will restructure this proposal to focus on why visual testing is essential in specific contexts, where it does not make sense, and different ways to implement it. Thanks for your feedback on this.

  • Pallavi ...
    By Pallavi ...  ~  8 months ago
    reply Reply

    Hello Anand

    Can you help me understand, would you be covering the UI validation for only mobile? Or web as well? 

    Also will you be showing any integration of tool example Applitools with Selenium during the demo?

    Thanks 

    • Anand Bagmar
      By Anand Bagmar  ~  8 months ago
      reply Reply

      Hi @Pallavi,

      I will be discussing visual validation as a concept, and why it is a critical aspect in the CI-CD way of working. Once the concept is understood, I will be discussing techniques to bring the concept into reality and different tools that can help in the aspect. One of the tools I will be referring to is Applitools from a visual validation perspective.

      This is NOT an Applitools demo and hence, I will not be showing its integration with Selenium, Appium, etc. However focus on how to extend the Automation strategy to include Visual Test Automation, and the value this brings to the team.

      • Pallavi ...
        By Pallavi ...  ~  7 months ago
        reply Reply

        Hi Anand, thanks for writing back and explaning the talk intentions. 


  • 20 Mins
    Talk
    Beginner

    In this talk, I will speak about the process and strategies on how can one analyze and select the right tools for test automation. We will look at some tools and processes that help to select the tools, frameworks, libraries, etc.

    I will demonstrate some helpful extensions and tools that will be very handy and useful during the process

    This process can be carried out before every test automation project or engagement. The selection of the right tools will help throughout the process of automation and will help to avoid delays, changes, re-work, incomplete automation, etc. and will save a lot of time, effort and resources for the organization.

    The session is designed for anyone in the Software Testing industry and the knowledge received from this session can be implemented right away in any of the upcoming or existing test automation projects.

  • 45 Mins
    Case Study
    Intermediate

    Challenges in Testing & Automating Games

    Testing real-time, multi-user games built for native apps and / or browser-based on phones / mobile devices / tablets / desktop browsers makes testing of regular products as apps, or websites appear like a piece of cake. Testing such real-time and multi-user games becomes even more challenging when you think about automating the same.

    I got an opportunity to build Functional Test Automation for a suite of games – and what an exciting time it turned out to be!

    These games are built either using Cocos2d-x, or Unity (cross platform game engines for mobile games, apps and other cross platform interactive GUI and are known for their speed, stability, and ease-of-use).

    The key challenges I encountered here were:

    • Millions of users, playing games on a huge variety of devices (Android & iOS native apps, Mobile-Web, and Desktop Web)
    • Limited unit testing
    • API testing & Functional Testing done in isolation (mini-waterfall approach)
    • Usage of Cocos2d-x & Unity for game rendering – which cannot be automated via Appium
    • Limited Functional Automation (for native apps)

    An approach to Functional Automation of Real-Time, Multi-User Game scenarios

    I overcame the above mentioned challenges by doing the following:

    • A better way of working (you can call it ‘Agile’)
    • Break down the walls by fostering a mindset of “build quality-in, as opposed to test for quality”
    • Built a new functional automation framework using java / testing / appium-test-distribution / reportportal / jenkins with focus on – specify test intent once and run on all supported channels (ex: Android, iOS, Mobile-Web, and Web)
    • Built a solution for Cocos2d-x layer automation
    • Created a vision of CI-CD for the organization, and setup code-based CI pipelines to enable end-2-end visibility
    • Made the framework extensible by providing ability to use same framework for multiple games

    The focus of this talk will be to:

    • Share an example of a particular use case
    • Share the solutions, including code snippets, implemented for:
      • Functional Test Automation Framework Architecture & Design
      • Ease of Test Implementation, while maintaining code quality and promoting reuse
      • Test Execution on local Vs CI, on-demand as well as on every new build

  • Liked Naveen Khunteta
    keyboard_arrow_down

    Naveen Khunteta - Best Practices to implement the test automation framework starting from Design - To -> Infrastructure - To -> Execution.

    45 Mins
    Talk
    Intermediate

    Best Practices - How to get the best 'Return ON Investment' (ROI) from your Test Automation.

    This has been observed that, most of the test frameworks wont be able to survive due to lack of expertise, no maintenance, no best practices being followed, and finally your test automation will be dead after few months, and there is no "Return ON Investment" from this. This is the most common problem, most of the companies are struggling and finally back to square to the Manual testing.

    My proposal : HOW to leverage your test automation in terms of best practices, best ROI, and how to adopt best automation culture in your organisation.
    I strongly propose some of the important points/suggestions to achieve this in your Organisation/Team.
    1. Test Automation Practices:
    • Design Patterns (Web/Mobile/API)
    • What to Automate/Not to Automate
    2. Common Automation Frameworks at Org Level:
    • How to design Generic Utilities, Libraries and different Components, which can be suitable for all the teams in the same Org.
    • Best practices to design your Tests (Automation).
      • Common Design Patterns
      • Common application level and Page libraries
      • Best Practices to use Assertions in your Tests (How and What to write for assertions). Most of the people don't write proper assertions and this is making your test unreliable and no defects found during execution.
    3. Inclusion of API/Backend libraries in your UI test automation as an external Maven/Gradle Dependencies to avoid un-necessary tasks, some of the important points to be considered here:
    • User Creation from APIs (No need to automate user creation from web/app for all the test cases)
    • API tests are stabled most of the time
    • API calls takes lesser time as compared to web, hence include API calls in your UI/App framework to save time.
    • Less flaky test

    4. Best Code Review Process (Do not merge your code into Master without proper Code Review)

    • Implement PR (PULL Request) Process
    • Static Code Analysis using SonarQube, Cobertura, JACOCO etc..
    • Get the benefits of Best Test Automation Quality Matrices
    • Sometimes, Manual (Functional Tester) should review your code (Assertion, test steps and use cases) to get the best coverage
    5. Quality is A Team responsibility:
    • Developers, POs, Manual QEs and Automation engineers should be included to get an overview of test automation coverage.
    6. Maintenance of the Frameworks
    • After couple of months, it should not make your life miserable if you don't maintain your libraries and framework properly.
    • Do not use Hard Coded values, make it simple and Generic.
    7. Infrastructure Setup for Test Design and Test Execution:
    • Proper Browser - OS lab setup
    • Proper Mobile Labs setup with different Devices - IOT, iOS, Android, iPad, Tablets
    • Proper CI - CD common configuration using Jenkins, Dev Ops, AWS, Docker and Cloud setup
    • Handling multiple Docker nodes using Kubernates (use of Selenoid, GRID on Cloud)