Smart Test Failure Analysis with ELK (Elastic Search, Log Stash & Kibana)

schedule Jun 29th 12:30 PM - 12:50 PM place Kalinga Hall 2 people 3 Attending

In contemporary test automation world, we are running 1000's tests every day. Even though most of our test cases are reliable and stable, debugging failures will consume lot of time, if not handled appropriately. This is an effort to reduce the analysis of failures and come up with probable root cause for each and every failure.

In this talk, we will explore all the approaches we have taken to drastically reduce the time to debug the failures & let you concentrate more on adding new tests. We will also talk about the approaches which could not yield results for us initially & why. We will look into building a failures' dashboard, driven by pattern based classification of logs(both server & automation tool) with reporting.

 
5 favorite thumb_down thumb_up 3 comments visibility_off  Remove from Watchlist visibility  Add to Watchlist
 

Outline/structure of the Session

  1. Introduction [1 Min]
  2. Classification of test execution failures [2 mins]
  3. Patterns for each type of failures [2 mins]
  4. Configuring Elastic search to consume logs [8 mins]
  5. Strategies to get reliable results of analysis [2 mins]
  6. Dashboards for daily execution snapshot & accuracy [2 mins]
  7. Q & A [3 mins]

Learning Outcome

This talk will equip you with analysis patterns of failures. Attendees would be empowered to separate flaky tests vs functional issues easily and fix them. Using examples explained in this session, you can build your own ecosystem for failure analysis.

Target Audience

Dev, QA & DevOps

Prerequisite

  1. Hands-on Experience in Test Automation
  2. How errors are handled & logged in Test Automation Framework
  3. Basic knowledge on Web/Application Server Logs
schedule Submitted 2 months ago

Comments Subscribe to Comments

comment Comment on this Submission
  • Pooja Shah
    By Pooja Shah  ~  2 months ago
    reply Reply

    Useful content, thanks for proposing this Manoj. Would like to know if there is any of your previous video link on any of your talk on any topic?

    If not, it would be great if you could just share/attach a short 1-2 minutes video shoot (face visible while speaking) 

     

    • ManojKumar
      By ManojKumar  ~  2 months ago
      reply Reply

      Hi Manoj,

      Do you want to send us a video of your talk(preview)?

      • Manoj Chiruvella
        By Manoj Chiruvella  ~  2 months ago
        reply Reply

        Pooja, Manoj,

        Posted preview video, let me know if you need any thing else.


  • Liked Simon Stewart
    keyboard_arrow_down

    Simon Stewart - Fix a Bug, Become a Committer

    Simon Stewart
    Simon Stewart
    WebDriver Creator
    Facebook
    schedule 3 months ago
    Sold Out!
    480 Mins
    Workshop
    Beginner

    Have you ever wondered how Selenium works under the covers? Do you get frustrated with locators not locating, pages not loading, or browsers behaving inconsistently from one run to the next? Selenium is an attempt to unify thousands of disparate elements across a wide spectrum of challenges into a single, common interface that works seamlessly with all the major browsers - and yet only a handful of volunteers work to maintain this gigantic effort. If you would like to enhance your own Selenium experience while contributing back to the software that has defined so many of our careers, come to this workshop. In it we'll dissect the different elements of Selenium, dive into its internals, learn how it was built and how to make changes to it, and even write a unit test you can contribute on the same day!

  • Liked Syam Sasi
    keyboard_arrow_down

    Syam Sasi / Jerry Zhao - How to build a device lab in your office in 48 hours!

    45 Mins
    Talk
    Intermediate

    Continuous testing is an integral part of continuous delivery pipeline. When it comes to mobile application, the testing become increasingly complex.

    As part of our internal quarter hackathon at Carousell, we developed an automation testing framework and device lab in 48 hours which has both parallel and distributed mode of running.

    We will share about how you can set up a similar device lab in your organization and the best practices to be followed.

  • Liked Shivaling Sannalli
    keyboard_arrow_down

    Shivaling Sannalli - Unify service and UI layer Automation - BDD way

    90 Mins
    Workshop
    Intermediate

    Software delivery is switching towards business driven development. This needs capturing requirements as presented by clients and translating these requirements into development tasks and tests.

    Clients want visibility on the quality of the requirements that are captured. Quality in the delivery of the project measured using the test pyramid — heavy unit tests, more integration and service layer tests, and fewer UI tests. Test pyramid is the ideal quality structure for Agile projects.

    In this presentation/session we will focus on how we can combine both service layer tests and UI layer tests in a BDD(Behavior Driven Development) way of Automation. That will bring test coverage visibility to business. Clients themselves can run this suite without having to understand technicalities of the implementation. Test scenarios will exactly be the requirements captured by business as behaviors.

    To accomplish BDD way of automation for both service layer automation and UI layer we will use following tools/libraries:

    BDD Tool: Gauge or Cucumber (both are open source)

    Service layer Automation : Unirest for JAVA

    UI layer Automation : Selenium WebDriver for JAVA

  • Liked Shweta Sharma
    keyboard_arrow_down

    Shweta Sharma / Mohit Aghera - Automated testing using Behat and Selenium

    480 Mins
    Workshop
    Beginner

    Do you want to deliver a well-tested website, covering all the test scenarios in every iteration? Excellent, then what is the problem? Oh, too much of manual effort in executing the regression test cases. Well, then automate it. Now, what? Automation, huh, it needs a lot of coding, too much of investment. OK, enough reasons given, the list can go on.

    Get started right on with automating your test scenarios with minimal code is what all Behat is about. Behat proved to be extremely useful to me in order to enter the world of automation. After understanding the architecture and its correct usage, my team and me could automate around 70% of the test cases by writing no/minimal PHP code. The lovely features provided by Selenium Webdriver can be easily integrated into Behat with the help of Selenium2Driver which comes out of the box. It is an automation testing framework, coupled with rich libraries in the form of extensions to enable users automate their acceptance tests, API tests and even create a Page Object model when needed. You can execute tests on various browsers or even headless browsers like phantomJS and headless Chrome for faster execution.

    Since Behat is built on the lines of Cucumber and promotes BDD, it has proved to be an extremely useful tool to ensure that were are first building the right system followed by the system is being built right. With the help of human readable scenarios written in the feature files, the automation system can be understood by all the three parties - Business people, developers and testers. Gherkin language enables the tester to build more and more strong and a rich Automation test suite than trying to get the code right. composer.json and behat.yml files enable the user to configure stuff in no time.

  • Liked Siddharth Kulkarni
    keyboard_arrow_down

    Siddharth Kulkarni - Dr Culture Shock - Or How I stopped worrying and embraced Org Culture

    45 Mins
    Talk
    Intermediate

    The org culture mantra is usually considered a silicon valley mumbo jumbo. Although many companies and teams rant about culture, very few in my opinion take it seriously. In this talk I would like to discuss the importance of Org culture and how it makes or breaks businesses and teams. I will lay out some key pointers that will help the influencers in the audience make decisions about their org or team culture. The talk will be in the context of culture in tech companies.

  • Liked Gil Tayar
    keyboard_arrow_down

    Gil Tayar - Frontend Testing: Stepping in and Collaborating with Developers

    Gil Tayar
    Gil Tayar
    Sr. Architect
    Applitools
    schedule 3 months ago
    Sold Out!
    45 Mins
    Talk
    Intermediate

    Testing is shifting left, moving closer to testing the code itself, and not the full product. But while managers dictate a shift to the left, the developers and testers themselves are confused as to how exactly to test the code. While the backend world already has established methodologies for testing, frontend developers and testers are still trying to figure out how to test their code.

    Testers need to step in and work with the frontend developers. Testers need to understand the framework by which frontend code is tested. They need to understand the various kinds of testing that can be performed on frontend code, and which tools can be used for this.

    I will give a recipe that you can follow to ease your fear of the unknown —writing tests for developer code. I will discuss the various kinds of testing and how they fit together in a coherent way, but most importantly - I will include sample code that you can use as a template in your own project, and if time permits—a bit of live coding will be done.