Like testing is an essential part of the software development lifecycle, automation is a non-negligible part of testing. Nowadays, most of us are somehow involved in automation, since it helps us to perform continuous testing and minimizes manual effort. It is great but also has several difficulties. Let's discuss how we can achieve coping with test automation challenges by going over real-life experiences.

What to expect from this session

As most of the QA people are doing test automation and there are several issues commonly faced by them, we will talk about proposals to cope with those test automation challenges. We will discuss ways to avoid broken tests after updated page layouts on the web apps that we are testing, to minimize the number of flaky tests, to automate scenarios where there is a hardware dependency, and to make implementation easier.

Common problems:

One of my recent projects was a warehouse automation system in which there are autonomous mobile robots moving around. I was supposed to test a scenario in which there is an obstacle in front of the robot, so it should find a new path and pass around the obstacle. How did I automate this test scenario? We will discuss several proposals for this and some other problems.

Some of the greatest common difficulties are:

• Coping with updates/changes
• Hardware in the SUT
• Implementation and Maintenance
• Time-consuming executions
• Reproduction of the issues found during executions

Fundamental solutions:

• Locators/Selectors improvement: Usage of test data IDs
• Simulation and test harness
• Non-functional testing
• Execution duration analysis and parallelization
• Logging and saving pieces of evidence in the pipelines

Elevator Pitch

We try to ensure the quality and our confidence in our deliverables. But how can ensure that we are ensuring the quality indeed? What is the way to measure the quality, let’s discuss in this talk.

Abstract

Motivation: 

Nowadays, great rivalry exists in the software world. Everyone tries to reduce time to market and cost with a broad context of features. To improve the competence of our product, we try to add several compatibility and scalability specifications. But sometimes less is more. To deploy in 1 week with several issues, or deploy in 2 weeks with no major issues?

We're talking about the elements that sustain an output here: Time, Coverage, Cost, and Quality. A bit different from others, measuring quality is challenging.

Correlated to the quality, the performance of Quality teams is also under the spotlight:

• What are they really doing? What is the outcome of QA activities?
• Are they successful? What do the QA activities change?

Problems: 

How can we measure the quality and the outcome of the QA teams/activities?

Solutions: 

Firstly, we will talk about being ready for measurement. Unless we do not have proper tools or state transition definitions, we won't be able to analyze data. Then the big question is what to measure. We can list hundreds of metrics, but which of them are the best to get some insights? Once we know what to collect, the way to minimize manual effort is automation. After getting raw data, we are good to create fancy monitoring tools. Graphs and dashboards are great ways to provide visual pieces of evidence.

Eventually, last but definitely not least is the interpretation of the results. Numbers are great, but what can we understand from them? Are there any action items we need?

Wrapping up, a lifecycle of a monitoring activity looks like this:

• Customizing environments to enhance transparency/visibility
• Choose metrics: Proposed decision criteria
• Automate measurement 
• Creating visuals
• Analysis

Results & Conclusion: 

I present several good practices to perform a quality monitoring. But eventually, the conclusion we will have is: There is no way to measure quality! We can't say we have 1 kilogram, 5 units, and 10 meters of quality.

So, what are we talking about, then? Of course, measuring key metrics is still a good opportunity to get some insights. We can learn a lot from the bugs reported, such as

• Which components are the most prone to errors?
• What kind of testing activities should we improve?
• What are the most common root causes of the issues?
• What is the ratio of reopened bugs?

An important part of the talk is presenting not only the working metrics but also the ones which do not work! What does the number of bugs or the number of test cases mean? I won't ask you to forget about those, but I will convince you not to be obsessed with these numbers solely.

Finally, I will introduce a set of metrics, which is not commonly used: Emotional metrics. Since we are talking about the quality, it is not only the product itself, the way we are developing, and the people involved. We can’t build quality with unhappy teams or people. Thus, let’s talk about team satisfaction as well.

Abstract

Elevator Pitch

Speaking in terms of software development, we can think of Front End libraries, backend services, mobile applications, and so on. This time we will talk about developing the test itself as a service. Ladies and gentlemen, let me share my journey to implement our TaaS.

Problem:

Software testing is becoming more challenging every day. Along with the technical requirements, we have to cope with a very dynamic way of development and an intense scope to be handled in a limited time. While several teams or groups are trying to cope with these challenges, we observe:

• There is no standard among these teams: Each is following a different approach/strategy (if they have:O)
• Some of them are struggling with problems, which were solved by others.
• Inefficiency due to duplication.

To have a better understanding of the implementation variations, I attached a slide where I showed how a very simple scenario can be automated in 4 different ways: https://drive.google.com/file/d/1c0aboWtk9WffpxNHCSuooI3QfvrrTsD0/view?usp=sharing

Solution:

To remove the imbalance, support those who are struggling with the problems that were solved by others, and ensure the code quality in all test frameworks used by several teams, we come up with the centralized test framework development idea.

The motivation was to develop a framework in which the most common problems were handled and serve the teams. For this purpose, we have executed several steps starting from the collection of the most common difficulties. In this way, we figured out which problems we should target. Then we designed the architecture to solve those problems and started our implementation. All the steps we have executed are as follows:

• Requirements definition
• Prioritization
• KPI and Metrics formulation
• Architecture & Documentation
• Backlog grooming
• Implementation
• Training
• Monitoring

At first glance, testing may look straight-forward and simple. But not only verifying and validating the specifications, we are supposed to improve the product and the product development processes. Revealing weaknesses and improvement rooms in the product helps us to make our services better, but how about preventing them in the first place?

Beyond reporting bugs, testers can increase the overall efficiency in various different ways. So, obviously it is not easy and needs to focus on numerous aspects. Sometimes a tester may concentrate on analyzing the new deployment package, and sometimes he/she may need to coach team mates to build a holistic quality lifecycle.

Thus, a creative, empathetic, good communicator tester can influence and improve the product, project, company and the community. These are only a subset of all skills a good tester holds. So, what other skills help us to perform well and efficiently?

I have collected various skills that a tester should have. I aimed to categorize several of them from different perspectives like communication, agility, problem solving, efficiency and others.

Software testing requires development skills in terms of automation and CI/CD on the one hand and social skills on the other hand. There are tough responsibilities to hold like release management. It looks like we have to be superman holding all these qualifications. But we are human beings. How can we succeed in superman performance as a human?

After discussing what types of skills are of great significance in software testing activities, finally we will discuss the key approach to achieve a good performance.

Main headlines of the scope are:

• Technical Skills
• Stress Management
• Critical Thinking
• Orchestration
• Agility
• Communication
• Self Improvement
• Efficiency
• Sharing

Story:

After I was hired by my company, they assigned me to a project. Soon after I was introduced to the project, I quickly realized that there are a few -maybe lots of :)- improvement rooms. I will share my story which ended up in a successfully built quality.

With correct steps, we can see how quickly a quality process can be developed. But what is a quality process and how can we ensure quality? Does quality assurance equal to running tests? As you can guess, the answer is no. So we will talk about the components of quality assurance.

By means of the initiatives I have introduced, we experienced not only the quality of the product, but also the efficiency of the agile ceremonies and visibility of information. Since efficient communication is a key player of quality assurance just like everything else, we will talk about conducting daily meetings in the best way.

Initially:

• No track of bugs
• No feedback cycle on tests
• No transparency of the progress
• No organized feature
• No metrics, no idea about coverage
• Escaped bugs
• Massive manual effort 

Solutions:

• Bugs Management initiative
• Test Review initiative
• Short and very understandable progress reports
• Feature Housekeeping initiative
• Monitoring, Traceability
• Pulling QA in Early Stages
• Automation: Common Locators, Parallel Running, Reporting Dashboard

Quality Assurance is an important part of the development lifecycle. We try to meet the customer expectancy in the best way and beyond this, try to improve their understanding of the requirements as well by means of communication. 

Quality is not only for the customers, but also for the developer and the supplier. The efficiency can be increased and the waste of time and cost can be minimized in various ways.

Considering all these, let's think one more time in detail. 

• What is really ‘Quality Assurance’?
• Why are we doing QA?
• When is it performed?
• Who is responsible for QA?
• How and in what ways can we assure quality?

Problem Statement:

We all know that Testing and QA is crucial by heart, but rather than saying that it is important, embracing it more matters. So, the real thing is creating quality ambassadors who have a strong quality mindset and quality oriented way of thinking.

How can we build quality?

Proposal:

Test or quality assurance is running tests and reporting the results.

Or is it? Besides, revealing the improvement rooms in the product, encouraging people to have the same perspective and coaching are some other components of QA.

Quality of the development itself, along with the developed product, can also be improved. We can assure the transparency of our processes, flow of information and priorities.

In this talk, I will demonstrate numerous aspects of quality assurance. We will pursue quality! What does having quality mean and what is QA further than execution of test cases, we will discuss. 

As we all know, automated testing is a great way to reduce manual effort since by this means we can leverage automation to replace the execution of tests by a human tester with virtual pipelines. So, we can eliminate not only a huge need for manpower, but also time and cost.

With such a great looking opportunity, why do we still have to spend so much time on QA activities even if we have automation? Being aware that automation is still very useful, perhaps the question we need to explore is, what doesn't it solve? Or what are the biggest pitfalls in building test automation itself?

In this talk, I collected my personal experiences focused on the challenges of test automation in global projects. 

Common problems:

Some of the greatest common difficulties are:

• Coping with update/changes
• Code Duplication
• Tricky Behaviors which are not easy to automate
• Hardware in the SUT
• Time consuming executions
• Reproduction of the issues found during executions

Fundamental solutions:

• Smart automation strategy
  • A good design pattern
  • Utilizing execution patterns
  • Well defined locators
  • Parallelization
• Regular clean-up
• Test harness with necessary simulation
• Logging and saving evidences in the pipelines

Motivation: I have experienced a lot of challenges in both technical and social manners and tried to develop solutions to cope with them throughout all the testing activities that I was involved in. Eventually, I have wrapped them up to make a list of common pitfalls which can be encountered by any tester and some golden rules to successfully get rid of them and manage a test project. 

The most common problems are:

• In continuous testing environments, due to time pressure, sometimes Quality focus may be shifted. How can we ensure to stick to the correct mindset even under stress?
• While trying lots of approaches, how can we optimize the best way? How can we decide to go with it? How can we monitor the progress, trends and results of each single POC?
• Test Reliability & Test Smells
• Efficiency

Some fundamental solutions are:

• Being truly agile: Adapt new solutions quickly.
• Manage the progress: Be aware of what is going on by setting KPIs to track & monitor with tools like CloudWatch, Grafana
• Technical part: Automation principles for the sake of robustness. Solutions to reduce flaky tests & analysis effort & costs.
• Holistic testing!

Do bugs speak?

Yes, they do. People speak different languages like English, German, French, Chinese etc. But is communication to bugs possible? It is important to understand them, because they really tell us something. There is valuable information underlying the defects of a software, and information mining from defects promises for improvements in terms of quality, time, effort and cost.

Problem Definition

A comprehensive analysis on all created defects can provide precious insights about the product. For instance; if we notice that a bunch of defects heap together on a feature, we can conclude that the feature should be investigated and cured. Or we can make some observations about the severity or assignee of similar defects. Therefore, there are some potential patterns to be discovered under defects.

Wrap-up

Defect analysis is very important for QA people, and especially for QA managers. We utilize lots of aspects to get an idea about the product itself or our procedures. For instance while monitoring defect distribution across testing types, we will discuss how to get an idea about the quality of our testing approach. I.e whether we are applying all types in a balanced way. (functional, performance, documentation, etc.) Or over another graph, in which we track the gap between open defects and resolved defects, we will discuss what action items we can take when the gap widens. Finally, with ML assistance, we will see how we can reduce manual effort and cost.

Results & Conclusion

In this session, we discuss data mining from bugs and usage of ML in defect management. Objective of the study is:

• To present in which ways defects can be analyzed

• To present how ML can be used to make observations over defects

• To provide empirical information supporting (b)

Motivation: After starting a new project, the initial planning is done and the very first development activities start. But as we all know, all products are tested, which means at some point (ideally soon after the development activities start) testing activities also start.

So, from the start of the testing till maturity, there is a long and tricky way. In this talk, I discuss what kind of challenges are experienced and how we can cope with them.

I will explain:

* How we managed to collect insights from bugs.

* How we managed releases

* How we reduced testing effort before shipment

* How we maintain the stability of CI/CD.

Problems:

• Uncertainties
• Hidden information
• Missing guidance
• Complex and complicated systems under test
• A lot of deployments and limited resource
• Adaptation problems in the team

Solutions:

• Information mining: Walk through and explore the system, Customer Surveys, Requirement Analysis, Specification Benchmarks
• Define processes: Bug Life Cycle, Test Case Life Cycle, Workflows: Code Review Guideline, Acceptance/Exit Criteria
• Decide Tools: Issues, Tasks, Tests, Results, Code
• Maintain Tests: Coverage, Suite Management, Markers
• Automation: Implement the skeleton: Flexible enough for further improvements, Robust and open for RCA, Reporting
• Define Levels and Subsets: Priorities for the executions

Results & Conclusion

In this talk, I will present building a good software testing life cycle and achieving quality in software projects from scratch.

We all observe that software testing continues to grow, proving that it is a living organism. 

Scope:

Software testing processes are started to be adapted into Software Development Life Cycle (SDLC) in waterfall (1) approaches. At the end of the development activities, verification and validation are performed to check the product before shipment to customers. What was the problem with Waterfall methodology? Testing activities were scheduled at the end of timeline and testers were out of time since previous activities are shifted.

Then, in agile (2) methodologies, we see testing activities in all phases of the Software Development Life Cycle (SDLC). It starts from the first sprint. In this point, challenges are bigger since it is a very dynamic environment with lots of changes in a short duration. 

To cope with complex scope to be verified in a limited time, automated testing started to appear in our life. Nowadays we meet lots of “Continuous X” terms, such as Continuous Integration, Deployment and Testing in terms of Devops (3) approaches. Can we go home and get some rest when we automate all cases? Of course not. We have to continue to track test results, maintain flaky results and keep quality in high standard. Still we have many manual tasks on healing, maintenance and analysis.

Nowadays, researches are looking for adaptation of Machine Learning (4) algorithms and other hot topics to testing processes to reduce the manual effort and improve quality. To sum up, improvement of software testing never ends, but sometimes the growth confuses people. What is the deal of Scrum? Why are people crazy about continuous integration and continuous delivery (CI/CD)? What is the difference between Agile and DevOps? We will go over lots of this kind of questions.

Results & Conclusion

Objective of the talk is: To provide some insights about the growth of software testing. Starting from the early times of software testing, we will try to overview the big picture and finally we will discuss what we can face in the future.

When I refresh my memory and consider my whole testing life, I again realize that life is full of lessons. Whenever I think about this, I come up with a saying by John W. Gardner: “Life is the art of drawing without an eraser.” We are all making mistakes and it is not possible to manage each single piece of processes perfectly. Still; what we can remember is, with additional brush strokes, those mistakes can evolve into nice colors inside the big picture. Besides, in the next run, we may take precautions against possible mistakes.

In short, after the quick review of my career, I decided to collect the required soft skills for a tester. Being a tester requires lots of skills, but it is not holding all of them in an unnatural way. We are humans, no supermans.

Motivation:

After executing a project to test a cloud-based microservices platform, we experienced a lot of challenges in both technical and social manners and tried to develop solutions to cope with them. Finally, I have wrapped them up to make a list of golden rules to successfully manage a test project. The purpose of this talk is to give insights about how a test automation project is managed.  

Problems:

• Technically, if not managed properly, automated testing will lead to extra costs and could even be less effective than manual testing.
• From another aspect, at some point, we had too many complaints in our retrospective meetings about the heavy deployment activities and redundant executions. We had lots of flaky tests, execution lists full of not clear test definitions. Everyone was sick and tired of maintenance issues and the team was not happy. We took actions to improve motivation in the team. 

Solutions:

• Being truly agile: Adapt new solutions quickly. 
• Manage the progress: Be aware of what is going on by setting KPIs to track & monitor with tools like CloudWatch, Grafana
• Technical part: Automation principles for the sake of robustness. Solutions to reduce flaky tests & analysis effort & costs: Code demos on Java API polling libraries, Selenium usage and others.
  • How we solved flaky tests: Adaptations into the code and observing the effect of each application. A few examples are polling mechanisms (safe wait methods) and test design techniques 
  • For maintenance efforts: We keep test history in the pipelines and develop helper methods to improve quality. 
  • Improve coverage from different aspects to reduce escaped bugs.
• Team spirit! 

Results & Conclusion

After application of the proposals, we observed that waste is eliminated by prioritization and removing duplicated or dispensable work. After all, we believe that this submission has interesting content which can make great attention. Instead of theoretical claims, we discuss faced challenges and applied solutions. We analyze effects of solutions with before-after situations, graphs and evidence. Instead of what to do, we go over how to do it.