Isabel will be presenting the following session
  • Isabel Evans

    Isabel Evans - Stuck in Limbo with Magical Solutions

    45 Mins
    Case Study

    As we face increased demands or speed, change and technical excellence, the pressure, and the need, for tool support for testing increases. Organizations want to automate aspects of testing. Is this possible? What effect does it have on the work and on the people doing the work? Success is not just about selecting and implementing tools and technical infrastructure. People ensure the success or failure of testing and of tool use; they must drive the project. 

    Isabel is midway through a research project to explore testers’ experiences with tools and automation. So far, she’s uncovered some illusions about tool usability, some attempts at magic, and many new questions.

    Interviews, workshops, and survey responses from nearly 200 testers revealed people found tools and automation to be problematic in many ways, leading to people expressing high levels of emotion, stress and distress. Over 30% of the respondents to the anonymous survey answered questions about their experiences with test tools in a way that indicated emotional responses. People talked about tools leaving them “stuck in limbo” and unable to do their work, expressed their frustration that tools are treated as “magical solutions for all of the test problems” and expressed their fears, and the demotivation they felt: “I think I should leave my job.”

    Part of the problem was the usability of the tools. This was the attribute most often discussed by testers when describing both their aspirations for tool, and their frustrations with tools. It is not just that some tools have been built without usability in mind, but that where usability had been part of the design criteria, it merely offered an illusion of usability, with a superficial veneer to the UI that didn’t address workflow, differing needs among testers, and their changing requirements. 

    In this reflection on her research and industry experiences, Isabel talks about learning new ways of working after decades in the industry, her findings so far, what she’s discovered that will help your testing now, and her plans for the rest of the research. She’ll be asking you to contribute to a survey to discover who is doing testing, and how they are doing it. This will feed into a model for designing test tools that support people, and make test tool design a people-centred activity.

1. What got you started/interested in modern software development methods?

I took my computer scoence degree in the 1970's, and moved into testing in the 1980's. I was working at a company in the late 80's on software maintenance on mainframe systems, as a member of the outsourcing supplier team. We realised that we needed to deliver maintenance updates regularly, frequently, reliably and with quality built in to what we were doing. So we devised a lifecycle model to allow 4 week cycles of defining the next set of changes, based on the customer's needs, design, build, test, and deploy. Although we didn't call it agile, it predated agile, the way we were working had many of the characteristics of agile, from the feedback loops to the need for good, tight engineering, to the need for everyone to test, and so on. I think then when I saw agile starting I thought "hang on..."

2. What do you think is the biggest challenge faced by the software product engineering community today?

I think there are several challenges. There is a challenge around diversity - that we have IT teams that reflect the world we live in, and take account of a multiplicity of people - whether by demographic measures or by culture, ways of living , of needs and expectations. There is a challenge around ethics - related to diversity, of course, yet also to ethics in data use, in machine learning, algorithm use, etc. And ethics around envirnmental challenges such as energy usage.  And there are challenges in terms of increasing technical complexity, and the increasing need for speed with quality.  I think the ethical and diversity questions are the more important challenges, but the challenges in terms of complexity, speed and quality probably feel more urgent to teams. Of course, the environemntal impact of IT is also now urgent - we have to consider the affect we have on the planet. Perhaps as an industry we need to pause, think, and reset our expectations in terms of what we should be doing - what is good for humankind, the environment, the planet - [should we be making these products at all?] and how we should be doing it - in terms of ethical considerations, diversity - [d owe have teams who represent humankind, the environment, the planet] - and then how we achieve that in terms of approaches we take to the engineering of products. My research is focused on the last of these; if we understand more about who is testing software and how they are testing, we'll have a better understanding of what tools are useful and appropriate.

3. What do you think are the most exciting developments in software product engineering today?

I am excited that I'm seeing ethics increasingly discussed at IT conferences.   

4. Why did you choose the topic(s) you will be speaking about at the conference?

I have been researching the expereinces of testers with the tool they use, and the title is based on quotes from some of the testers who provided their stories. People felt stuck in limbo, frustrated, and that they were being asked to solve problems as if by magic. I want to share my findings so far back to the industry, because I think other testers will relate, their colleagues and managers need to understand the frustrations, and also because anyone designing or building test tools and automation should be aware of the effects of their design decisions. It's not just about superficial improvements to usability... it's about the whole affect on people's workflow - and even their self-worth. I also have some evidence that the same problems affect other disciplines and their tools... As I move into the second part of my research I need to know who is doing testing - and so I also want to ask the Agile India community to contribute to my current data collection survey.

5. What are some of the key takeaways from your session(s) at Agile India?
  1. Insights from research into the experiences of real practitioners testing in real projects
  2. Understanding of some of the usability and human blockers to successful test tool implementation
  3. A chance to inform the future of test tool design by contributing to the research data:

I’ve been researching into the experiences of people doing testing with their test tools. Data collected and analysed so far indicates I need to find out more about who is doing testing, and how they are doing it.

Your help is vital in the next stage of my research. If you test software, or if you use the results of testing software to inform your work, please respond. I am running a survey by online interview to ask for detailed information about you. The survey link is available to you and open until end 2021:   The data you provide will feed into a model for designing tools to support testing; your contributions will help to form that model.

Please help by completing the survey, and by passing it to colleagues who might be able to contribute.

Thanks so much!

6. Which sessions are you particularly looking forward to attending at Agile India this year?

that's a hard question - so many great speakers and interesting topics! I have not yet decided!

7. Any personal remarks/message you want to share with the software community?

In your career think of the effect you are having - on your colleagues and your customers, but also on world, the environment, all the living things around you. We only have a small planet.  Second: be prepared for a long career and one where you are constantly learning.