Running too many tests could be expensive in the agile world. Selecting the right test cases to run has always been a tough task.

In this talk, I intend to explain how machine learning could help us determine test cases to run from a large test suite. I start explaining the problem itself and the variables that we can take into account in order to decide the most representative tests.

We will explore examples of situations of data relationships that would not seem to be obvious for a human but a machine could detect.

Then I explain a bit on machine learning and how it could be applied to this problem. We will look into how much reliable this solution would be and what can we do to implement it, as well as alternatives to this solution.

Last I show an example and open for questions.


Outline/Structure of the Demonstration

  1. What are we trying to solve?
  2. Examples of the problem and how do we currently solve it
  3. Rule-based system
  4. Machine learning solution
  5. Demo
  6. Questions

Learning Outcome

  • This talk would provide new ways of automating tasks and would inspire the audience.
  • It would also open discussions about the future of testing and the ethics of relying on computers to do human tasks.

Target Audience

Anybody interested in AI, ML or in improving their efficiency for test case execution

Prerequisites for Attendees

No previous requirements, I think all levels could get something interesting out of it.

schedule Submitted 1 year ago

Public Feedback

comment Suggest improvements to the Speaker
  • Naresh Jain
    By Naresh Jain  ~  1 year ago
    reply Reply

    Noemi, thank you for the proposal. This is a very interesting and hot topic. I'm sure there would be a lot of interest in the session.

    Can you please provide a specific example where you've applied ML? What kind of Algos you've used? An example would really help.

    P.S: I looked at your blog but did not find anything on that.


    • Noemi Ferrera
      By Noemi Ferrera  ~  1 year ago
      reply Reply

      Hi there,

      Sorry for the delay replying but I was away for the weekend and had to catch up after.

      I'm still in the prototype for the demo. I have two entries of my blogs about this but I have yet not posted them. It's hard to explain this in an easy way (so I keep changing things) and I have been a bit busy recently (I moved to China from Europe in November and I am still adapting a bit).

      In terms of algorithms, I would explain it a bit more if I am chosen, but basically depends highly on your system. It should be a classification algorithm, but the specifics might vary. I can also do some demo on rule based system, because is another good solution if you don't want to go as deep as NN.



      • Naresh Jain
        By Naresh Jain  ~  1 year ago
        reply Reply

        Thanks for the response, Noemi.

        The participants would certainly dig the session if we go into details (NN/ML). People are looking for something hard-core. There are lot of generic, rule-based options out there, but a good ML/NN based solution/example would really help.

        Can you take a use-case from your work and explain how you've applied ML?

        We like your topic, but the proposal has nothing concrete for us to really understand the quality of your work. I hope you understand that as program committee we need to see more details before we can select speakers.

        • Noemi Ferrera
          By Noemi Ferrera  ~  1 year ago
          reply Reply

          Hello again,

          I have not used this at work (at least not yet) I'm still in early stages and it's a side project of mine. The idea of the talk+demo is to give a specific suggestion on how ML can be used in the testing field.

          Maybe I can dig out my previous talk and send it to you somehow so you have an idea? Can I do this through this platform? The intention is to build on top of that and show how it can be used and an implementation for the rule based system alternative for those who think ML is too difficult.

          I am currently lacking of real life examples in which apply this, so all I am playing with is mock data. The real examples would make the algorithm change, so it cannot be apply for every solution, it needs to be specifically created depending on your application.

          Is this the sort of talk you are looking for? Would you prefer me not giving any actual examples and just doing the same talk as in ministry of testing Dublin (if I manage to send it to you)?

          Let me know your thoughts, happy to participate if it is of interest. Also, happy to learn from more experienced speakers.

          Thank you!

          • Naresh Jain
            By Naresh Jain  ~  1 year ago
            reply Reply

            If you can share the video that would be great. I googled for it, but could not find it.

            Also, even if its a toy project with mock data, as long as people can understand the concept and walk away with ideas of how they can apply this at their work, it would be great. 

            Is your project open sourced? Can you share it?

            • Noemi Ferrera
              By Noemi Ferrera  ~  1 year ago
              reply Reply

              Unfortunately, there was no video taken that day, only pictures, but I can find the slides when I get home. Would they help? I can also pass on the details of the organizers if that works too (for you to believe that I did the talk :) ).

              The specific it's still in research. I am not making any new algorithms or anything, I'm playing around with different data and output.

              You'll probably understand it better if you read my slides (I think I have the transcripts too), but the important thing is to define the right variables for your application and train the system long enough. I was thinking about treating the system more like a black box for the example, but maybe I can show a couple of algorithms and their 'opinions' on which cases to run?

              Also, another problem I have with the mock data is that it's really hard to train, because it cannot show real defects. So, yes, it tells me which tests to run, but I can't really tell if that was good enough because I can't compare with the defects that were actually found. I was hoping I can get my hand in a real system before the conference , but to be honest this would require a much longer research to prove it's worth it to use. My intention is to prove is doable and could be useful in certain cases, rather than proving it's the best solution. Does this make sense? I hope I'm expressing myself well enough by written.

              • Naresh Jain
                By Naresh Jain  ~  1 year ago
                reply Reply

                Please note that we totally trust you. We are not questioning whether you presented or not. We are just trying to understand the content better.

                We also greatly appreciate you being honest and open. That is very much appreciated.

                Please share the slides, that would really help.