Metamorphic Testing for Machine Learning Models with Search Relevancy Example

Accuracy of a Model can be improved in several levels and multiple variables, boundaries and guidelines. With the well known problem statement and solution, it is difficult to evaluate for all the given cases the model would be predicting expected outcomes. Machine Learning Models are solving for the problems for which results are unknown, most of the times. This arises a problem of Test Oracle. Recent surveys and work have shown that this difficulty can be reduced by some of the blackbox testing techniques such as Metamorphic Testing, Fuzzing, Dual Coding et.,

Even though the output of a Model is not known, we can make few predictions based on the Metamorphic relations. A metamorphic relation refers to the relationship between the software input change and output change during multiple program executions. Many metamorphic relations are created based on the transformation from training data set or test data set. We further classify them into Coarse-grained Data transformation and Fine-grained data transformation.

We will discuss different transformations. Will go through the example of a Search relevancy problem and will analyse the application of Metamorphic testing to verify the Machine model built.


Outline/Structure of the Experience Report

  • Introduction - 2 min
    • Will discuss about the difficulty of Machine learning model testing
  • Test Oracle - 1 min
    • What is Test Oracle?
    • How it is constructed
  • Problem with Test Oracle - 2 mins
    • Why typical Test Oracle not applicable to Machine Learning
  • Pseduo Oracle - 1 mins
    • What is Pseudo Oracle and it's advantages
  • Metamorphic Testing - 3 mins
    • What is Metamorphic testing
    • Metamorphic testing with simple Example
  • Transformations - 3 mins
    • Coarse-grained Data transformation
    • Fine-grained data transformation
  • Search Relevancy - 3 mins
    • What is Search Relevancy
    • Machine Learning Model
    • Problem in Testing the Machine Learning Model
  • Application of Metamorphic testing in Search Relevancy - 4 mins
    • How we used Metamorphic testing for our Model
  • Advantages of Metamorphic Testing

Learning Outcome

Detail understanding Metamorphic testing.

How to improve the quality of the Machine Learning model using Metamorphic testing

How to use metamorphic relations for designing test cases for Machine Learning model

Target Audience

Software Engineer, Data Scientist, AI Enthusiast, QA Engineers, Project Managers

Prerequisites for Attendees

Basic knowledge of Machine Learning and development

schedule Submitted 3 years ago