schedule 04:45 PM - 05:05 PM place Esquire

There's an old joke that goes something like this.  A tourist in New York City asks a resident how to get to a famous concert venue,  saying, "How do you get to Carnegie Hall?"  To which the resident replies, "Practice."  Aside from the slight difference in topic, this is almost the same as asking, "How do I get to high quality code that I can use Continuous Delivery on?"  You could easily say, "Practice."

By this, what you mean is that you need to test early and often to ensure that you have high quality code.  Everyone knows that unit testing is a large part of that equation.  Managers (and others) have learned that you need to watch the code coverage metrics to see that you are testing every line of your code.  People feel comfortable that if you test all the code that you're never going to be blind-sided by a bug that you missed during development.  But, as most of us who have actually paid attention to code coverage know, that is misplaced comfort.

That's because it's too easy to fall prey to one of two fallacies about code coverage.  First, it's too easy to game the metric, if you need to impress your management into complacency.  And second, it is a poor metric to judge quality by.  Unit testing is the answer for high quality code, but how much testing is enough?  There are qualitative metrics, such as you have enough unit tests when you don't have production problems, but that really begs the question.  There are static and dynamic code metrics that you can produce, but those also fall short of the goal.  However, there is one technique to use that goes a long way to answering the question unambiguously.  Mutation Testing.

Mutation Testing assumes 100% code coverage.  It then takes your nice happy code and messes with it.  Negate a conditional.  The code now does something else.  Do your unit tests find that bug?  Good!  How about changing a conditional.  Oops.  The mutation survives, and the unit tests don't find it?  Bad!!

This session describes the problem and one tool that can be used to fix it for good.  The tool is called PIT.  We look at the Java mututation tool called PIT.  We will see the results in a couple of small projects, and then see what it looks like in a not so small open source project.  We will see the role that mutation testing can have on quality,  and how we would use it in our build automation to get us further down the road to having successful deliveries all the time - not train wrecks at just the wrong time.

 
1 favorite thumb_down thumb_up 11 comments visibility_off  Remove from Watchlist visibility  Add to Watchlist
 

Outline/structure of the Session

  1. 5 minutes of introduction to the problem and the tool
  2. 15 minutes of demonstration of the tool on 2 small examples (to explain what the tool is doing) and then on JUnit (to see the extent of the usefulness, both good and bad).

Learning Outcome

  1. A better appreciation of why unit testing is not enough if code quality is the goal.  
  2. A good introduction to PIT (for Java development) 

 

Target Audience

developers, architects, technical managers, mid level (and above) managers

schedule Submitted 1 year ago

Comments Subscribe to Comments

comment Comment on this Proposal
  • Joel Tosi
    By Joel Tosi  ~  1 year ago
    reply Reply

    Hi Howard,

       I took a peek at your slides - they were pretty hard to read (partially probably my eyes) - there were lots of words and small font.  How do you use your slides?  I'm concerned that the words become distracting, but let me know if that is not the case.

     

    As this session is 45 minutes - would a 20 minute demo do it justice?

     

    Lastly, help me understand how you weave PIT into traditional test driven design practices.

    Best,

    Joel

    • Howard Deiner
      By Howard Deiner  ~  1 year ago
      reply Reply

      Hi Joel!

      This would be the first presentation on this topic.  The slides you viewed are Prezi, from past presentations, and may not be easy to view, nor do they represent how the presentations are performed.  The slides are meant to give attendees something to take home to remind them of the salient points that were talked about during the presentation.  As this is a demonstration of only 45 minutes, there will probably be far fewer ones, although content such as what the mutations do (such as removal of returns, etc.) will have to be present (again, I want people to have some references after they leave).

      The premise of the presentation is that people rarely only write code after they have a a new failing test.  And even when they do, they will only make the test pass, and many times forget to test all the combinations within the code.  That is where mutation testing shines.  Mutation testing takes the seemingly perfect unit tests (after all, they have 100% code coverage) and shakes out test cases (usually around boundary conditions) that might otherwise cause issues later on.  In other words, to get to quality code, standard unit testing / code coverage is not enough.

      Depending on how the the writing goes, I may be able to coax an additional 5 minutes on top of the 25 minutes I proposed.  I recently introduced the topic on a unit testing for managers session I just gave, and found that it was 11 minutes for just a simple walkthrough.  I wish there was a 60 minute session slot available.  That would give more time for demos.  I just don't think that I can do pure demo on this, as too many people are not familiar with the topic, much less the tools.

      Thanks for the feedback!  If there are any other things that you could suggest or critique, I'd be very appreciative.

      --  howard

      • Naresh Jain
        By Naresh Jain  ~  1 year ago
        reply Reply

        Hi Howard,

        Mutation Testing is an interesting topic. I've tried it on a few large projects and gave up. It has a tendency to produce many "false positive". Will you share some experience on how to deal with this problem?

        • Howard Deiner
          By Howard Deiner  ~  1 year ago
          reply Reply

          Hi Naresh!

          I've experienced the same issues with PIT, some of which can be addressed in the Java code, and some that can not.  I believe that the Microsoft solution, based in Pex, is better.  But I'm having issues with paying for Pex and the Visual Studio edition that it runs in.  For most teams, Mutation Testing is still aspirational, because the tooling isn't the best yet.  I can include some of the false positive examples that I've run across to put the early adopters on the lookout.

          --  howard

          • Naresh Jain
            By Naresh Jain  ~  1 year ago
            reply Reply

            Thanks for the prompt response. I've used Jumble, Judy, Jester, PIT, Nester and Pex. But I think they all have the same issue. Based on your comment it looks like Mutation Testing is not ready for prime time?

            • Howard Deiner
              By Howard Deiner  ~  1 year ago
              reply Reply

              Hi Naresh!

              I've used PIT successfully, even though there are some rough spots.  I would say that the current set of Mutation Testing tools are right in the chasm of the Moore "Crossing the Chasm" curve: between an early adopter and an early majority.  The early adopters are willing to put up with all the issues and hardships, while the early majority needs/wants more stable tools.  I do think those tools are coming, and sooner than later.  For example, Visual Studio intgrated with tools like Visual Mutator make it much easier to figure out where/how mutation testing gets inserted into test code (at least enough to satisfy weak mutation testing).

              Anyhow, many people are still completely unaware of what mutation testing is.  I'd like to bring their level of understanding at least a little more into or past basic awareness.

               

              --  howard 

              • Naresh Jain
                By Naresh Jain  ~  1 year ago
                reply Reply
                Thanks for the clarification Howard. If the objective of the session is to bring a basic level of understanding of mutation testing, then how about a 20 mins demo using PIT on some real (open-source) project which has very good test coverage?
                • Howard Deiner
                  By Howard Deiner  ~  1 year ago
                  reply Reply

                  Hi Naresh!

                   

                  That could work.  I'm thinking the Cucumber-JVM core project.  You wouldn't have any session a little longer?  There's a fair amount of explanation as to why and how this works.  Thanks!

                   

                  --  howard

                  • Naresh Jain
                    By Naresh Jain  ~  1 year ago
                    reply Reply

                    Cucumber-JVM is a good project. I was also thinking the JUnit project itself would be great to try mutation testing on. IMHO 20 mins would be a good time to introduce the topic to people and let them explore more details post the talk.

                    If you agree, please update your proposal.

                    • Howard Deiner
                      By Howard Deiner  ~  1 year ago
                      reply Reply

                      Hi Naresh!

                      I (finally) updated the proposal (after first getting PIT to work in the JUnit 4.13 snapshot).  Let me know how this looks now.

                      Thanks for the interest and your helpful suggestions!

                      --  howard

                    • Howard Deiner
                      By Howard Deiner  ~  1 year ago
                      reply Reply

                      Hi Naresh!

                       

                      The JUnit project actually makes more sense, because it's an order or more smaller than Cucumber-JVM.  Let me try it out, and update later in the day.  Thanks much!

                       

                      --  howard