Thursday, September 3, 2015

Mutation Testing - 9 Lessons Learned

Mutation testing is amazing.  It still shocks me that it has not caught on despite being around for decades.  This post is in regards to what the value add that comes from doing mutation testing.

Don't know what mutation testing is? Check out these posts:
Mutation Testing 101:

A mutation is a single syntactic change to the code. Each mutant is different from the original by a single mutation. These small changes can have big results; such as changing an operator from addition to subtraction.  Then your unit tests execute against the mutant code and in theory our tests should fail because the code is wrong.

    public int additionOrSubtraction(boolean isAddition, int x, int y) {
        if (isAddition) { //mutant: replace isAddition with !isAddition
            return x + y; //mutants: replace + with -,*,/,%
        } else {
            return x - y; //mutants: replace - with +,*,/,%

Lesson #1: Mutation Testing improves code reviews

Do you do code reviews? I bet they focus 95% on the code that will go to production and maybe a quick peek at the tests.  Despite the fact that a huge portion of the code base are unit tests we really don't review them often due to there being a finite amount of time.  Mutation testing came to my attention when searching for a method to improve the code review process.  With mutation testing the framework adds to the metrics to measure the quality of the tests suite.  Metrics provide an unbiased way of illustrating the tests that add value and the tests that just provide a false sense of security.

Lesson #2: Not all mutations add value

Some mutations are just silly and getting 100% mutation test coverage can be more trouble than it is worth.  The most common example of this I have seen are mutations that remove log statements.  Verifying logging statements add little to no value in 99.9% of cases.  The PIT mutation engine does not differentiate between the different types of void method calls so logger calls are fair game.  These failures can be ignored and maybe the case can be made to try and exclude them from the remove method call mutator.

Lesson #3: Start small

Mutation testing is very CPU and time intensive. For example in a previous post I mentioned how a build that normally takes 30 seconds to compile and run all the unit tests took over 9 minutes with mutation testing.  The first place I start is actually in Eclipse and run a smaller test suite using PitClipse.  This way you can see if your tests even can withstand a mutant onslaught. Also the mutationUnitSize configuration point of PIT allows for the slower ramp up of the number of mutations that are right for your project:

> mvn org.pitest:pitest-maven:mutationCoverage -DmutationUnitSize=2

Lesson #4: Laziness creates surviving mutants

This lesson was learned from the same mutator can caused the example in lesson #2. Not verifying the results of void method calls causes more mutations to fail than almost anything else.  In many cases this is just due to pure laziness.  I just found a project  where all builder method calls to populate POJOs only had an assert to make sure the result was not null.  So the tests did not care what was populated in those beans and just focused on them not being null.  This is a perfect example of having 100% code coverage but the tests themselves being worse then if they did not exists at all.  Do yourself a favor and verify every element in the objects returned. I know it takes time and many asserts but that extra effort may save you from a production bug born from laziness.

Lesson #5: People won't see the value of mutation testing at first and that is okay

Mutation testing sounds interesting to people as a theoretical exercise but the benefits of it are often hard to see.  It takes a mature organization that is already dedicated to a strong unit test base to even consider adopting it.  To get it adopted you, yes you, need to be the zealous advocate for mutation testing.  Just do it.  Update your builds and put a mutation testing job in your continuous integration tool like Jenkins.  If your team hopes to do continuous delivery you need a rock solid test base.  Start pointing out mutations that result from recent commits just like you would if the commits result in bad code coverage.  Do not try to convince people of the value of mutation testing but instead show them the lack of defects that could result from weak tests.

Lesson #6: Mutation Testing is not just for Java

Although the mutation testing tool I am outspoken about is for Java that doesn't mean that there are not frameworks for other languages.

(I have not used any of the following tools and am therefore unable to attest to their value)

Lesson #7: Be relentless in advocating for mutation testing

Mutation testing has been around since 1971.  The days have long passed when the computer power is not available to make make mutation testing a viable tool.  If you don't push for it to be adopted it likely won't because we haven't reached that critical mass to give it a life of its own.  So do it and email out the reports. Do a lightning talk or brown bag session on mutation testing to get the word out.  Honestly I don't care if you even need to borrow heavily from or steal the Mutation Testing Slideshow that I recently created. Get the word out.

Lesson #8: 100% mutation kill score is possible but expensive

As previously stated some mutations are silly not really useful but there are times when there is no margin for error and your code needs to work.  Say for example you are working on code for a medical device.  There is no room for error when someones life is on the line.  You need mutation testing to do everything you can to be defect free in cases such as that. However some mutants just aren't worth the time needed to kill them.  It is similar to writing unit tests to test all getters and setters in POJOs.  Is it really worth your time to write tests for basic getter and setter methods. I don't know that is a personal decision but squashing some mutants might add even less value than those unit tests.

Lesson #9: Mutation testing makes you a better programmer

Knowing the ways your code could break and how good your tests need to be to protect against that is not a skill that is developed overnight. Mutation testing ensures that you don't cut corners and take the easy route.  Write solid code and bombproof unit tests.

Links to other posts related to Unit Testing and Mutation Testing:


  1. Actually on point #2 pitest does avoid mutating logging statements.

    By default it will avoid creating mutants on lines with calls to the following packages

    * java.util.logging
    * org.apache.log4j
    * org.slf4j
    * org.apache.commons.logging

    If the logger you are using is not in one of these packages pitest can still be configured to ignore it.

  2. When you do not want to invest heavily into research and development teams for building your own software, you can always partner with an expert outsourcing software development India company that can take up this burden to provide exceptional expertise and save you huge expenses of maintaining, testing as well as discarding the unviable software experiments.
    Winzip Serial Key

  3. I've been using this tester to work It is a new and very convenient. I understand it is very fast and now I use at work.

  4. With global demand of saving energy and wisely making its consumption, there has been a rise in the number of people working in this field.
    read our blog

  5. if you are also working in this industry, then there is a great software for you to make this job a bit less challenging. There are a lot of people who are using petroleum engineering software in Houston, already. buy Revit 2013

  6. Various engineering mechanisms utilize this software, such as communication, electronics, process, chemical, mechanical, and electrical engineering. cheap SolidWorks 2016

  7. If there's an ideal CRM software package that works for every company and every situation, it hasn't been discovered yet, simply because every company has slightly different needs for their customer relationship management needs as well as software implementation. SugarCRM Philippines