Mutation Testing with PITest

What if you are not sure, whether you can fully trust your test suite? What if your doubts stop you from making necessary refactorings? If we can't trust our tests, our code base will degenerate and drown in technical debt. PITest can help!

PITest is a JVM-based mutation testing tool. Mutation testing is a form of code coverage evaluation. All you need to run it is a working, “green” test suite. PITest changes the byte code of your production classes by applying certain mutations (e.g. inversion of boolean expressions, changing return values to null etc.). If your test suite discovers the resulting mutants and turns red, the mutant is considered killed. Obviously, an ideal test suite will discover all mutants.

Mutating WebTester

We tried PITest on the latest release candidate of testIT WebTester 2. WebTester 2 uses a Maven build and configuring PITest amounted to adding a plugin and pointing it to the production classes and the test suite package.

<plugin>
    <groupId>org.pitest</groupId>
    <artifactId>pitest-maven</artifactId>
    <version>1.1.10</version>
    <configuration>
        <targetClasses>
            <param>info.novatec.testit.webtester.*</param>
        </targetClasses>
        <targetTests>
            <param>info.novatec.testit.webtester.*</param>
        </targetTests>
    </configuration>
</plugin>

<groupId>org.pitest</groupId>

<artifactId>pitest-maven</artifactId>

<param>info.novatec.testit.webtester.*</param>

</targetClasses>

<param>info.novatec.testit.webtester.*</param>

</targetTests>

</configuration>

</plugin>

We then ran PITest by executing the Maven goal mutationCoverage against our AssertJ support module.

$ mvn pitest:mutationCoverage

Our initial run of PITest took about 10 seconds, 10 times longer than the average run of our suite of 101 unit tests. It generated 46 mutations. But only seven mutants were swatted down by our tests!

PITest generates an HTML report with line and mutation coverage statistics grouped by package.

What wasn’t covered?

Missing fluent API check

AssertJ offers a fluent assertion API, so our extension does the same. But there was not a single test checking that the returned instance was (1) non null and (2) the same assertion instance.

Being the conscientious developers that we are, we added tests for each assertion method checking the fluent API semantics.

Default null check not covered

The base classes for most of our assertions executes a null check on the actual value. But none of our tests ever checked if a null value is handled correctly.

Oops. An obvious miss in hindsight.

Conclusion

Eva Rinaldi – Wolverine – CC BY-SA 2.0

It took us just a few minutes to setup PITest, and it already proved to be a great help in making our unit test suite more robust. The mutations applied to the byte code seem to correlate with common programming mistakes (e.g. simple typos or actual misreadings of boolean logic). Give it a try! How robust is your test suite?