Maintaining comprehensive suites of tests can take a lot of effort. Sometimes so much that it makes us doubt that all this work is worth the moderate increase in confidence that our software is correct.
Testing is always a balance between cost and risk reduction, and picking the right testing method can greatly improve this trade-off for us. That’s why we always keep our eyes open for better ways to test our software.
Property-based testing was introduced over a decade ago as Haskell’s QuickCheck and libraries are available for many functional programming languages. And since the Java world has become more aware of functional programming techniques, we think it is time to have a look.
Property-based testing is a generative testing method, as opposed to the usual example-based tests. As developers, we specify desired properties of the component under test, then have the testing framework attempt to disprove them. The framework does this by generating hundreds of arbitrary inputs and checking for each whether the property holds. If a check fails, the framework provides us with the generated inputs for which the property doesn’t hold. Running such tests in a CI pipeline gives a lot of confidence that we didn’t miss any corner cases.
We show this for a simple example using the library junit-quickcheck.
Coin Changer
Our example is a simplified version of the Coin Changer Kata. Imagine a machine that changes a given amount into coins of 200, 100, 50, 20, 10, 5 and 1 cents. The machine should return the smallest number of coins for a given value, e.g. for an amount of 102 cents it should return three coins, 100, 1 and 1. If you never practiced this kata, you may want to take a few minutes to implement your own coin changer.
In example-based testing, e.g. the tests we use to drive the development of the actual coin changer, we would manually specify an input amount and the coins we expect to be returned. This is useful while we develop the solution, but we can miss important corner cases!
Let us instead specify a fundamental property of a correctly implemented coin changer: the returned coins always sum up to the specified amount. In junit-quickcheck we would express this as:
Notice that we annotated the test method with @Property and that it is parameterized, taking a single integer as input. The JUnitQuickcheck runner will recognize this method as a property and generate input values to check it. We configure the input value generation with another annotation, @InRange, limiting it to values between 0 and 500. The assertion in the last line is from the excellent assertion library AssertJ.
Another property of a correct coin changer is that it returns the minimal number of coins required. A coin changer implementation that returns 40 cents in four 10 cent coins is incorrect, while a correct version would return two 20 cent coins.
More generally, smaller coins summing up to a larger coin should be replaced by that larger coin. We can express this as a property-based test:
As you can see, the tricky part of property-based testing is finding and expressing the properties to verify. This forces us to formally think about the assumptions we make about our code and putting them to the test. This in turn significantly improves our understanding of the problem and our solution.
In our experience it is often unnecessary to specify the behavior of your code in detail. A few simple properties that can be expressed without duplication of production code already provide a great complement to example-based tests.
Comment article