Approval Testing - Enable Riskless Change

Wait, what is Approval Testing?

In a nutshell, you can say that approval testing is a testing approach that captures the output of a system and compares it to a previously approved version. If both versions match, the test passes. Otherwise, it fails and you have the possibility to review the changes. If a change is intentional, you can approve the current version.

This approach is quite different from the more familiar assertion-based testing technique. Using assertion-based testing means to write tests for every single behavior that should be preserved. If a possible behavior change is not considered sufficiently, it could pass unnoticed and probably might result in major problems at the end of the refactoring process. This can be interpreted as blacklisting of changes. You have to think of a change in advance and create a test to detect it.

On the other hand, using approval testing means only the exact current behavior is accepted. Even the smallest change is seen as a difference and leads to a failing test. If such a change intentional, it has to be approved explicitly. This can be seen as a whitelisting of changes.

Which technique is chosen depends much on the context. In the case of risky changes to a not well-understood legacy code base, the whitelisting approach is more suitable because everything that is different from the approved version will be recognized.

Interested in details? Check the article of Geoff Bache and Jeremias Rößler for more information about approval testing as well as blacklisting and whitelisting of changes.

jApprove – Approval Testing in Action

To easily apply approval testing, I needed an appropriate tool. Therefore, I created jApprove. A simple JUnit 5 extension that supports developers to quickly set up approval tests for their Java applications. I got the main idea for jApprove from Approvals, which enables approval testing for several other programming languages. The main focus of jApprove is on API testing, especially for REST/JSON. But the idea generalizes and basically everything that can be represented as a string can be verified.

It is still under development, but feel free to have a look at how it works! jApprove is available on Maven Central, so you can use it directly in your project. It also comes with plugins for Maven and Gradle.

To use jApprove in your Gradle project, simply add the following dependency to your project’s build.gradle:

Also add the corresponding Gradle plugin:

In the case you’re using Maven, add the following dependency to your project’s pom.xml:

Also add the corresponding Maven plugin:

To truncate the Maven commands, I recommend to add the plugin group to your settings.xml (usually located in the ${user.home}/.m2/directory):

That’s it! Now, it is possible to write approval tests.

Let’s create a simple object returning a health status as a JSON Object:

Of course, jApprove truly shines when handling large and deeply nested JSON objects but to keep things simple, this example is sufficient.

Now, let’s write the first Approval Test:

An approval test is simply created by annotating the method under test with @ApprovalTest and pass a Verifier-object as a parameter. In this case we use as specialized JsonVerifier for our JSON object. The actual test looks very similar to a usual JUnit test using the Arrange, Act and Assert (AAA) pattern. The crucial difference is the final step where the verifier is called instead of asserting. The verifier reads the content of the JSON and compares it to a previously approved version. This version is stored as the so-called baseline that can be set as a parameter of the @ApprovalTest annotation. If no parameter is set, a name is generated.

The test can be started like a usual JUnit test (e.g. by clicking the respective button within an IDE). Because there is no approved version, the test obviously fails when running the first time. The error message informs you about this circumstance in the console. As a side effect, the file status.txt has been created in the build/baselineCandidates/ directory. This directory is used to store all unapproved versions. To approve this version, use the following command:

In the case you’re using Maven, use this command instead:

With the approve command the approve function of the respective plugin is called. The --baseline and -Dbaseline options are used to specify the baseline of the respective test. In this case, the baseline is called status.

If you start the test again, it should pass successfully. Yes it should! But it doesn’t. According to the error message occurring in the console, the timestamp is different. This makes sense, because the timestamp represents the current system time and it is different every time you run the test. The solution is to exclude this timestamp from the comparison. This can simply be done by the ignore-method containing the field that should be excluded:

Because this is implemented with JsonPath, you can exclude fields even in deeply nested JSON objects. Finally, if the test is started again, it passes successfully. Meanwhile, the file status.txt has been moved from build/baselineCandidates/ directory to a new baselines/ directory, where all approved versions are stored.

If you now make changes to the the code, you can simply rerun the test to verify that the behaviour is still the same. If there are changes, the test fails and the error message shows the differences. At this point, if the changes are desired, you are able to approve them and create a new baseline version. Otherwise, you can revise your work and run the test again.

What’s Next?

This blog post just outlined the basic functionality of jApprove. But if it has aroused your interest, you are welcome to visit the jApprove GitHub Repository. There you can find a detailed documentation of all features and further examples. I am also about to extend this tool by further features. Feel free to try it by yourself and to contribute to jApprove!