user-icon Stefan Ludwig
01. December 2016
timer-icon 3 min

Enforcing Non-Functional Test Requirements

Most code we write – be it production or test – has non-functional requirements:

As an example, unit tests should …

  • not take longer than 10 ms
  • not interact with the file system
  • not make use of a database
  • not waste time waiting for something

With this post I will demonstrate how to check one of these requirements using JUnit 5. For this I will implement an extension to measure the execution time of tests and throw an exception in case a test runs longer than 10 ms.

In the callback method beforeTestExecution the current time is stored in nanoseconds. We use nanoseconds because System.currentTimeMillis() is not precise enough when our threshold is in the low milliseconds.

This value is used as the start timestamp in afterTestExecution. Here we calculate the difference between now and when the test started to get it’s exact duration, or at least as close as we can get it. If this duration is higher than 10 milliseconds, we throw an exception stating the rule and the actual duration of the test.

To use this rule we can write a test like this:

But since it is more than likely that we will implement some other rules in the future, we should come up with a better solution than declaring each rule for each test class. Luckily JUnit 5 offers composite annotations which allow us to do the following:

For each test class annotated with @ApplyUnitTestConstraints, all the specified extensions will be loaded. This means we can add rules later, without having to change any of the tests.

Of course this implementation of the rule is rather simple. It is not configurable, a fixed threshold of 10ms might be prone to false positives, and we might want to be a little bit more lenient with throwing an exception.

So the final rule could look a little more like this:

In this version of the rule, we define three thresholds:

  • WARNING logged for tests that run longer than 10 ms
  • ERROR logged for tests that run longer than 25 ms
  • EXCEPTION thrown for tests that run longer than 50 ms

Each of those thresholds can be overridden via system properties.

Where to go from here?

This is of course only one of many possible rules we could implement to create an early warning system for our tests. But why limit ourselves to rules?

If we change our @ApplyUnitTestConstraints annotation to a more generic @UnitTest, we introduced categorization to our test suite. This allows us to apply specific extensions and properties to certain types of test.

As an example: Unit tests often make use of mocks. Now we could write a simple extension to setup our mock objects for all unit test:

We could also add @Tag("unit-tests") to our @UnitTest annotation in order to allow for the selective execution of ‘all unit tests’ via command line or in a certain build phase.

With JUnit 5 and extension the possibilities are endless!

As always, you can check out the source code for this post on GitHub.

Comment article