Cucumber without BDD - is there any value in it?

Cucumber and behaviour driven development (BDD) are closely related for most people. But at its core, cucumber is a test runner that aims to combine the gherkin notation with glue code to validate a system under development. Technically, it can be used within teams that do not do actual BDD. In this blog post, we will talk about Cucumber without BDD and the question if there is any justification for it. At first, we will briefly cover the reasons for doing this and a little warning. Then we will talk about potential benefits. Afterwards, the downsides and pitfalls of said approach will be covered before forming a conclusion.

Why would a team use Cucumber without BDD?

This might have a several reasons, but I will only name those I witnessed myself. I will neither argue towards nor against them, just list them for us to understand where those people are coming from.

1. Some people wanted to convince business experts to try BDD. That failed, but Cucumber got established as a tool.
2. Teams can try to satisfy test management tools. In a project of mine that was the Xray Jira Plugin. The project had to document testing efforts for potential audits. The new system provided offered two ways of doing so: manual testing and cucumber tests in gherkin notation. Used correctly, this can be a chance to get into a better direction, enabling BDD and thus strengthening the team communication.
3. Other teams tried to solve special cases with a BDD like approach where QA and business people only wanted to test a small part of the possible cases. And they wanted to define their expected behavior for the remaining ones, so it can be automated. More on that later.

I am pretty sure there are several more reasons for that to occur. Do you have another reason? Please leave us a comment so we all can learn together.

Cucumber without BDD might be a band-aid fix

If we use cucumber just to have a file we can actually understand as developers, it is a sign for an unsolved problem.  The true benefits of frameworks such as cucumber get visible, if any business experts are at least interacting with the examples. They could use them as a baseline for a discussion about further improvements.

Is this not the case? Then we should think carefully about the structure of our other tests. Do we cover up improper testing? Yes? Then it is likely that we waste a lot of time writing more complex glue code than simple (i.e. JUnit) Tests. It gets even more important, as those kinds of problems rarely arise alone. Often times other things are lacking as well, such as system decomposition or testable architecture.

A team following this approach I worked with had a poor architecture which lead to bad testability. They “fixed” the problem by writing cucumber tests that were executed on the API or at best component integration level. Testing even the smallest of things was a chore. Huge parts of the system, i.e. the database state, had to be constructed instead of just performing a method call to get to the same result. In the end the examples were written with two goals: to formulate test readable and to satisfy the test management system. The latter is a problem, I do not want to tackle in this post.

Examples that are incomprehensible for non-tech people were the result. But with a little of work, we eventually could fix at least that. In some areas, we also improved the architecture and in the process the testability. And comparing features that followed the new structure with other parts of the system shows two things: Written examples can be used as a tool to talk to business people and simple things can be tested using simple techniques in the new approach.

But enough warnings, let’s get back to the topic.

Benefits of feature files written by developers

Feature files, if done properly, are written in human readable language. And by that we mean English, German, Spanish and alike. The reports created by executing the examples will lead to a documentation that is up to date, validated and most important: understandable for developers and business people alike. And hopefully it covers the features with the highest value or greatest risks.

Helps to define or fix features

So let’s assume we have a bug in our production environment. We might analyze the reports with our business experts and see if an example is wrong or something is missing. The focus should be on the reports, not the plain examples, as those provide a better overview for several reasons. We all understand the current state of our system as it is written in natural language. Chances are, we will find the problem easier and faster. Even better: In a good amount of cases, missing examples fit into our existing glue code and are automated with little effort. And the same chain of arguments can be made for developing new or extending existent features. If we benefit like this on a regular basis, Cucumber without BDD can be worth the effort.

Again: if you have to read it out to business experts and act as a translator, you could use something that is easier for development to start with. In the project I mentioned before, there were such situations. That was exhausting because I did not only have to translate the examples, but there was a need for more automation work afterwards as well.

Can be an approach for specific scenarios

Recently I saw a team, that uses .feature-files only in certain situations. In pretty rare ones as well. But those cases are important for the business experts. And they seem to appreciate understandable and automated testing for those things. Do they have a complete and living documentation of their software? For sure not. But they found a problem where they want both sides to be able to understand the tests and the solution works for them. In other situations, they noticed an increased time investment needed without using the benefits and replaced it with other approaches.

To be clear: We could solve those scenarios with less complex tooling. You could use parameterized tests with a .csv-file for example and save the hustle of integrating cucumber all together. But choosing cucumber here is not a problem as long as it works for your team. Just be aware of other options and consider them. It can save a good amount of time.

The team I worked with had some additional benefits: Their test management tool supports Cucumber by default and provides a way to pass audits. So it was a net win on their end, but the situation was pretty specific.

Potential downside of Cucumber without BDD

The potential benefits mentioned above seem attractive. But they come at a cost and as always, we need to justify that. Our goal is to deliver high quality software that is beneficial for our organization. And in the present case, there are some traps that might even damage our efficiency.

The language of the examples is not used by business people

Developers and business experts use different languages. And even worse, they often times use the same words for completely different things. To experience the benefits of an approach where Cucumber is used without having real BDD, non-developers have to be able to understand the files without having too many questions. Otherwise, we need a translator and I think anyone reading this starts to notice a pattern here.

The fix for this issue is rather simple. If teams discuss examples, for instance in a scrum refinement meeting, they can iron those things out easily. And even better, the team can develop a common language step by step. This is not only useful for discussing examples but for all sorts of communication.

Examples get highly technical

Related to the above is the use of purely technical terms within examples. Let me explain this with an example that imitates things I saw in a former project:

This is troublesome in many ways. The first thing that might be visible is, that this example is obviously not written from a user perspective as they are rarely interacting with REST-Controllers. Further, both http status code and request body are technical terms business experts most likely do not understand. They do not care about them and they should not need to. And there are other problems in the example as well. But let´s rewrite that example instead of picking it apart.

Obviously, this is both an easy example and an exaggeration. But things like this happened a lot in a former project of mine. This lead to the .feature-files being for developers only and we meet our overarching issue of cucumber usage with little to no benefits again as the business people will not be able to use them at all.

Improper file naming

How could a file be named badly? In the team I mentioned before, .feature-files where separated and named with the tags of the stories. So files were called like XYZ-1234.feature. But why is this wrong? Feature file should represent features as suggested by the extensions name. A user story is not equal to a feature. Features can be implemented in several user stories and thus, the examples as well as the documentation would be distributed. Further ticket number tend to get forgotten. This leads to us having to search for information in several files if we need to have a look at the examples.

Assuming the examples above, they should not live in a file called XYZ-1234.feature. Those examples are a part of the document upload feature. So the proper file name could be something like document-upload.feature. If we want to reference user stories, we can use Cucumbers tag feature.

Without using the benefits, Cucumber without BDD is just expensive

Given real-world problems, efficiency is an important topic. In companies, we do not write software for fun. We do so to generate business value in any form. Most of the time, it can be measured in things like revenue or customer satisfaction. Now imagine those conditions: Instead of writing a plain JUnit, Jest or Selenium Tests, we add another layer of complexity into the mix. Writing the examples in gherkin notation and gluing it to our system takes time. And with that, it costs a good chunk of time and money. And if we do not use the benefits that Cucumber and BDD can provide, there is no return on invest.

Verdict

To summarize the topic: In an ideal world, business people, developers and testing experts work together to come up with good examples that can be automated while developing the features and create a living and ever-growing documentation. In reality, that does not always happen. Teams might benefit from examples in gherkin notation nevertheless. But there are some requirements that have to be met.

First of all, business people have to be able to understand the examples. Otherwise, they are not able to give proper feedback. And it should provide a benefit to the project in some form. Be it in a specific scenario or as a basis for discussion. And lastly teams should be cautious and check, if they only use it as a band-aid fix or a cover-up for deeply rooted problems such as subpar architecture.

Did I miss a point or do you have other questions related to testing approaches? Please feel free and write a comment or contact me via E-Mail.