Unleashing the power of software

Software is eating the world – Marc Andreessen.

Have you ever asked yourself, why is that? Why is software having such a big impact in every industry?

Let’s say you have an idea and you just need a couple of weeks to convert that idea into a product or service. Or imagine you have a product for which you get constant feedback from your users and therefore you can improve the product on a daily basis. Both, the speed of change and the feedback cycle is what make software so powerful.

The speed and continuous improvements in software give your product a lot of flexibility for adapting to market changes no other industry can’t match. Envision a continuous stream of small changes that perfectly adapts to the always changing user requirements.

Delivering quality software in short cycles gives you an advantage over your competition. However, delivering quality software in short cycles is not an easy task. You need a certain way of thinking and structure in your organization to achieve such a feat.

Most of the way of thinking and structure you need is about eliminating overhead, tearing up barriers that cause waste and lower productivity.

Cloud

It is 2020 and we are still talking about cloud. Cloud is almost essential to achieve the goal of delivering software in short cycles. It should not matter if we are talking about a public cloud provider or on-premise. The main advantage of the cloud is not cost savings, is speed.

When you develop for the cloud, you have a lot of services at your disposition. Does your product need messaging and a NoSQL database? The cloud has them. Not only does it have them but you can create those resources in a matter of minutes, it is just a click away – or a couple of lines in your automation script. You don’t need someone else to do that for you or your team. If you need those resources, you simply create them.

Imagine you get feedback from your users and you realize you need to change some functionality and probably add some new features. In order to do that, you realize you need more resources like storage, distributed databases etc. Not a problem in the cloud. You can start using those services after a couple of minutes.

Your product became too popular and now you get a lot of load? You can easily scale on demand in the cloud without having to invest time and resources for that.

Now, many people are wary of vendor lock-in in the cloud. They do not want to use a certain service because it means lock-in even if that would mean ease of development and value creation at low cost. Remember, you are in the business of creating value. You want to differentiate your product from the competition.

If some service gives you leverage for achieving your goal, you should really consider it. The problem is not vendor lock-in. The problem is trying to avoid lock-in at all costs. By doing that, you end up using the common denominator of all cloud providers. You think, you can create that special service you need by yourself or at least emulate most of its functionality. You will then have a significant amount of developers implementing and maintaining that service instead of having them developing your product. Essentially, you end up with a half baked service function-wise at many times the cost compared to the service offered by the cloud provider. Instead, you can see the lock-in as a way to “buy time”. You use a specialized service your cloud provider offers, so that you can concentrate on creating value for your users, once your product is successful, then you can start thinking how to avoid the lock-in, or, if is even worth avoiding it.

You will need to make the decision, the trade-off, about the level of the stack you want to use. The lower the level, i.e. IaaS, the more resources you have to pour into tasks that don’t create direct value. Likewise, the higher the stack level, the more resources you have at your disposition to create value. Yes, a higher level stack costs more and it is perhaps not as flexible as low level stack, but it will probably give you more speed and more resources to develop your product. You have to evaluate what is more important to you.

Cloud alone won’t give you speed. You also need trust. Trust in your team, so that they can create all resources and services needed for the application to work. If you migrate to the cloud but still have approval processes for creating resources, you are artificially applying a brake (and frustration) to your team.

Software engineering practices and company processes

Many companies, even those in the most conservative industries, have improved significantly in this area. Practices such as continuous integration and continuous delivery are already in place at many companies.

Unfortunately, almost nobody takes advantage of such practices. The development teams at a company are able to integrate and to deliver many times in a day. However, some company policies prevent them from doing that. They impose an artificial barrier. The reason? Risk avoidance. The main argument they name is that delivering software many times in a day is too risky. It could make everything unstable. The irony though, is that such practices do exactly the opposite: they minimize risk.

Delivering small software increments is the best way to achieve stability. A team can achieve that by having well functioning delivery pipelines in such a way that, if a deployment goes wrong, they can roll back to the last working deployment in just a couple of minutes. That is only possible if the change in question is small enough. Rolling out a big change is messy and complicated, rolling it back when it fails is usually more complex or at least it adds way too much complexity to the release process.

Some companies are still doing only 4 or 5 releases a year. The overhead of such releases is quite big. They need to plan a lot for every possible scenario. They need processes such “release management  or “release planning” which is basically coordinating a lot of teams on what they may and may not release. By doing that they are feeding another big problem: service coupling. They couple a service to another one to ensure compatibility. They also limit the speed of one team artificially by preventing the team from releasing some features because other team could not deliver the necessary changes for that release to work.

Doing small continuous releases still needs coordination but the overhead is way lower. There are also techniques like feature toogle that allow you to deploy features and activate them later without needing a new release.

Doing a cloud migration but still releasing a couple of times a year, is like driving a Porsche in a 30 km/h limit street. You have a super fast capable car but you are not using its whole potential. Essentially you could have just bought a cheaper and less capable car and saved a lot of money.

Software development

You don’t add innovation to a culture, you get out of its way. – Adrian Cockcroft.

Software development is a creative process. Writing software is about making decisions. The developer needs freedom in making those decisions to find the best possible solution.

Overhead culture drags down productivity. Imposing things like top-down developer practices and architecture will keep good developers away from your organization. It will also limit the output of those developers still working for you.

Unfortunately, there are still some dubious practices at many companies. I have known of many such practices. I will list a few such as:

• Imposing an arbitrary test coverage. Most of the time close to 100%.
• Having a check list describing all possible ways the application can fail and the possible solutions to those failures. This list is created before the development starts.
• Having an ivory tower architect or architectural board, which is a person or a team making decisions in isolation to your team or project.
• Having product owners approving every deployment.

This does not mean, your developers should use any tool and language they want just because. This means developers need room for creativity, for trying new stuff, to experiment. You need to trust your developers. You need to get out of their way. Remember, if something goes wrong, you can always roll back in minutes. This is the base of innovation, which leads me to the final and most important point of this post.

Agile culture

This is the most difficult aspect about software. Without agile culture you cannot take advantage of the power of software even if you have all other aspects in place like cloud, software engineering practices, software development etc. This is the aspect at which most companies fail.

I am deliberately not talking about agile / lean methodologies. They don’t bring much if the organization does not have an agile culture. Innovation is about giving room for experimentation, trying things and see if they work or fail. Failure itself is the best feedback. You learn a lot more from your mistakes than from your success. This is only possible in an organisation that embraces agile culture.

This is the reason why many big companies create small companies to experiment with “agile”, because their overhead culture is so embedded to the core of the enterprise, it is impossible for agile culture to flourish inside them. The problem is those small companies operate in isolation or, worse, they get their employees from the small companies back to corporate where they cannot work in an agile manner which means they throw away the learnings of their employees.

One of the main ideas of agile is this: short iterative feedback cycles. The duration of the cycle is important but the feedback itself is more relevant. Just deploying 10 times at day to production without taking feedback into account, does not provide any value. Where do you get the feedback from? From your users. How do you get the feedback? With data. It may sound kind of obvious, yet just a few companies do it.

A good value creation cycle looks more or less like this: You deploy a new version of your software, you do some measurements of the feature usages with data, you react accordingly to that data and do some changes in the software and deploy it again. This is the holy grail of software. This is how you can innovate. You create a new feature or product, you test it and you can validate the results. With software, you can do this in high frequency.

What about the feedback data? The feedback could be, for instance, a data stream from your application with which you can analyse how the feature is being used and even if it is being used at all. This is probably one of the biggest sources of waste in many companies: wasting resources in features nobody uses only because management thinks it is a good feature.

Many companies think that kind of feedback can only be implemented by startups or other similar companies. In reality, that kind of feedback can be implemented by any company that cares about it. The technology is there for your company to exploit it.

Conclusion

The main take away of the whole post should be this: everything you do in your company should focus on improving the speed and feedback cycle of your software development process.

Exploit the capabilities of the cloud instead of avoiding cloud lock in at all costs. Do not introduce artificial brakes like long approval processes.

Minimize risks by deploying small changes frequently.

Give developers freedom to create the best solutions to your problem. Do not bring down their productivity by overwhelming them with useless overhead. And, most importantly: trust them!

Embrace agile culture. This could potentially mean that you need to radically change how your organization works. Your competition is probably doing it. Some other small company you are not aware of is probably doing it.