Debugging on Kubernetes - the perfect developer machine

Building Applications and Debugging on Kubernetes is difficult. How can we debug our Application that runs in the cloud and is dependent on services that can not be run locally? In this post we will use currently available tools to make our applications "debuggable" within our cluster.

Cloud native development, cutting your service into granular units and developing microservices is already a challenge for developers. Mix in Kubernetes and its YAML files, deploying your application onto a cluster while keeping factor 10 of the 12 factor guidelines: Dev-prod parity and you have a challenge which can seem nearly impossible.

To solve those problems a lot of tools and little shell scripts came into existence, and as it is with the extremely fast paced life cycles the cloud has, a lot of them already died. Those tools solve some problems but opens a new one: What tool can do what and what should I use to help me with my problems? In this blog post I will try to help with one of the problems I had the most with developing services for Kubernetes: local development and debugging on Kubernetes.

The easy way: kubectl port-forward

I guess everyone had this problem: You develop a new service or a new feature for an existing service, which needs to access some other service that runs on the cluster and is not accessible locally from your machine. Already you have a problem, since starting your whole application landscape with all dependencies and needed services most of the time is not possible on a single machine a developer has access to. But running your new service in the cloud comes with a lot of work and, quite possible, deployment specifics most of us don’t want to touch while developing a new service. The easy way to do this comes with kubernetes: kubectl port-forward.

kubectl port-forward allows you to forward traffic from your local machine into one pod on the cluster. This way you can forward all the services you need from your cluster to your local machine and use them in your new service. Especially if you develop with e.g. Spring Boot and can have a local configuration profile, this method is easy, fast and mostly in line with the dev-prod-parity we try to keep. But not fully, since you need to configure a local configuration profile or at some place edit the access URLs to your backing service to use your local host one. Also it can be quite a challenge to keep track of all ports and services assigned to it, especially when working on multiple services, each using different ports.

The practical way: kubefwd

A way to solve the chaos, that can happen by using kubectl port-forward, is kubefwd. It allows to forward a whole namespace with all services running inside it. Also kubefwd edits your local hosts file to add DNS names for each service. Basically, instead of managing all your port forwards your own kubefwd takes over this job for you. This allows you to access your services by their service-names, just as you would on your Kubernetes cluster. This allows you to use the same configuration profile for all your connections in your application.

The fancy way: telepresence

kubefwd is awesome for services and inter-service communication, but what if, like most of us do, you use environment variables to configure some part of your service in the cluster and you need to debug exactly this part? This is where telepresence comes into play.

Telepresence is a two way proxy to your cluster. Once installed you can run the telepresence CLI. This spawns a new pod within your cluster that you can use to access a shell in your cluster, which is nice, but would not need a standalone tool. Even better then running a shell in your cluster is the –swap-deployment flag. This flag allows you to swap out a deployment within your cluster with one that uses any image you want from your local machine. This new container runs on your local machine, but gets all traffic from your cluster as well as proxies all your traffic to it into the cluster. And what makes this so great, all the environment variables, Kubernetes configurations and everything that is normally only accessible in your cluster is proxied to be accessible in this container too. It is even possible to use Kubernetes clients in your application and talk to the Kubernetes API from within a running container on your local machine.

But this post is about debugging, and of course, debugging is also possible, when a bit more tricky. Unfortunately I couldn’t find an easy, one-command-without-deeper-knowledge-of-docker-way to enable this. But since the documentation of telepresence is awesome, there already is a finished script, which you can use to run a maven container, expose the debug ports and set all needed configurations to build your application (if it is a maven/java based application) and start it (the same is true for PHP). Then you can simply attach your remote debugger within the IDE of your choice and you are done. You now can set breakpoints and debug your application as it would run within your kubernetes cluster.

telepresence --swap-deployment <deployment> \
 --docker-run --rm \
 -e MAVEN_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -v$(pwd):/build \
 -v $HOME/.m2/repository:/m2 \
 -p 8080:8080 \
 -p 5005:5005 \
 maven mvn -Dmaven.repo.local=/m2 \
 -f /build spring-boot:run

telepresence --swap-deployment <deployment> \

--docker-run --rm \

-e MAVEN_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -v$(pwd):/build \

-v $HOME/.m2/repository:/m2 \

-p 8080:8080 \

-p 5005:5005 \

maven mvn -Dmaven.repo.local=/m2 \

-f /build spring-boot:run

Further tools and reading

As always, there are other tools, which can help you access and debug your application. Unfortunately I could not try them as long as I would like so I collect them here.

squash
squash is a debugger for kubernetes. It spawns a squash pod which attaches a remote debugger to your application pod, which you can attach to. At least that is what I understood of it, as the documentation is still a bit shallow. I had some problems getting it to run and could only get it to run within VS Code, which is not my preferred IDE for my use case.
docker-compose
This is the “I run everything local” approach. If the services you use are small enough to do this, maybe this is the solution for you. I personally don’t like to run more then I need and so use docker-compose only rarely.
minikube
While minikube can help you deploy all the services and run them on kubernetes, and you can attach your local container registry, the major problem of debugging can not really be solved with minikube alone. Telepresence however can interact really well with minikube, so the combination might solve some problems like missing permission on a given Kubernetes cluster.

Also I’d like to link some resources and people who already worked on this topic and I used as input for this post