Scale your Spring Boot Application in Kubernetes

Objective
At the end of this post we have a running Horizontal Pod Autoscale (HPA) that scales my SampleApp on Actuator metrics. To reach this goal we will deploy Prometheus, which is used to scrape all our Pods. After that we will deploy the k8s-prometheus-adapter, which enables K8s to use the provided metrics for scaling.

Architecture Diagram showing the services needed to scale on custom metrics
Prerequisite
Before starting with deploying everything, I want to list and introduce the required and the recommended tools.
- minikube or any other way to run Kubernetes locally
- Any way to build a Spring Boot Application, I use maven
- go, to get cfssl and other Tools that are really great
- hey to generate some load and traffic
- Finally jq to make the K8s API readable
After all the preparation we can start with the hand-on part
Sample Application Spring Boot
As we want to scale on Spring Boot Actuator metrics (at least for the sake of this demo), we need an application that provides these metrics to an endpoint. For this you have to copy the following dependencies into your pom.xml.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
<!-- Prometheus, allows to publish metrics in Prometheus format --> <dependency> <groupId>io.prometheus</groupId> <artifactId>simpleclient</artifactId> <version>0.5.0</version> </dependency> <!-- Spring Boot Metrics collector and management --> <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-registry-prometheus</artifactId> <version>1.0.6</version> </dependency> <!-- Provides basic Metrics like Health state --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> </dependency> |
Those dependencies add the Endpoint /actuator/prometheus, which publishes the actuator metrics in the Prometheus format.
Deploying everything to Kubernetes
Let’s get to the most interesting part – deploying everything to Kubernetes.
Namespace
First you will have to deploy the namespace custom-metrics. Most of the parts will run in this namespace, for example the Prometheus instance.
Prometheus
Prometheus can either be deployed via the prometheus-operator or (as I did) via some Kubernetes yml files (see here). To keep this easy, you don’t have to configure anything for Prometheus to get your deployed pods, those will be detected later via Kubernetes annotations. The given configuration will add some metrics for Kubernetes objects like cAdvisor.
k8s-prometheus-adapter
This is where the magic happens. To use the provided metrics and Prometheus query tools as a scaling target you have to provide the K8s API Endpoint custom.metrics.k8s.io. This will be done by the adapter. Let’s get this deployed.
I highly recommend reading the documentation for each of the following steps as I can’t go into all the nitty-gritty.
Generate certificates
Yes, you will have to generate some certificates. I used the gencerts.sh script from the Prometheus Operator. You will need to install cfssl and openssl (if not already done) for this.
Deploy Adapter
After creating the certificates you can deploy the k8s-prometheus-adapter with this yaml. This also provides one rule that the adapter will use to fetch metrics.
Configure Adapter
The configuration is provided as YAML within an ConfigMap (already deployed) in Kubernetes.
1 2 3 4 5 6 7 8 9 10 11 12 13 |
rules: - seriesQuery: 'http_server_requests_seconds_count{job="kubernetes-pods", app="sample-app"}' seriesFilters: [] resources: overrides: kubernetes_pod_name: resource: pod kubernetes_namespace: resource: namespace name: matches: http_server_requests_seconds_count as: requests_per_second metricsQuery: rate(http_server_requests_seconds_count{job="kubernetes-pods", app="sample-app", uri="/requests", <<.LabelMatchers>>}[5m]) |
This provides the metric http_server_requests to use within any HPA. The Adapter provides this metric for each Pod, mapped by the label kubernetes_pod_name. The value of this metric is generated by the following Prometheus query:
1 |
rate(http_server_requests_seconds_count{job="kubernetes-pods", app="sample-app", uri="/requests", kubernetes_pod_name="<Inserted Pod name>", kubernetes_namespace="<Inserted Namespace>"}[5m]) |
Sample App
To deploy the Sample App you need a Container Image and a deployment yaml. I will not describe how to create the image needed, as there are mulitple online resources available. For the deployment you can use just copy the following deployment.yaml and enter your image. Note that the important thing are the annotations. You need those so Prometheus scrapes your Actuator metrics from the pods.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
apiVersion: apps/v1 kind: Deployment metadata: name: sample-app labels: app: sample-app spec: replicas: 1 selector: matchLabels: app: sample-app template: metadata: labels: app: sample-app annotations: prometheus.io/path: '/actuator/prometheus' prometheus.io/port: '8080' prometheus.io/scrape: 'true' spec: containers: - name: sample-app image: coschapoehler/scale-spring-boot-sample:0.0.1 ports: - containerPort: 8080 |
Test if everything works
Lets check if every deployed object works. First we check our Prometheus instance. It should run and in the web interface show a kubernetes-pods job scaping our deployed sample app.

Prometheus Interface showing scraped kubernetes Pods
If this works we can skip the check our Sample App deployment as Prometheus can only pick it up if everything works correctly. Next we check the adapter. For this, simply query the Kubernetes API:
1 |
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/" | jq |
If the output looks like this, everything is ready to go.

Output showing the registered API Endpoints for custom metrics
HPA on custom Metrics
Lets deploy the HPA and see, how Kubernetes autoscales our application.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: name: sample-app namespace: default spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: sample-app minReplicas: 1 maxReplicas: 10 metrics: - type: Pods pods: metricName: requests_per_second targetAverageValue: 500 |

HPA Autoscaling based on custom metrics
Pitfalls that happened to me
- Internal Server Error
My metricsQuery within the Config was invalid. You can see this by trying to query a metric using
1kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/<namespace here>/pods/<podname here>/<metric name>" | jq
If this returns an Internal Error, than most likely your config is not correct. - Spring Boot App not providing metrics
Most likely this is caused by not exposing the endpoints. In your application-properties (or yaml) dont forget ro expose at least the prometheus endpoint
1management.endpoints.web.exposure.include=prometheus (or expose all endpoints, when working locally) - HPA not picking up metrics
For whatever reason, at some point the deployed HPA did not pick up any metrics. To fix this is redeployed it and it worked, but I don’t think this is the best idea. Check the API Endpoints first to see that the value for metrics is correctly written. After that, check you HPAs events and see what the tell you. Most likely a redeploy will help, but if not it might be a problem with your configurations
I hope this helps you to find your problems and maybe solve them. Thanks for the read.
P.S. All my Ressources can be found in my repository: https://github.com/Etone/SpringBootActuatorScalingK8s
Recent posts

Automated API Security Testing with OWASP Zap and Open API

How to support different JWTs in your Spring Boot application

Testcontainers - Bring Your Integration Tests to a New Level

How can the MDC context be used in the reactive Spring applications

Spring Boot ConfigurationProperties on Base Classes in Kotlin

Comment article