Prometheus Metrics Autoscaling in Kubernetes
Prometheus Metrics Autoscaling in Kubernetes
This set-up demonstrates how we can use the Prometheus adapter to autoscale deployments based on some custom metrics.
Join the DZone community and get the full member experience.Join For Free
One of the major advantages of using Kubernetes for container orchestration is that it makes it really easy to scale our application horizontally and account for increased load. Natively, horizontal pod autoscaling can scale the deployment based on CPU and Memory usage but in more complex scenarios we would want to account for other metrics before making scaling decisions.
Enter Prometheus Adapter. Prometheus is the standard tool for monitoring deployed workloads and the Kubernetes cluster itself. Prometheus adapter helps us to leverage the metrics collected by Prometheus and use them to make scaling decisions. These metrics are exposed by an API service and can be readily used by our Horizontal Pod Autoscaling object.
We will be using Prometheus adapter to pull custom metrics from our Prometheus installation and then let the Horizontal Pod Autoscaler (HPA) use it to scale the pods up or down.
- Basic knowledge about horizontal pod autoscaling
- Prometheus deployed in-cluster or accessible using an endpoint.
We will be using a Prometheus-Thanos Highly Available deployment.
Deploying the Sample Application
Let’s first deploy a sample app over which we will be testing our Prometheus metrics autoscaling. We can use the manifest below to do it
This will create a namespace named nginx and deploy a sample Nginx application in it. The application can be accessed using the service and also exposes nginx vts metrics at the endpoint
/status/format/prometheus over port 80. For the sake of our setup, we have created a DNS entry for the ExternalIP which maps to .
These are all the metrics currently exposed by the application:
Among these, we are particularly interested in
nginx_vts_server_requests_total. We will be using the value of this metric to determine whether or not to scale our Nginx deployment.
Create Prometheus Adapter ConfigMap
Use the manifest below to create the Prometheus Adapter Configmap.
This config map only specifies a single metric. However, we can always add more metrics.
Create Prometheus Adapter Deployment
Use the following manifest to deploy Prometheus Adapter
This will create our deployment which will spawn the Prometheus adapter pod to pull metrics from Prometheus. It should be noted that we have set the argument
--prometheus-url=http://thanos-querier.monitoring:9090/. This is because we have deployed a Prometheus-Thanos cluster in the monitoring namespace in the same Kubernetes cluster as the Prometheus adapter. You can change this argument to point to your Prometheus deployment.
If you notice the logs of this container you can see that it is fetching the metric defined in the config file.
Creating Prometheus Adapter API Service
The manifest below will create an API service so that our Prometheus adapter is accessible by Kubernetes API and thus metrics can be fetched by our Horizontal Pod Autoscaler.
Testing the Set-Up
Let’s check what all custom metrics are available
We can see that
nginx_vts_server_requests_per_second metric is available. Now, let’s check the current value of this metric.
Create an HPA which will utilize these metrics. We can use the manifest below to do it.
Once you have applied this manifest, you can check the current status of HPA as follows:
Now, let's generate some load on our service. We will be using a utility called Vegeta for this.
In a separate terminal run the following command.
Meanwhile monitor the Nginx pods and horizontal pod autoscaler and you should see something like this
It can be clearly seen that the HPA scaled up our pods as per the requirement, and when we interrupted the Vegeta command, we got the vegeta report. It clearly shows that all our requests were served by the application.
This set-up demonstrates how we can use the Prometheus adapter to autoscale deployments based on some custom metrics. For the sake of simplicity, we have only fetched one metric from our Prometheus server. However, the adapter Configmap can be extended to fetch some or all the available metrics and use them for autoscaling.
If the Prometheus installation is outside of our Kubernetes cluster, we just need to make sure that the query end-point is accessible from the cluster and update it in the adapter deployment manifest. With more complex scenarios, multiple metrics can be fetched and used in-combination to make scaling decisions.
Feel free to reach out should you have any questions around the set-up and we would be happy to assist you.
This article was originally published on https://appfleet.com/blog/prometheus-metrics-based-autoscaling-in-kubernetes/.
Published at DZone with permission of Sudip Sengupta . See the original article here.
Opinions expressed by DZone contributors are their own.