Monitoring YugabyteDB in Kubernetes With the Prometheus Operator and Grafana
See how to monitor YugabyteDB in Kubernetes with the Prometheus Operator and Grafana.
Join the DZone community and get the full member experience.Join For Free
Using the Prometheus Operator has become a common choice when it comes to running Prometheus in a Kubernetes cluster. It can manage Prometheus and Alertmanager for us with the help of CRDs in Kubernetes. The kube-prometheus-stack Helm chart (formerly known as prometheus-operator) comes with Grafana, node_exporter, and more out of the box.
In a previous blog post about Prometheus, we took a look at setting up Prometheus and Grafana using manifest files. We also explored a few of the metrics exposed by YugabyteDB. In this post, we will be setting up Prometheus and Grafana using the kube-prometheus-stack chart. And we will configure Prometheus to scrape YugabyteDB pods. At the end, we will take a look at the YugabyteDB Grafana dashboard that can be used to visualize all the collected metrics.
Before we get started with the setup, make sure you have a Kubernetes 1.16+ cluster with kubectl pointing to it. You can create a GKE cluster or an equivalent in other cloud providers or use Minikube to create a local cluster.
We will be using Helm 3 to install the charts, make sure you have it installed on your machine as well.
The kube-prometheus-stack Helm chart installs the required CRDs as well as the operator itself. The chart creates instances of Prometheus and Alertmanager, which are the custom resources managed by the operator.
It also comes with a set of components which one would expect to have while setting up monitoring in a Kubernetes cluster. These components include: node_exporter for collecting node level metrics, kube-state-metrics which exposes Kubernetes related metrics, and Grafana for visualizing the metrics collected by Prometheus. The chart also comes with default Grafana dashboards and Prometheus alerts.
To setup Helm with the prometheus-community and YugabyteDB repositories, run the following command:
To create a values file for kube-prometheus-stack, execute:
The above kube-prom-stack-values.yaml file tells Grafana to import the YugabyteDB dashboard from the given GitHub URL.
Now we will create a namespace and install kube-prometheus-stack in it.
To check if all the components are running fine, check the status of pods from the
NOTE: To keep things simple, we are not providing any other options to the Helm chart. Make sure you go through the README of kube-prometheus-stack and grafana chart to explore other options that you may need.
With the monitoring setup done, let’s install the YugabyteDB Helm chart. It will install YugabyteDB and configure Prometheus to scrape (collect) the metrics from our pods. It uses the ServiceMonitor Custom Resource (CR) to achieve that.
To create a namespace and install the chart, run the following command:
If you are using Minikube, refer to this documentation section and add the resource requirements accordingly.
To check the status of all the pods from the
yb-demo namespace, execute the following command. Wait till all the pods have a
What Is a ServiceMonitor?
Similar to a Pod or Deployment, ServiceMonitor is also a resource which is handled by the Prometheus Operator. The ServiceMonitor resource selects a Kubernetes Service based on a given set of labels. With the help of this selected Service, Prometheus Operator creates scrape configs for Prometheus. Prometheus uses these configs to actually scrape the application pods.
The ServiceMonitor resource can have all the details like port names or numbers, scrape interval, HTTP path where application is exposing the metrics and so on. In the case of the
yugabytedb/yugabyte Helm chart, it creates ServiceMonitors for YB-Master as well as YB-TServer pods. Take a look at the API reference document for more information.
To make sure that Prometheus is configured correctly, port-forward to the Prometheus pod and visit
It should have targets with the name
yb-demo/cluster-1-yugabyte-yb-… listed there.
The targets in the above screenshot are folded for brevity.
Now that we have our metrics collection working, let’s take a look at the Grafana dashboard to visualize those metrics. The kube-prometheus-stack Helm chart also provisions Grafana for us. The Grafana instance provisioned by kube-prometheus-stack will have above Prometheus added as a data source. It will also have the YugabyteDB Grafana dashboard added by default.
To view the web user interface of Grafana, port-forward to its pod.
Visit the Grafana user interface at
http://localhost:3000 and login with the credentials
admin / prom-operator.
To view the YugabyteDB dashboard, search for the word ‘YugabyteDB’ by going to the ‘Search’ menu on the left (the magnifying glass icon).
The dashboard should look something similar to this:
We saw the Grafana dashboard, but it was almost blank. To get more metrics from YugabyteDB, we will run some client workload on it. We will be using the YugabyteDB workload generator to do so.
The following commands will create workload generator pods which will try to insert and retrieve some data on both the YCQL and YSQL layers.
Check if both workload generator pods are running:
Delete all the workload generator pods after 5 minutes and visit the Grafana user interface at
The graph panels should look something similar to these:
You can pick up the panels which are relevant to you and create your own dashboard as required.
To delete all the resources which we created as part of this post, run the following commands. These include Helm releases of YugabyteDB and the Prometheus Operator as well as the volumes created by the yugabyte chart.
That’s it! You now have a YugabyteDB cluster running on Kubernetes, with Prometheus and Grafana to monitor it.
In this blog post, we showed you how you can use the Prometheus Operator for monitoring YugabyteDB. We walked you through the steps of configuring and deploying kube-prometheus-stack Helm chart and accessing the YugabyteDB Grafana dashboard. The Prometheus Operator makes it very convenient to provision and maintain Prometheus on Kubernetes.
Published at DZone with permission of Bhavin Gandhi. See the original article here.
Opinions expressed by DZone contributors are their own.