Monitoring Your Kubernetes Cluster the Right Way
To provide a more sustainable description for building a Kubernetes environment, I founded the Imixs-Cloud project.
Join the DZone community and get the full member experience.Join For Free
Monitoring a Kubernetes cluster seems not to be so difficult as you look at the hundreds of blogs and tutorials. But there is a problem – it is the dynamic and rapid development of Kubernetes. And so you will find many blog posts describing a setup that may not work properly for your environment anymore. This is not because the author has provided a bad tutorial, but only because the article is maybe older than one year. Many things have changed in Kubernetes and it is the area of metrics and monitoring that is affected often.
For example, you will find many articles describing how to set up the advisor service to get container metrics. But this technology has become part of kubelet in the meantime so an additional installation should not be necessary anymore and can lead to incorrect metrics in the worst case. Also, the many Grafana boards to display metrics have evolved. Older boards are usually no longer suitable to be used in a new Kubernetes environment.
Therefore in this tutorial, I would like to show how to set up monitoring correctly in the current version of Kubernetes 1.19.3. And of course, also this blog post will be outdated after some time. So be warned ;-)
To provide a more sustainable description for building a Kubernetes environment, I founded the Imixs-Cloud project. This project, published on Github, provides a concept for the installation, management, and also monitoring of a Kubernetes cluster. I invite you to participate in this project to ensure that it is up-to-date in the future. So let’s start…
The Metrics Server
To get the first insight into your Kubernetes cluster you need a source of container resource metrics like CPU, memory, disk, and network. These are also referred to as the “Core” metrics. These metrics are not provided out of the box if you have installed a Kubernetes node. The open-source project metrics-server provides a very easy to manage metric service collecting and aggregating the core metrics in your cluster in an efficient way. It is based on the Kubernetes Metric API and so it will be used by other Kubernetes add-ons, such as the Horizontal Pod Autoscaler or the Kubernetes Dashboard.
To install it you can run:
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/
It may be necessary that you customize the components.YAML file concerning the TLS support and the preferred address types. For that reason, in the Imixs-Cloud project, I provide a slightly adapted version here.
The metrics-server will start grabbing the node metrics. Until the first data is available it may take some seconds. You can check the metrics now with kubectl top:
$ kubectl top nodes NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% master-1 297m 14% 1358Mi 36% worker-1 1424m 35% 13913Mi 89%
Of course, the kubectl top is a very archaic way to get metric data. The K9s tool gives you more insights about your cluster and can output the data from the metric API. This includes also a simple text-based dashboard to visualize the core metrics of your cluster.
Prometheus and Grafana
The Core metrics collected by the metrics-server are stored in memory. To collect and monitor metrics over time an additional mechanism for aggregating this data is needed. As all the Kubernetes internal metrics are exposed using the Prometheus exposition format, a Prometheus service can be used to aggregate metrics not only from the metric-server but also from other components as also from individual business applications.
In the Imixs-Cloud project, I provide a deployment stack including all necessary services to monitor a Kubernetes cluster with Prometheus and Grafana. This deployment stack can be applied to other Kubernetes environments as well. The stack is based on kustomize which allows you to get better insights on how the stack is built and also a convenient way for customizing the stack for your individual needs. You can find the complete deployment setup on Github.
To get detailed insights into a Kubernetes cluster only two additional metric services are needed:
The Kube-state-metrics is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects. It is not focused on the health of the individual Kubernetes components, but rather on the health of the various objects inside, such as deployments, nodes, and pods. You can find a deployment example here.
The Prometheus node-exporter is another exporter service to provide hardware and OS metrics exposed by *NIX kernels, written in Go with pluggable metric collectors. This service is needed to get details about the nodes running in your cluster and is deployed as a DaemonSet on all nodes. You can find a deployment example here.
Scabbing the Metric Data With Prometheus
All the configuration details on how Prometheus scrapes the metric data from the different metric APIs are defined in the config file config/Prometheus.YAML. This file can be used to customizes the different jobs and of course, it can not only be used to monitor cluster metrics but also for metrics provided by any business application running in the cluster. The following job example shows how to define a new target from a business application
- job_name: 'kubernetes-pods' kubernetes_sd_configs: - role: pod relabel_configs: # select pod by application label - source_labels: [__meta_kubernetes_pod_label_app] separator: ; regex: (my-business-app) replacement: $1 action: keep - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: kubernetes_pod_name
The configuration is provided as a config map that needs to be generated before you can start the deployment. To create the config map run:
$ kubectl create configmap prometheus-config --from-file=./config
/prometheus.yml -n monitoring
The full deployment definition I am using here can be found on Github. A basic deployment can be run directly via kubectl and the base kustomize setup hosted on Github:
$ kubectl apply -k https://github.com/imixs/imixs-cloud/management
The deployment will start the additional metric services, a Prometheus server, and the Grafana service. The advantage of using kustomize in comparison to a helm chart is that you can easily see all the deployment details and you can customize and adapt it for your individual needs.
Finally, you can build your own Dashboards or you can just use the Dashboard provided by the Imixs-Cloud project. You can import the JSON file provided at Github.
I would like to invite you to participate in the Imixs-Cloud project and help to create a simple and clear monitoring solution for Kubernetes. If you have questions or have new ideas, just post them via Github.
Published at DZone with permission of Ralph Soika. See the original article here.
Opinions expressed by DZone contributors are their own.