Centralized Logging for Kafka on Kubernetes With Grafana, Loki, and Promtail.
If there are multiple zookeeper and Kafka pods, a single window would be a boon for administrators. Grafana provides Loki and Promtail this functionality.
Join the DZone community and get the full member experience.Join For Free
In one of my another articles, I discussed how to set up strimzi (also known as Kafka on Kubernetes) on minikube. Also, we discussed how to set up Grafana and Prometheus to fetch metrics from Kafka and zookeeper instances. But wouldn't it have been more helpful and more administrator-friendly if Grafana could also be used to monitor logs of all the pods? If there are multiple zookeeper and Kafka pods, a single window would certainly be a boon for administrators and management.
Grafana provides Loki and Promtail a functionality to aggregate logs and view them from the same Grafana UI.
I have tested this in Fedora 33, minikube version: v1.15.1.
- helm: Download Linux amd64 distribution. Also, set it in the system path. I have downloaded the latest one. At the time of writing this article, the latest version is Helm 3.5.2.
- Grafana Helm charts
Install Loki and Promtail via helm-charts. To do this, we have to add the helm repository with these charts. Finally, we will find the Loki and Promtail pods that are created. Also, a daemonset promtail is created. We also see
service/loki listening on port 3100.
Check the logs of the Promtail pod and check that it is connected to the Loki pod with
service/loki on port 3100.
Here we find that Promtail is trying to connect to
loki-gateway. But it should connect to service Loki on port 3100.
To configure the correct url so that the Promtail agent connects to Loki correctly, we will edit the daemonset. We will have to add the argument
Let us connect Grafana with Loki. I have downloaded Grafana from the Grafana Download Page. But before that, we have to port-forward the Loki port.
Before we go to Grafana configurations, I want to highlight a few important concepts here:
- If we check the details of the daemonset promtail, we will find that a path of nodes are mounted. An interesting one is
/var/log/podswhich has all pods logged of that node. Promtail keeps seeking these logs and sends these to the Loki instance.
- In a production environment, you might need Promtail pods only in a worker node, but not in infra nodes or a master node. This can be achieved with nodeAffinity:
- If we want a more customized Promtail agent setup, then we might need to modify the promtail.yaml configuration which is mounted as secret. The secret's content is base64 encoded which can easily be decoded. Once decoded, we can get the configurations and customize them further. We can even use the following commands:
Follow the screenshots to configure Grafana to connect to Loki instances and monitor logs.
That's it, guys. I hope you find this article interesting and informative.
Opinions expressed by DZone contributors are their own.