Sending Alerts From Strimzi to Slack
Sending alerts to external systems using Prometheus Rules is an integral feature of Strimzi. Learn to utilize Slack as a mechanism for automated alerts.
Join the DZone community and get the full member experience.Join For Free
In one of my other articles, we saw how to configure Prometheus and Grafana for Strimzi (Kafka on Kubernetes). In this article, we will find out how we can send alerts to an external system like Slack using Prometheus Rules.
I followed upstream Strimzi documentation. Alerts are very important for any enterprise, as these automated alerts notify System Administrators or Production Support teams of any issue.
As per Strimzi documentation:
When an alert expression becomes true, the condition is met and the Prometheus server sends alert data to the Alertmanager. Alertmanager then sends out a notification using the communication method configured for its deployment.
So let us start the integration.
The prerequisite for this article is to have a setup as described in this article.
Carefully go through prometheus-rules.yaml.
It is a sample having prometheus alert rules. It has rules for both Kafka and Zookeeper processes like:
Here we will modify one of the rules so that we have a proof of concept in our setup. I am going to modify the rule ScrapeProblem and set for: 1s, just for POC. As soon as I stop one of Kafka pods, I should get receive an alert like "Prometheus was unable to scrape metrics from....".
Apply this custom resource again and make sure changes are reflected.
Set up a workspace and channel in Slack for this POC. We will also get a webhook URL. I am testing on Slack with Free Plan.
Browse —> https://slack.com/get-started#/createnew —> Login (Being a POC, I logged in with my personnel Gmail account) —> Create a Workspace
- I entered text "alert-poc" in text-box as workspace name.
- For step 2 i.e. "What's your team working on right now", I entered "strimzi-poc". This is also the channel name.
- I skipped 3rd step.
- This will set up Slack Workspace and channel for us.
Get a webhook URL for the Slack workspace we just created so that Strimzi can be configured with it. With this Strimzi, we will start sending alerts.
- Browse —> https://api.slack.com/apps?new_app=1
- Enter "App Name" and Select "Development Slack Workspace". I entered "strimzi-alert-slack-app" as app name and "alert-poc" as "Development Slack Workspace".
- Basic Information —> "Add features and functionality" —> click on "Incoming Webhooks" —> enable it, it is off by default.
- Click on "Add New Webhook to Workspace" -> Select channel to post alerts -> I selected "strimzi-poc" channel as created in previous step.
- It will generate a webhook URL like: https://hooks.slack.com/services/RANDOM/PATH . I haven't copied the exact URL.
We will create a secret based on sample alert-manager-config.yaml. We have to modify slack_api_url with slack webhook URL and channel with #strimzi-poc.
We can either use the Slack app or browse it from the browser. Go to https://slack.com/get-started#/landing.
If we click on these Slack alerts, we will find that these are forwarded to Alertmanager listening on port 9093. In minikube, we would have to port-forward this 9093 port of Alertmanager pod and then we can check even more alert details.
That's it, guys! We finished the integration of AlertManager in Strimzi (also known as Kafka on Kubernetes) with Slack. I believe this article will provide more insights into Prometheus, Alerts, rules, and sending alerts to external systems like Slack.
Opinions expressed by DZone contributors are their own.