DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations
11 Monitoring and Observability Tools for 2023
Learn more
  1. DZone
  2. Testing, Deployment, and Maintenance
  3. Monitoring and Observability
  4. Grafana Alerting and Email Notifications

Grafana Alerting and Email Notifications

In this article, we are going to learn how to create alerts in Grafana and send these alerts over email notifications. Let's get to it!

Satish Sharma user avatar by
Satish Sharma
·
Jun. 01, 18 · Tutorial
Like (7)
Save
Tweet
Share
134.92K Views

Join the DZone community and get the full member experience.

Join For Free

Use Case: We shall be monitoring active connections on a MySQL database server and when the number of active connections crosses the threshold value then an alert shall be displayed on the dashboard and an email will be triggered on the configured email id with the snapshot of our graph indicating breach value and other information.

Environment:

  1. Grafana: grafana-5.1.0
  2. fakeSMTP-2.0: SMTP server (works with any SMTP server)
  3. Operating System: Windows (for others steps may vary)

Assumptions:

I assume that you already have setup and integrated Prometheus with Grafana. If not, then you can follow this article. I also assume that you have set up the monitoring for the MySQL server database. If not, then you can follow this article.

For demonstration purposes, I am using a Fake SMTP server. It can work with any valid SMTP server. If you have your dashboard ready, then let us get started by configuring our SMTP server.

So let us get started.

Setup SMTP in Grafana

  • Go to "conf" directory of your Grafana distribution
  • Open your configuration file (as we did the setup using defaults so I am using "defaults.ini"). Navigate to SMTP/Emailing settings and update your SMTP details. As we have our fake-SMTP server running on localhost and on port 25. My "defaults.ini" has below configurations.
#################################### SMTP / Emailing #####################
[smtp]
enabled = true
host = localhost:25
user =
# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
password =
cert_file =
key_file =
skip_verify = false
from_address = admin@grafana.localhost
from_name = Grafana
ehlo_identity =

Setup New Notification Channel

  • Login to Grafana with username and password.
  • Click on the "Bell" icon on the left sidebar.
  • Select "Notification channels."Image title
  • Click on "Add Channel." This will open a form for adding new notification channel.
  • Give a name to this channel. I am using "MySQL email channel."
  • Select Email from "Type" as we want to send notifications over email.
  • Check the "Send on all alerts" in case you want email on all alerts.
  • Select the checkbox of "Include image" in case you want to include the image of the panel as the body in the notification email.
  • Add the target email in "Email addresses" text area. You can use multiple email address separated by ";".

Image title

  • Click on "Send Test" if you want to verify your settings. This will send a sample email using the SMTP details we configured earlier. Click on "Save" to add this channel. Below is the screenshot of the test email sent by Grafana.

Image titleCreate Alert

  • Select the panel where you want to create an alert. And click on "Edit." I am using the "connections" panel we added in our dashboard to monitor active connections.
  • Click on the "Alert" tab and click on "Create Alert." This will open the form for configuring alert.
  • Select the Alert Config which is selected by default.

Name: Give a suitable name to this alert.

Evaluate Every: The time interval on which you want this rule to be evaluated, where "h" means hours, "m" means minutes, and "s" means seconds.

I am using 5s (5 seconds). In production, you might want to set this to higher values.

  • Click on the function name in the "Conditions" section and change it to the desired function. I am using "Count()" as we want to validate our rule against the count of active connections.
    In function "query (A, 5m, now)".
    • The first argument (A) is the query id to be used form all the queries which we have configured in "Metrics" tab. As we have the connection count query in section "A".Image title
    • The second and third argument is used to form the time range. Our query means 5 minutes back from now.
  • You can validate your rule by clicking on 'Test Rule.' If the rule has any errors, then it will display the error. My final setup is as follows. Image title
  • Select the "Notifications" section. This will have our earlier configured notification channel in "Send to."
  • You can use multiple channels by clicking on the "+" icon and choose from the list displayed. Add the message you want to be included in email in "Message" section.

Save your configuration by pressing "CTRL+S" provide the comments if asked for. My final configurations are as below.Image title

Add Alert List (Optional)

  • Navigate to your MySQL dashboard we created.
  • Click on the graph icon on the top bar and select "Alert List."
  • This will add a panel displaying list and status of all the configured alerts. Image title

This is all the settings/steps we need to create the alert and email notification in Grafana. You shall be able to see a dotted line on the graph whenever the rule is fired also you will receive an email on the SMTP settings you configured. You can see the spike in the graph where the threshold value is breached and the Alert was fired.Image title

And here is the email I received on the fakeSMTP-2 server with the snapshot of the graph and breaching values.

Image title

Below is the screenshot of my final dashboard with all alerts and notifications configured. You can get the JSON for this dashboard from this GitHub repository and can directly import this dashboard for your Grafana + Prometheus + mysqld_exporter setup.

Image title

Grafana

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • JWT Authentication and Authorization: A Detailed Introduction
  • A Beginner's Guide to Infrastructure as Code
  • 5 Software Developer Competencies: How To Recognize a Good Programmer
  • 3 Main Pillars in ReactJS

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: