DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workkloads.

Secure your stack and shape the future! Help dev teams across the globe navigate their software supply chain security challenges.

Releasing software shouldn't be stressful or risky. Learn how to leverage progressive delivery techniques to ensure safer deployments.

Avoid machine learning mistakes and boost model performance! Discover key ML patterns, anti-patterns, data strategies, and more.

Related

  • Optimize AWS Costs With CloudWatch's Advanced Metrics, Dashboards, and Alerts
  • Managed MQTT Broker Comparison — Console/Dashboard Features
  • Unified Observability Exporters: Metrics, Logs, and Tracing
  • 7 Tips for Using Instrumentation and Metrics To Align Site Reliability With Business Goals

Trending

  • Emerging Data Architectures: The Future of Data Management
  • Failure Handling Mechanisms in Microservices and Their Importance
  • *You* Can Shape Trend Reports: Join DZone's Software Supply Chain Security Research
  • Zero Trust for AWS NLBs: Why It Matters and How to Do It
  1. DZone
  2. Testing, Deployment, and Maintenance
  3. Monitoring and Observability
  4. Prometheus Monitoring With Grafana

Prometheus Monitoring With Grafana

How you construct your Prometheus monitoring dashboard involves trial and error. Grafana makes this exploration very easy and Prometheus has good built-in functionality.

By 
Daniel Berman user avatar
Daniel Berman
·
May. 24, 17 · Tutorial
Likes (4)
Comment
Save
Tweet
Share
35.6K Views

Join the DZone community and get the full member experience.

Join For Free

the combination of prometheus and grafana is becoming a more and more common monitoring stack used by devops teams for storing and visualizing time series data. prometheus acts as the storage backend and grafana as the interface for analysis and visualization.

prometheus collects metrics from monitored targets by scraping metrics from http endpoints on these targets. but what about monitoring prometheus itself?

like any server running processes on a host machine, there are specific metrics that need to be monitored such as used memory and storage as well as general ones reporting on the status of the service. conveniently, prometheus exposes a wide variety of metrics that can be easily monitored. by adding grafana as a visualization layer, we can easily set up a monitoring stack for our monitoring stack.

let’s take a closer look.

installing prometheus and grafana

for this article, i’m using a dockerized deployment of prometheus and grafana that sets up prometheus , grafana , cadvisor , nodeexporter , and alerting with alertmanager .

the default configuration used in this image already defines prometheus as a job, with a scraping interval of 15 seconds, so the server is set to monitor itself.

- job_name: 'prometheus'

   scrape_interval: 10s

   static_configs:

     - targets: ['localhost:9090']

to install this stack, use these commands:

git clone https://github.com/stefanprodan/dockprom
 
cd dockprom
 
docker-compose up -d

open up grafana at http://<serverip>:3000 and use admin and changeme as the password to access grafana.

defining the prometheus datasource

our next step is to define prometheus as the data source for your metrics. this is easily done by clicking creating your first datasource .

the configuration for adding prometheus in grafana is as follows:

add prometheus in grafana

once added, test and save the new data source.

what to monitor

now that we have prometheus and grafana set up, we can begin to set up monitoring.

prometheus ships a number of useful metrics that can be monitored. for example, since prometheus stores all chunks and series in memory, we can build a panel based on the prometheus_local_storage_memory_chunks and prometheus_local_storage_memory_series metrics.

prometheus memory chunks series

based on your ram, you will want to monitor these panels for any specific thresholds passed. you can fine-tune the amount of ram using the storage.local.memory-chunks configuration directive, while prometheus recommends that you have at least three times more ram available than needed by the memory chunks alone.

to get an overview of how well prometheus is performing, we can measure the ingestion rate for the samples using the prometheus_local_storage_ingested_samples_total metric. what you need to monitor is whether the rate displayed aligns with a number of metrics you know you are ingesting.

measure prometheus ingestion rate

likewise, it can be useful to monitor the actual amount of time between target scrapes that you have configured for prometheus using the prometheus_target_interval_length_seconds metric. this will help you to identify latency issues whenever a sudden rise in latency is something that you will want to explore.

prometheus rise in latency

another useful metric to query and visualize is the prometheus_local_storage_chunk_ops_total metric that reports the per-second rate of all storage chunk operations taking place in prometheus.

per-second rate of all storage chunk operations

using the prometheus stats dashboard

grafana comes with a built-in official dashboard for prometheus called prometheus stats that was developed together with the prometheus team. after you set up prometheus as the datasource, simply select the dashboards tab and import the listed dashboard.

prometheus stats dashboard

as you can see, this dashboard contains the following visualizations (some are similar to the ones described above):

  • uptime : the total amount of time since your prometheus server was started.
  • local storage memory series : the current number of series held in memory.
  • internal storage queue length : ideally, this queue length should be “empty” (0) or a low number.
  • sample ingested : displays the samples ingested by prometheus.
  • target scrapes : displays the frequency that the target — prometheus, in this case — is scraped.

a prometheus benchmark dashboard

while designed for benchmarking prometheus servers, the prometheus benchmark dashboard can be used to get a sense of the additional metrics that should be monitored.

to install and use this dashboard, simply go to dashboards > import and paste the url for the dashboard.

prometheus benchmark dashboard

while this is probably overkill for the day-to-day monitoring of your instance, this advanced dashboard includes some useful panels that are not displayed in the prometheus stats dashboard.

for example, the memory graph gives you an idea of the memory consumed by prometheus (same goes for the cpu graph).

memory graph

the indexing graph gives you a good picture of the indexing rates for your prometheus instance.

indexing graph

endnotes

you can slice and dice the various metrics that prometheus self-reports about itself any way you want. prometheus exports a long list of metrics that can be seen by browsing to http://<serverip>:9090/metrics . you can experiment with the various queries in the graph editor at http://<serverip>:9090/graph .

so, how you construct your prometheus monitoring dashboard will ultimately involve quite a lot of trial and error. the good news is that the combination with grafana makes this exploration extremely easy, and prometheus has some good built-in functionality to help you along the way.

Grafana Metric (unit) Dashboard (Mac OS)

Published at DZone with permission of Daniel Berman, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Optimize AWS Costs With CloudWatch's Advanced Metrics, Dashboards, and Alerts
  • Managed MQTT Broker Comparison — Console/Dashboard Features
  • Unified Observability Exporters: Metrics, Logs, and Tracing
  • 7 Tips for Using Instrumentation and Metrics To Align Site Reliability With Business Goals

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!