Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Monitoring with Prometheus

DZone's Guide to

Monitoring with Prometheus

Learn how to set up a basic instance of Prometheus along with Grafana and the Node Exporter to monitor a simple Linux server.

· Performance Zone ·
Free Resource

Container Monitoring and Management eBook: Read about the new realities of containerization.

Monitoring applications & application servers is an important part of the today’s DevOps culture & process. You want to continuously monitor your applications and servers for application exceptions, server CPU & memory usage, or storage spikes. You also want to get some type of notification if CPU or memory usage goes up for a certain period of time or a service of your application stops responding so you can perform appropriate actions against those failures or exceptions.

With monitoring, there are a number tools out there such Amazon CloudWatch, Nagios, New Relic, Prometheus and others. Some of them are free and some are paid. In this post, we will take a look at the Prometheus monitoring, a popular free & open-source project for monitoring dozens of services.

What Exactly Is Prometheus and How It Is Different?

Prometheus is an open-source and one of the popular CNCF projects written in Golang. Some of its components are written in Ruby but most of the components are written in Go. This means you have a single binary executables. You download and run Prometheus with its components; it's that simple. Prometheus is fully Docker compatible, and a number of Prometheus components within Prometheus itself are available on the Docker Hub.

You will see how to spin up a minimal Prometheus server with a Node Exporter and a Grafana components in Docker containers to monitor a stand-alone Linux Ubuntu 16.04 server. Let’s see first that what are the main mandatory components in Prometheus from ground-up.

Prometheus Components

Promtheus Server

Prometheus has a main central component called Prometheus Server. As a monitoring service, Prometheus servers monitor a particular thing. That thing could be anything: it could be a an entire Linux server, a stand-alone Apache server, a single process, a database service or some other system unit that you want to monitor. In Prometheus terms, we call the main monitoring service the Prometheus Server and the things that Prometheus monitors are called Targets. So the Prometheus server monitors Targets. As said earlier, "Targets" can also refer to an array of things. It could be a single server or a targets for probing of endpoints over HTTP, HTTPS, DNS, TCP and ICMP (*Black-Box Exporter), or it could be a simple HTTP endpoint that an application exposes through which the Prometheus server gets the application's health status from.

Each unit of a target such as current CPU status, memory usage (in case of a Linux server Prometheus Target) or any other specific unit that you would like to monitor is called a metric. So Prometheus server collects metrics from targets (over HTTP), stores them locally or remotely and displays them back in the Prometheus server.

The Prometheus server scrapes targets at an interval that you define to collect metrics from specific targets and store them in a time-series database. You define the targets to be scraped and the time-interval for scraping metrics in the prometheus.yml configuration file.

prometheus-app-api

You get the metrics details by querying from the Prometheus’s time-series database where the Prometheus stores metrics and you use a query language called PromQL in the Prometheus server to query metrics about the targets. In other words, you ask the Prometheus server via PromQL to show us the status of a particular target at a given time.

Prometheus provides client-libraries in a number of languages that you can use to provide health-status of your application. But Prometheus is not only about application monitoring, you can use something called Exporters to monitor third-party systems (Such as a Linux Server, MySQL daemon etc). An Exporter is a piece of software that gets existing metrics from a third-party system and export them to the metric format that the Prometheus server can understand.

prometheus-app-exporters

A sample metric from a Prometheus server could be the current usage of free memory or file-system free via a Node Exporter in the Prometheus server.

It is important to know that metrics that you get from a third-party system are different from the Prometheus metrics. Because Prometheus uses a standard data-model with a key-value based metrics that might not match with the third-party system this is why you use exporters to convert them. To keep things simple, I won’t go into the details of each syntax of a Prometheus metrics and how they are different.


Visualization Layer with Grafana

You use Grafana, the visualization layer, as the 3rd component to visualize metrics stored in the Prometheus time-series database. Instead of writing PromQL queries directly into the Prometheus server, you use Grafana UI boards to query metrics from Prometheus server and visualize them in the Grafana Dashboard as a whole, as we will see it in action shortly.

Alert Management with Prometheus Alert Manager

Prometheus also has a Alert Management component called AlertManager for firing alerts via Email or Slack or other notification clients. You define the Alert Rules in a file called alert.rules through which the Prometheus server reads the alert configurations and fires alerts at appropriate times via the Alert Manager component. For example, if the Prometheus server finds the value of a metric greater than the threshold that you defined in the alert.rules file for more than 30 seconds, it will trigger the Alert Manager to fire an alert about the threshold and the metric. We will see how AlertManager works with Prometheus and how do we setup in the Prometheus stack in some later post.

The above 3 components are the basis of the entire Prometheus Monitoring system. You need the central Prometheus server, a target and a visualization layer. Let's see how to setup a minimal Prometheus stack with the above 3 components to monitor a simple Ubuntu 16.04 server using docker-compose.

Setting Up Prometheus in Docker Containers

Its time to setup Prometheus stack on your Linux Server. You will setup Prometheus server for collecting metrics, Grafana for visualization and Node Exporter for monitoring the Ubuntu 16.04 Linux host system. You will use docker-compose to spin up services for Prometheus, Grafana and Node Exporter but let’s first spin up a stand-alone Prometheus server service with the docker-compose file given below:


version: '3'

networks:
  monitor-net:
    driver: bridge

volumes:
    prometheus_data: {}

services:

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    volumes:
      - ./prometheus/:/etc/prometheus/
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--storage.tsdb.retention=200h'
      - '--web.enable-lifecycle'
    # restart: unless-stopped
    expose:
      - 9090
    ports:
      - "9090:9090"
    networks:
      - monitor-net


The Prometheus docker-compose service uses the official prom/prometheus image, savesthe Prometheus data (such as the Time-Series Data) into a named volume, and gets Prometheus configuration YAML file from the host file-system which we defined above in the command section of the compose file. The --storage.tsdb.retention=200h flag is important as this will clear the TSDB every 15 days which is good for saving disk space. And you finally expose the service ports to the host from the container so you can access them over the Internet.

This compose file will spin up the Prometheus Server with no Exporter or application. With the stand-alone Prometheus Server running inside the Docker container, you can monitor the Prometheus server itself because it also scrapes the Prometheus server as defined in the prometheus.yml file. For example, you can see how many alerts were failed since the Prometheus server is up and running and other type of metrics about the Prometheus. Now let’s add Node Exporter service in the above defined compose file.

Adding the Node Exporter

The next thing, you sould do is to spin-up the Node Exporter container and attach it to the Prometheus server as shown in the below YAML file:


version: '3'

networks:
  monitor-net:
    driver: bridge

volumes:
    prometheus_data: {}

services:

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    volumes:
      - ./prometheus/:/etc/prometheus/
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--storage.tsdb.retention=200h'
      - '--web.enable-lifecycle'
    # restart: unless-stopped
    expose:
      - 9090
    ports:
      - "9090:9090"
    networks:
      - monitor-net

  nodeexporter:
    image: prom/node-exporter:latest
    container_name: nodeexporter
    user: root
    privileged: true
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)'
    restart: unless-stopped
    expose:
      - 9100
    networks:
      - monitor-net


Here you added another docker-compose service called nodexporter. As mentioned eariler, an exporter is a piece of software that translates metrics from a third-party system to a metric format that Prometheus can understand. The Node Exporter exports hardware and OS metrics to the Prometheus server which gets and stores it in its time-series database. You mount the host system volumes and pass a couple of flags to the Node Exporter service to help discover information about the host system using procfs & sysfs mount points. procfsand sysfs are file-systems in Unix-like operating systems that shows information about processes and other system informations like storage in a hierarchical file-like structure and directories. This structure varies from distro to distro. You mount these host directories as volumes to the Node Exporter service so the Node Exporter can see the host’s system information and you pass some command-line flags with the value of the locations of these directories to the Node Exporter container.

If you run Prometheus server, you should now see or execute the PromQL queries with the node_ prefix in the Prometheus server. Running these PromQL queries in Prometheus will show you some info about the host system processes, storage and other metrics.

Adding Visualization with Grafana

Grafana is a data visualization & Monitoring tool with support for a number of databases including Prometheus TSDB. With Grafana,you can create some fancy graphical UI for metrics that you collect from Prometheus server as shown below:

grafana-dashboard

What happens here, though, is you write PromQL queries in Grafana Dashboard items instead of writing in the Prometheus server, Grafana pulls those metrics from the Prometheus server at an interval that you choose at the top-right corner of the Grafana Dashboard, and displays them graphically in its Dashboard. Grafana itself runs in the Docker container so add the Grafana service and our final docker-compose file looks like:


version: '3'

networks:
  monitor-net:
    driver: bridge

volumes:
    prometheus_data: {}
    grafana_data: {}

services:

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    volumes:
      - ./prometheus/:/etc/prometheus/
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--storage.tsdb.retention=200h'
      - '--web.enable-lifecycle'
    # restart: unless-stopped
    expose:
      - 9090
    ports:
      - "9090:9090"
    networks:
      - monitor-net

  nodeexporter:
    image: prom/node-exporter:latest
    container_name: nodeexporter
    user: root
    privileged: true
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)'
    restart: unless-stopped
    expose:
      - 9100
    networks:
      - monitor-net

  grafana:
   image: grafana/grafana:latest
   container_name: grafana
   volumes:
     - grafana_data:/var/lib/grafana
     - ./grafana/datasources:/etc/grafana/datasources
     - ./grafana/dashboards:/etc/grafana/dashboards
     - ./grafana/setup.sh:/setup.sh
   entrypoint: /setup.sh
   environment:
     - GF_SECURITY_ADMIN_USER=${ADMIN_USER:-admin}
     - GF_SECURITY_ADMIN_PASSWORD=${ADMIN_PASSWORD:-admin}
     - GF_USERS_ALLOW_SIGN_UP=false
   restart: unless-stopped
   expose:
     - 3000
   ports:
     - 3000:3000
   networks:
     - monitor-net


Copy the above docker-compose file contents and run docker-compose up -d to spin-up Promtheus server, Node Exporter and Grafana containers and run docker-compose ps to check the statuses of the containers. Here you are also mounting the host directory file-system to the Grafana service, so it can access the Dashboard’s JSON from the host system.

Now finally, go to your system’s public IP address with the port 3000 and enter admin:admin username and password to see your server stats in the Grafana Dashboard. You will see something like shown below:

grafana-dashboard



Conclusion

Here you saw a simplest example of how to setup a Prometheus server with Node Exporter and Grafana to visualize our Ubuntu 16.04 Linux server stats. You can store Prometheus time-series data to a 3rd party storage such as InfluxDB rather than the local file-system. It’s important to note that Prometheus is not about logging and tracing. We will see how to add Prometheus Alert Manager component to the above Prometheus stack to send Email or Slack alerts if something goes wrong and how to use service discovery with Prometheus.

You can find the complete source code for the post here. Read the Prometheus documentation to learn more about Prometheus components and its architecture.

Take the Chaos Out of Container Monitoring. View the webcast on-demand!

Topics:
docker ,docker-compose ,prometheus ,grafana ,monitoring ,linux ,ubuntu ,performance

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}