How to Cook Delicious Monitoring

Monitoring is something that was important before DevOps era, it is important now, and will be important long after this era is over. According to a book written by Gene Kim, The DevOps Handbook, DevOps can be described as a set of three underpinning principles. The first principle is the flow of work, from left to right, from developers to operations. The second principle is the flow of feedback, from right to left, from operations to developers. The last one is the principle of continuous learning and improvement. Does this remind you of something? When I look at these three principles, I see the blood circulatory system. Indeed, arteries deliver oxygen, just like the first principle delivers code. Veins expel carbon dioxide like the second principle exposes problems in our system. And finally, the third principle, the principle of continuous learning and improvement. Based on the feedback provided by previous two principles, we work to eliminate imperfection, like white blood cells that fight against viruses.

Just like our body, the monitoring system requires a strong immune system. But what makes the monitoring strong? Intelligence and simplicity. What do we have to do to give birth to the strong monitoring? We have to mix the right components. One example of the right components, in my opinion, are Prometheus and Consul.

Prometheus is a system which uses pull strategy. This is how the authors of Prometheus describe its advantages:

Why do you pull rather than push?

Pulling over HTTP offers a number of advantages:

• You can run your monitoring on your laptop when developing changes.
• You can more easily tell if a target is down.
• You can manually go to a target and inspect its health with a web browser.

Indeed, a first comparison shows a significant difference from the systems with a push strategy. However, you can hardly "cook" delicious monitoring, leveraging only the advantages listed above. Here is a more interesting addition. Prometheus scrapes metrics from the exporters, in turn, the exporters give all available information, regardless of what you want or expect to see. At some point in time, you realize that some important metric is missing on your dashboard, and you desperately need it to correlate with another metric. Prometheus has it, and it has a full history for this metric.

It is cool, because nobody wants to overload dashboards with tons of charts, and at the same time you will be happy to find complete history which Prometheus collects for you in the background. The list of the advantages is not complete without ability of Prometheus to grab the newly registered exporters from systems know as Service Discovery (SD). Consul is a great SD with wide functionality, but for now, we need it to be service discovery only. Let’s see how it works. The following piece of prometheus.yml uses consul for searching the exporter by the group name and by the tag, probably are two most useful techniques.

Using high-level abstraction we can imagine the exporter, as a metrics group of a specific type. For example, the node exporter is operating system metrics, such as memory, disk, CPU and so on.
(1) The simplest case. Along with information on how to reach consul, the section instructs Prometheus to create targets for all nodes in node_exporter group without exception ( 1.1 ).
(1.1) List of groups.
(2) What is this for? Everything is spinning around labels in Prometheus, they are widely used in queries. At first glance it may look a bit cumbersome until we take the section apart, line by line.
Prometheus uses meta tags as a buffer storage. We work with consul, so the storage contains consul tags: - source_labels: [__meta_consul_tags]

The pattern breaks the line into the groups. Unfortunately, the process is semi-automated, and to find the target group we manually shift pointer to the group before the target. There are three groups in this example.

Group 1: env=test,
Group 2: svc
Group 3: system_metrics

Zero-based numbering is used here, hence the second element has the index of 1 or {1}. Although the pattern works, our obligation is to plan Prometheus config carefully, because the number of labels in __meta_consul_tags should not exceed the number of patterns ( - source_labels in the section (2) ).

Everything is ready to create the Prometheus label.

target_label: '${1}' -> Group 2: svc
replacement: '${2}' -> Group 3: system_metrics

The result you can see on the screenshot.

What is interesting in the section (3) is that action: keep , drops targets for which RegEx does not match. In other words, it searches for the previously marked services.

Conclusion

In the beginning, I mentioned that intelligence and simplicity are characteristics of the good monitoring system. Is it intelligent? Yes. It knows where to search for the exporters, and how to identify them. Is it simple? Yes. It is not simpler than any other flexible multifunctional tool. This is the price we pay for flexibility.

Now it is really delicious. Enjoy your meal!