DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Join us today at 1 PM EST: "3-Step Approach to Comprehensive Runtime Application Security"
Save your seat

How to Cook Delicious Monitoring

Take a look at how you can whip up a great serving of monitoring with two essential ingredients.

Sergii Bieliaievskyi user avatar by
Sergii Bieliaievskyi
·
Jan. 03, 19 · Tutorial
Like (2)
Save
Tweet
Share
6.60K Views

Join the DZone community and get the full member experience.

Join For Free

Monitoring is something that was important before DevOps era, it is important now, and will be important long after this era is over. According to a book written by Gene Kim, The DevOps Handbook, DevOps can be described as a set of three underpinning principles. The first principle is the flow of work, from left to right, from developers to operations. The second principle is the flow of feedback, from right to left, from operations to developers. The last one is the principle of continuous learning and improvement. Does this remind you of something? When I look at these three principles, I see the blood circulatory system. Indeed, arteries deliver oxygen, just like the first principle delivers code. Veins expel carbon dioxide like the second principle exposes problems in our system. And finally, the third principle, the principle of continuous learning and improvement. Based on the feedback provided by previous two principles, we work to eliminate imperfection, like white blood cells that fight against viruses.

Just like our body, the monitoring system requires a strong immune system. But what makes the monitoring strong? Intelligence and simplicity. What do we have to do to give birth to the strong monitoring? We have to mix the right components. One example of the right components, in my opinion, are Prometheus and Consul.

Prometheus is a system which uses pull strategy. This is how the authors of Prometheus describe its advantages:

Why do you pull rather than push?

Pulling over HTTP offers a number of advantages:

  • You can run your monitoring on your laptop when developing changes.
  • You can more easily tell if a target is down.
  • You can manually go to a target and inspect its health with a web browser.

Indeed, a first comparison shows a significant difference from the systems with a push strategy. However, you can hardly "cook" delicious monitoring, leveraging only the advantages listed above. Here is a more interesting addition. Prometheus scrapes metrics from the exporters, in turn, the exporters give all available information, regardless of what you want or expect to see. At some point in time, you realize that some important metric is missing on your dashboard, and you desperately need it to correlate with another metric. Prometheus has it, and it has a full history for this metric.

It is cool, because nobody wants to overload dashboards with tons of charts, and at the same time you will be happy to find complete history which Prometheus collects for you in the background. The list of the advantages is not complete without ability of Prometheus to grab the newly registered exporters from systems know as Service Discovery (SD). Consul is a great SD with wide functionality, but for now, we need it to be service discovery only. Let’s see how it works. The following piece of prometheus.yml uses consul for searching the exporter by the group name and by the tag, probably are two most useful techniques.Image title

Using high-level abstraction we can imagine the exporter, as a metrics group of a specific type. For example, the node exporter is operating system metrics, such as memory, disk, CPU and so on.
(1) The simplest case. Along with information on how to reach consul, the section instructs Prometheus to create targets for all nodes in node_exporter group without exception ( 1.1 ).
(1.1) List of groups.
(2) What is this for? Everything is spinning around labels in Prometheus, they are widely used in queries. At first glance it may look a bit cumbersome until we take the section apart, line by line.
Prometheus uses meta tags as a buffer storage. We work with consul, so the storage contains consul tags: - source_labels: [__meta_consul_tags]

Image title

regex: ',(?:[^,]+,){1}([^=]+)=([^,]+),.*'

The pattern breaks the line into the groups. Unfortunately, the process is semi-automated, and to find the target group we manually shift pointer to the group before the target. There are three groups in this example.

Group 1: env=test,
Group 2: svc
Group 3: system_metrics

Zero-based numbering is used here, hence the second element has the index of 1 or {1}. Although the pattern works, our obligation is to plan Prometheus config carefully, because the number of labels in __meta_consul_tags should not exceed the number of patterns ( - source_labels in the section (2) ).

Everything is ready to create the Prometheus label.

target_label: '${1}' -> Group 2: svc
replacement: '${2}' -> Group 3: system_metrics

The result you can see on the screenshot.

Image title


What is interesting in the section (3) is that action: keep , drops targets for which RegEx does not match. In other words, it searches for the previously marked services.

Conclusion

In the beginning, I mentioned that intelligence and simplicity are characteristics of the good monitoring system. Is it intelligent? Yes. It knows where to search for the exporters, and how to identify them. Is it simple? Yes. It is not simpler than any other flexible multifunctional tool. This is the price we pay for flexibility.

Now it is really delicious. Enjoy your meal!

operating system

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Do Not Forget About Testing!
  • How To Create and Edit Excel XLSX Documents in Java
  • Using the PostgreSQL Pager With MariaDB Xpand
  • AIOps Being Powered by Robotic Data Automation

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: