O11y Guide: Bringing Monoliths Into the Cloud-Native World

This is the fifth article in the series covering my journey into the world of cloud-native observability. If you missed any of the previous articles, head on back to the introduction for a quick update.

After laying out the groundwork for this series in the initial article, I spent some time in the second article sharing who the observability players are. I also discussed the teams that these players are on in this world of cloud-native o11y. For the third article, I looked at the ongoing discussion around monitoring pillars versus phases. In the fourth article, I talked about keeping your options open with open-source standards.

Being a developer from my early days in IT, it's been very interesting to explore the complexities of cloud-native o11y. Monitoring applications goes way beyond just writing and deploying code, especially in the cloud-native world. One thing remains the same: maintaining your organization's architecture always requires both a vigilant outlook and an understanding of available open standards.

In this fifth article, I'm going to look at architectural choices you might encounter when older monolithic applications and monitoring tools are still part of an organization's infrastructure landscape.

Having previously covered the virtues of open standards and less official commonly adopted standards in the open source o11y world, it's now time to see how this same effect has been applied to monitoring older applications and services.

Not All Cloud Native

Not every cloud architecture can make the claim to run only cloud-native or Kubernetes workloads. Many still have older components, services, or applications that run as monoliths. These monoliths are not broken up into microservices, do not auto-scale in a container world, and might not even run in a container at all.

This does not preclude you from monitoring them in some fashion just as you would your cloud-native workloads. In the past, this might have been done using one of the Application Performance Management (APM) tools, but often requires multiple tools.

There are so many options used over the years, and contrary to popular belief, not all architectures have moved on from monitoring their monoliths. We'll always uncover dark corners of aging infrastructure that uses tooling we thought we'd never see again in the light of day.

Let's explore a few of the options you are more likely to encounter as examples of how we can initially merge these aging efforts into our cloud-native observability platform.

Some (Aging) Options

While you might find almost anything under the sun still running in the older corners of some infrastructure architectures, it's going to be tough to cover them all. In the following section, for example, I'm not going to talk about Nagios. It can be integrated in the same way as the options I'm going to cover below and I'm choosing to focus on examples that you're more likely to encounter first.

Now that we have that settled, let's explore some of the solutions you will encounter for monitoring more traditional monolithic solutions and infrastructure: Graphite, StatsD, and collectd.

Graphite

This is an open-source project first released in 2008 that provided a time-series metrics monitoring system. It was innovative in that it provided an original network-based approach to ingesting and handling metrics data from external systems without using the SNMP protocol.

The basic architecture is pushing metrics from the monitoring targets using a client. This client has to understand that the central server collecting the metrics isn't always able to receive and ingest the data sent. The server ingests and stores the time-series data allowing you to query it to create your monitoring dashboards. Metrics are pushed to the Graphite server in plain text or as a binary set of metrics using its own protocol. For a more in-depth look at the Graphite platform, there is an architecture and concepts overview here.

If you encounter this platform, there are options available from the open-source project Prometheus, where you can leverage the Graphite exporter. This gives you the chance to integrate your existing monolithic applications and infrastructure metrics into your cloud-native o11y efforts without an immediate migration effort.

StatsD

At one time, StatsD was the most popular monitoring solution for instrumenting code and thereby exposing custom application metrics. It's open source and allows you to expose the inside of your monolithic applications and is very closely tied with the concepts of Application Performance Monitoring (APM).

Developers instrument their code using client libraries for whatever language they are using. There is a central StatsD daemon running that pulls metrics, generates aggregated metrics, and relays them onward to a monitoring or graphing backend (a favorite being Graphite). It uses a fire-and-forget protocol for communication known as UDP, meaning low communication overhead. It was very advantageous to have the metrics daemon detached from the application and therefore not ever crashing or affecting application performance.

StatsD also has options available from the open-source project Prometheus when looking to integrate that existing monolithic infrastructure into your cloud-native o11y efforts. Integrating initially by using the StatsD exporter from the Prometheus ecosystem avoids the need for an immediate migration effort.

collectd

This is another metrics data collection solution with a plugin architecture of more than 100 options. Some common collectd plugins are CPU, memory, disk usage, aggregation, network traffic, NGnix, MySQL, etc. It is written in the C language for performance and portability which means it can run on systems without a scripting language or cron daemons, such as embedded devices. It needs to be installed on every machine it monitors as it uses a push mechanism to send its metrics to a backend system such as Graphite.

Integrating collectd into your modernization plans can also, you guessed it, be achieved using the collectd exporter. This again gives you the time to leverage existing monolithic infrastructure in your cloud-native o11y strategy, buying some time to plan for the eventual migration path still needed.

Open Path to Cloud-Native

What exactly do the three examples discussed above have in common when we are looking to integrate aging monitoring infrastructure platforms and tools into our cloud-native o11y strategy? They all have the ability to first be integrated by using open-source monitoring solutions for cloud-native environments.

Lucky for us there are many options to ensure that you can mitigate these old dependencies without ripping and replacing them. The open-source world gives us the ability to ingest older metrics platforms, APIs, and other components using a vast ecosystem of exporters. This gives you time for bringing your monoliths into the cloud native o11y world.

Next up, I plan to start exploring a new project called Perses. This is an up-and-coming open-source metrics dashboard project and will be where I start getting some practical experience in my cloud native o11y journey.