Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Event Monitoring Visibility Made Easy: Sensu Check Metric Extraction With InfluxDB and Grafana

DZone 's Guide to

Event Monitoring Visibility Made Easy: Sensu Check Metric Extraction With InfluxDB and Grafana

Check it out!

· Performance Zone ·
Free Resource

Good luck monitoring an event like this...

Good luck monitoring an event like this...


Monitoring dynamic, distributed environments often results in costly monitoring solution choices for developers seeking scalability, but your monitoring solution doesn’t have to be expensive. Sensu, InfluxDB, and Grafana — open source tools with enterprise counterparts — can work together to create a complete, scalable monitoring solution encompassing event monitoring, data ingestion and dashboarding.

Monitoring solution - InfluxDB, Sensu, Grafana

Figure 1 – Collect, record and visualize your metrics with open source tools.


You may also like: Why You Want Easy-to-Setup Grafana Dashboards

What Is InfluxDB?

InfluxDB is the essential time-series toolkit — an open-source time-series database designed to handle high write and query loads. It can store metrics, events, logs, traces from systems, sensors, queues, databases, and networks; and can ingest millions of data points per second.

There are over 260,000 InfluxDB instances in use around the world, covering a number of use cases (DevOps monitoring, IoT and sensor monitoring, real-time analytics, network monitoring, and machine learning) across several industries. Since it is an open-source project, InfluxDB is the place to start if you need a time-series database.

Unlike other types of databases adapted for time series, InfluxDB was purpose-built for handling time-stamped data making it easy to accumulate, analyze and act on the metrics your systems are instrumented for.

What Is Sensu?

Sensu is a monitoring event pipeline that collects, processes and routes different event types including discovery, availability, telemetry, and alerts. The pipeline makes Sensu extremely powerful and completely customizable.

Sensu monitoring pipeline

Figure 2 – Sensu: a monitoring event pipeline.


Built to address the needs of a modern cloud computing environment, Sensu is popular for monitoring applications, services, and infrastructure. The Sensu framework consists of client and server applications that communicate via a message bus.

What differentiates Sensu in its space is its focus around pipelining. You can collect via many different ways, extend beyond that, and then filter, mutate and handling metrics. The Sensu backend has an API. So if you can transform your event format into the Sensu event format, Sensu can ingest it.

What Is Grafana?

Grafana is an open-source metric analytics and visualization suite. It is most commonly used for visualizing time series data for infrastructure and application analytics but may be used in other domains including industrial sensors, home automation, weather, and process control.

How to Use InfluxDB and Sensu for Metric Extraction

The free and open-source version of the Sensu monitoring event pipeline is Sensu Go: it is a revamp of the original Ruby project, with some improvements surrounding metrics. Sensu offers multiple mechanisms to monitor performance metrics.

  • StatsD is a metric aggregator used to collect values, such as gauges, counters, timers, and sets. Each Sensu agent has an embedded StatsD daemon that’ll listen for that UDP traffic.
  • Sensu service checks are commands executed by the Sensu client that monitor a condition or collect measurements. The service checks collect data on monitored nodes and follow the same protocol as Nagios service checks.
  • Each Sensu agent will run that collection of checks. Then the checks will output that data, produce an exit code, and indicate a specific state.
  • Sensu will parse the output of that STDOUT (standard output) and produce metrics; hence, metric extraction.
  • The more complex your stack gets, the more likely you’ll need to set up both application performance monitoring (APM) and network and infrastructure monitoring.

Application performance monitoring and network infrastructure monitoring

Figure 3 – Among the types of monitoring that Sensu can handle.


Sensu currently supports four different output metric formats: InfluxDB, OpenTSDB, Graphite, and Nagios.

The metric type (each database name mentioned above) is the identifier that defines which format the check output should be parsed and mapped to the corresponding field output metric format.

Sensu Check Configuration

For ingesting metrics into InfluxDB, we define a check called check-cpu-influxdb. It’ll be set to run every 10 seconds on any node that you want it subscribed to.

Sensu check configuration - InfluxDB

Figure 4 – Sensu check configuration and event data.


In the screenshot above:

  • The command at the bottom is a simple shell script that will print out CPU usage in the InfluxDB line protocol.
  • The last two fields on the left of the screen indicate that check output metric extraction will occur.
  • The event produced contains not only the execution context (such as status, output, duration) but also entity information about your monitored node, and most importantly, the extracted metric.

Now that we have a collection of metric points within a Sensu event, we can send it to InfluxDB.

Sensu InfluxDB integration snapshot

Figure 5 – How Sensu integrates with the InfluxDB time-series database.


The handler configuration on the right of the screen takes the event data and invokes the Go binary Sensu InfluxDB Handler — a Sensu Event Handler that sends metrics to InfluxDB. Here’s how the handler works with Sensu:

  • Sensu can collect metrics using check output metric extraction or the StatsD listener.
  • The collected metrics pass through the event pipeline, allowing Sensu to deliver the metrics to the configured metric event handlers.
  • The InfluxDB handler allows storing, instrumenting and visualizing the metric data from Sensu.
  • The handler accepts configuration options as either command-line flags or environment variables.
  • Then, additional metric tag enrichment can happen as part of the Sensu event pipeline.

Sensu monitoring event pipeline

Figure 6 – Overview of the monitoring event pipeline.


The monitoring event pipeline dataflow, as shown above, is as follows:

  • The Sensu backend will send service checks to monitor nodes with installed Sensu agents.
  • The agents will execute the check and extract the metrics in any of the four supported formats.
  • The backend will receive that event data and pass it through the monitoring event pipeline.
  • In this specific use case, you can filter this event, only if it contains metrics, mutate that event to enrich any metric tags and add additional context about the data and source of the metrics.
  • Then you would handle the events by sending them off to a time-series database.

Any telemetry event that the agent receives will be processed by the backend. This is important because, in order to have complete visibility of your app, system, services, and infrastructure, you’ll likely have to receive data from multiple sources. As you add different event types (like availability and alerts), you’ll be thankful that the pipeline is dynamic enough to support reusability all under the same hood.

Sensu dashboard example

Figure 7 – A Sensu dashboard which prioritizes critical events over normal statuses


While the Sensu dashboard does provide excellent visibility into the overall health and state of your system, it doesn’t directly visualize time series data. Enter Grafana time-series analytics and visualization, which provides a critical and convenient overview of your infrastructure since the “all under one roof” methodology can simplify your workflow and telemetry.

Grafana dashboard

Figure 8 – Grafana dashboard displaying time-series analytics


In the above dashboard, there’s a single data source as far as Grafana is concerned because we let Sensu do all of the heavy liftings. The Sensu checks shown here display metrics from both Graphite and InfluxDB, while the StatsD daemon tracks all the API calls and request rates.

As for how development teams can use data stored from metric collection checks in InfluxDB, see our previous blog post-Sensu and InfluxDB: Storing Data from Metrics Collection Checks — where we also provide setup tutorial and introduce the integrations provided by the InfluxDB Sensu Plugin.


Further Reading

Workflow Automation

What Is the Difference Between Metrics and Events?

Topics:
event monitoring ,sensu ,influxdb ,grafana ,performance ,tutorial

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}