Transport Cinnamon Matrices From Lagom to Prometheus

Monitoring is a pain when it comes to distributed applications, and even more when you have shared or non-shared variables to monitor in your application.

In this article, I'll explain two tools which can ease the monitoring efforts, one for generating metrics called Cinnamon and other to visualize them, called Prometheus.

Let's have a quick brief intro to these two:

• Prometheus - An open-source monitoring system with a dimensional data model, flexible query language, efficient time series database, and modern alerting approach.

• Cinnamon Metrics - (a.k.a Lightbend Telemetry) provides insight into applications built with Lightbend technologies. It does so by instrumenting frameworks and toolkits such as Akka. The instrumentation is done by a Java agent that runs when your application is starting up. It collects information, at runtime, about your application based on a configuration that you must provide. One can use Lightbend Telemetry for free during development, but you must have a valid license to use it in production.

In this tutorial, we will see how we can export our Cinnamon metrics into the monitoring tool Prometheus. Let's start with the configuration.

Configuration on the Lagom Side

/* Configuration for Cinnamon to ElasticSearch */
cinnamon {
  chmetrics {
    reporters += "elasticsearch-reporter"
    elasticsearch-reporter {
      hostString = "http://127.0.0.1:9200"
      index-date-format = "yyyy-MM-dd"
      basic-auth {
        username = "YourESUsername"
        password = "Password"
      }
    }
  }
}

Similarly, you will require Prometheus Exporter:

/* Configuration for Cinnamon to Prometheus */
cinnamon {
    prometheus {
    exporters += http-server
    http-server {
      host = "localhost"
      port = 9001
    }
  }
}

This configuration will start exporting your cinnamon metrics defined in the Lagom code at your defined host and port.

Now let's do the configuration at Prometheus level. On Prometheus, these metrics can also be checked at http://hostname:9090/graph, where the hostname is the IP address where Prometheus is installed.

Configuration on the Prometheus Side

We are using the property "file_sd_config" in Prometheus configuration which fetches the properties defined in the file as mentioned and ships them to the server.

- job_name: lagom
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  file_sd_configs:
  - files:
    - /prometheus/config/lagom.yml
    refresh_interval: 5m

This points to a config file named lagom.yml, which contains the path where Lagom is exporting the Cinnamon metrics. This is the best way to define the target for Prometheus where you use a file to define your URLs or other configurations.

Other ways are to hardcode the path, like so:

- job_name: kafka_exporter
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - localhost1:port1
    - localhost2:port2

In our case, Lagom is running on DCOS on top of conductR, so the address will be something like this:

cat /opt/prometheus/config/lagom.yml - targets: [ "lagom-impl-snapshot-8d8q1-s3.conductr-2.1.14.slave.mesos:9001" ] labels: environment: test4

It can vary by where and how you are deploying your app.

That's all for today. See you guys in the next blog, where we will see how one can dynamically fetch the metrics from a Lagom app running on DCOS.