Monitoring Akka-Based Applications With Cinnamon and DataDog

Application monitoring is an important part of the software development life cycle and aims to check if the application is performing as expected or not. With applications becoming Reactive and having distributed architecture, and the industry moving towards Fast data, there is a great urge in the market for tools and integrations that could provide cutting edge monitoring for an application deployed in production.

In this article, we will explore how Lightbend Telemetry could be used to efficiently monitor the performance of Akka-based applications running on a single node. Telemetry makes it possible to gather information on metrics, events, and traces from Akka, Scala, and Lagom-based applications. I have chosen Akka-based applications for this blog as Akka provides an efficient and convenient way to implement a Reactive and Fast data application.
Lightbend Monitoring has two essential parts: telemetry and backend plugins.

1) Telemetry (Cinnamon)

By definition, telemetry means the process of recording and transmitting the readings of an instrument/system. Lightbend Telemetry (a.k.a. Cinnamon) collects information (metric, events, and traces), in runtime about an application based on the provided configuration. Cinnamon is a Java agent that runs in the same JVM as the application. Using Lightbend Telemetry is free during development, but one must have a valid license to use it in production.

2). Backend Plugins and Visualization Tools

Cinnamon collects metrics from the Akka-based Reactive application and reports them to the integrated backend. Various options among backend plugins are OpsClarity, DataDog, CodaHale Matrics, OverOps, etc. Visualization means graphically depicting the collected metrics and could be achieved via OpsClarity, DataDog, Grafana etc.

For this post, we will be using DataDog as our Backend/Visualization plugin. It provides a predefined dashboard to jump-start the monitoring of our Reactive applications. So, let's get to it! 

Step 1: Generate Bintray Credentials

Bintray credentials are required to get access to the required libraries for Cinnamon.

1). Login to your Lightbend account and generate the Bintray Credentials from here.

2). On your system, create a directory named .lightbend and in it create a file, commercial.credentials, with the bintray credentials you just generated. You may choose any directory structure for your credential file: ~/.lightbend/commercial.credentials is the Lightbend convention. The content of the file should be as follows:

realm = Bintray
host = dl.bintray.com
user = bintray_user_name
password = bintray_password

Step 2: Setting Up the Project to Use the Cinnamon Agent

I have cloned this simple Akka-Scala application from GitHub to demonstrate the functionality of telemetry. In your Akka-based application:

1). Update the sbt version to 0.13.7 or higher in the build.properties file and update your Scala version to 2.11.8 or higher (I have not tried this with a lower version of Scala than this) in build.sbt.

2). Create (if you haven't already) a plugins.sbt file in the project directory of your application and add the following code into it:

addSbtPlugin("com.lightbend.cinnamon" % "sbt-cinnamon" % "2.4.2") // Cinnamon plugin

credentials += Credentials(Path.userHome / ".lightbend" / "commercial.credentials") // Specify the path of the file created in step 1 above which contains bintray credentials

resolvers += Resolver.url("lightbend-commercial",
 url("https://repo.lightbend.com/commercial-releases"))(Resolver.ivyStylePatterns)

3). Lightbend Telemetry currently supports Akka 2.3.x/2.4.x/2.5.x. Set the Akka version of your project accordingly.

4). Add the following library dependency and configurations for Cinnamon in your build.sbt

libraryDependencies += Cinnamon.library.cinnamonAkka //Akka based application monitoring via Cinnamon

lazy val app = project in file(".") enablePlugins Cinnamon
cinnamon in run := true
cinnamon in test := true
cinnamonLogLevel := "INFO"

Step 3: Configuring Cinnamon Core and Selecting Actors for Telemetry

Since your application could create thousands or even tens of thousands of actors, the default setting is configured with actor telemetry turned off. You need to select which actors to observe in your application.conf file (present in src/main/resources) which is the same file used to set up your Akka system. Some metrics support configured thresholds. If the threshold is exceeded then an event is recorded.

With respect to my project, the following Actor configuration is defined in the application.conf file:

cinnamon.akka {
  actors {
   "com.example.*" { //All the actors present under com.example package would be monitored
     report-by = class
     thresholds { // Thresholds for metrics are defined here
       mailbox-size = 1000
       stash-size = 50
       mailbox-time = 3s
       processing-time = 500ms
     }
   }
 }
}

Refer to the Lightbend documentation here for possible ways to use Actor selection and settings.

Step 4: Backend Integration

I am using DataDog as my backend and visualization tool because it's hot in the market and provides a great predefined dashboard for an awesome visualization of the collected metrics. The following steps enable backend integration:

1). Create an account on DataDog. Since DataDog is a paid privilege, I have used their 14-day trail for this blog.

2). Install a DataDog agent on the host machine where your application will be running. The host machine could either be a cloud or your local machine, you just need to be connected to the internet. The DataDog agent could be installed from here based on your operating system.

3). Under the Integrations menu on the DataDog website, search for the Lightbend reactive platform and install the integration.

4). Add the following library dependency for DataDog

libraryDependencies += Cinnamon.library.cinnamonDataDog //For using DataDog as backend agent

5). Add the following configuration to the application.conf file of your project.

cinnamon.datadog {
  statsd {
    host = "127.0.0.1"
    port = 8125
    frequency = 60s
  }
  report {
    histogram = ["min", "max", "p98", "p99", "p999"]
  }
}

Here, statsd is the metric collector (DogStatsD for DataDog) and since the DataDog agent is running on the same machine on which the Akka application is running, the statsd host is 127.0.0.1 (localhost). 8125 is the default listening port for DogStatsD. Frequency is the time interval after which the DataDog agent sends the metrics to your DataDog account. We may change defaults by altering the /etc/dd-agent/datadog.conf file followed by DataDog agent restart.

All Set to Test Monitoring

Run your Akka application with some load, if possible, to test how the actors are performing. Now log in to your DataDog account and under the Dashboard menu select the Lightbend Akka Overview dashboard from Integration Dashboards in the right pane of the page.

My dashboard with two actors looks as follows:

Conclusion

In this blog, we discussed how application monitoring is important, especially for Reactive applications. We then saw how Cinnamon could be used to monitor Akka-based applications using DataDog as a backend as well as a visualization tool.

You may view a sample application with the latest sbt, Scala and Akka versions.

Hope this helps, Share your thoughts in the comments below.

Happy Monitoring!

References

1). https://app.datadoghq.com
2). http://developer.lightbend.com/docs/monitoring/latest/home.html
3). https://www.datadoghq.com/blog/monitor-lightbend/