How to Analyze Heroku’s New Runtime Metrics

Heroku Labs, Heroku’s platform for providing “experimental features that are under consideration for inclusion into the Heroku platform,” recently launched log-runtime-metrics for “enabling visibility into load and memory usage for running dynos.” With log-runtime metrics, Heroku now inserts the following per-dyno stats directly into the log stream:

• Memory use
• Swap use
• Load average

log-runtime-metrics is currently free to use and enabling the feature is as easy as running the following command: heroku labs:enable log-runtime-metrics.

Important! Don’t forget to restart your dynos after you’ve enabled log-runtime metrics (this had me stuck for a good 10 minutes before realizing I forgot to restart).

The load and memory usage metrics appear as system logs in the Logplex log stream while per-dyno metrics are generated every 20 seconds.

You should note that adding log-runtime-metrics can make the “dyno” key a bit confusing to use at first. Before adding log-runtime-metrics, searching the dyno keyword in a log management tool would have returned simple dyno names like web.1 to help you identify individual dynos. When enabling log-runtime-metrics, metrics use the key “source” to identify dynos by name in your dyno formation (e.g. web.1) while the key “dyno” is used to identify the app id and a UUID to uniquely identify every distinct dyno on the Heroku platform. The purpose behind this is to keep the source value persistent while allowing you to have many different dyno values (which becomes useful when deploying or restart dynos). As you can see below, this change can make grouping by the key “dyno” a little misleading. I only have one dyno running, but the variation in how the Dyno key is used can make it appear that I have 2 Dynos running if I’m not paying attention.

From here, Heroku offers a number of simple keys to search to calculate different metrics. A full description of each metrics can be found at Heroku’s log-runtime-metrics article, but here’s a quick reference list:

memory_total represents the sum of resident, cache, and swap memory being used by the dyno, measured in megabytes.

log-runtime-metrics provides averages of CPU load at 1, 5, and 15 minute scales. These averages reflect the number of CPU tasks that are queued up. The dyno manager calculates this average by taking the count of runnable tasks for each dyno every 20 seconds.

For a full explanation of Heroku Lab’s new log-runtime-metrics, visit their article here.

To start analyzing your logs, create a free Logentries account in less than 60 seconds here.