Over a million developers have joined DZone.

Docker + Elasticsearch: How to Monitor the Official Elasticsearch Image on Docker

Trying to monitor Elasticsearch and Docker? Here's how to monitor both, and ensure proper performance.

· Performance Zone

Download Forrester’s “Vendor Landscape, Application Performance Management” report that examines the evolving role of APM as a key driver of customer satisfaction and business success, brought to you in partnership with BMC.

The official Elasticsearch Image on Docker Hub has already generated more than 1.6 million pulls. It is probably the easiest way to get a development setup — which includes Elasticsearch — to the application stack. The reason for this crazy number? A rapidly growing number of organizations are using Elasticsearch and Docker in production. Needless to say, monitoring Elasticsearch is essential in production, and you can find a detailed analysis of this topic (including the “top 10 Elasticsearch metrics to watch”) in the free eBook: ElasticsearchMonitoring Essentials. Docker is disruptive in many ways, and there are many things that are slightly different and worth mentioning.  These include:

  1. Changed deployment for Elasticsearch and its monitoring tools using Dockerfile, Docker Compose or various Orchestration Tools
  2. There is a new Layer to monitor: Container Metrics and Events, see: Docker Events and Metrics monitoring andSPM for Docker
  3. Logging has changed: containers log to the console and logs needs to be retrieved from Docker-Daemon instead getting them from the Elasticsearch log file.  Check out our post on the subject: Innovative Docker Log Management
  4. Official Images may not provide options for monitoring (such as JMX).  However, the official Image for Elasticsearch provides an option to pass parameters to the Java Runtime Environment.  We we will use this option for Elasticsearch monitoring in this post. You should also be aware that the official Elasticsearch Image does not include any plugins, and commercial monitoring from Elastic can’t be distributed in this Image for licensing reasons.  Our monitoring tool of choice is SPM.  If you are not familiar with SPM — but have heard of it — or if you use Marvel, have a look at Marvel vs. SPM.

Next, I’m going to demonstrate a setup to monitor multiple Elasticsearch nodes on a single Docker Host. The final setup will provide the full Monitoring and Logging package:

  • Detailed Application Metrics for Elasticsearch, deployed on Docker
  • Detailed Container Metrics and Docker Events  
  • Centralized Logs for all Containers by SPM for Docker

So let’s first decide on one of the following options to monitor Elasticsearch on Docker.  You can:

  1. Build your own Elasticsearch container with the included monitoring components. I’m not going to go into details about this option today; rather, I’m going to focus on the official / trusted build.
  2. Use a standalone agent, which queries metrics from the Elasticsearch container. This requires a setup for JMX and Docker networking configurations for the monitor and Elasticsearch. The metrics, gathered by remote agents, are limited and, in the Docker context, running an external monitoring process plus Elasticsearch processes consumes more resources.  And the next option …
  3. Inject an SPM in-process monitoring agent into Elasticsearch. This option has the lowest resource usage and has support for advanced monitoring functions like Transaction Tracing and AppMap.

I chose to implement Option #3 in this blog post because it provides the best insights into Elasticsearch. This means the Elasticsearch container needs file-system access to the SPM monitoring agent. Sematext provides the SPM Client (which includes the monitoring agent and metrics sender) pre-installed in a Docker Image, referred as “SPM Client Image/Container” in the following instructions and published on Docker Hub as “sematext/spm-client”.  The main trick here is to mount a volume from SPM-Client Container into Elasticsearch Containers in order to load the monitoring library.

Let’s have a look at the desired setup and how to get there:


Monitoring Setup for Elasticsearch on Docker

1) Pull the Docker Image for SPM Client

Here is the SPM Client Image – the important thing to know about this image is that it provides a VOLUME (/opt/spm), which can be mounted  from the Elasticsearch Container. Download SPM-Client Image:

docker pull sematext/spm-client

2) Configure and start SPM-Client Container

The SPM Token is a unique identifier for monitored applications – if you haven’t yet created an SPM App for Elasticsearch, then create one here.

# Save the SPM Token in an Environment Variable
export SPM_TOKEN=4feb144c-4da8-xxxx-xxxx-b0b8e06e743a
# Choose name for the Elasticsearch nodes
export ES1=ES_1
export ES2=ES_2
# Create SPM Config String - let's configure 2 nodes in this example
export SPM_CONFIG="${SPM_TOKEN} es javaagent jvmname:${ES1}"
export SPM_CONFIG="${SPM_CONFIG};${SPM_TOKEN} es javaagent jvmname:${ES2}"
# Start SPM-Client with the defined configurations
docker run -d --name spm-client -h spm-client -e SPM_CONFIG="$SPM_CONFIG" sematext/spm-client

3) Start Elasticsearch Containers

We are ready to fire up multiple Elasticsearch nodes (in our example the nodes ES_1 and ES_2). In the command we see options for the SPM In-Process monitor to inject a .jar file from SPM Client Volume — the ES_JAVA_OPTS string is taken from SPM install instructions — using the SPM Token and naming the JVM (because we like to run N instances on the same host).

# start first node
export ES_JAVA_OPTS_1="-Dcom.sun.management.jmxremote -javaagent:/opt/spm/spm-monitor/lib/spm-monitor-es.jar=${SPM_TOKEN}::${ES1} -Des.node.name=${ES1}"
docker run --name $ES1 -h $ES1 -e ES_JAVA_OPTS -p 9200:9200 -d --volumes-from spm-client elasticsearch
# start second node
export ES_JAVA_OPTS_2="-Dcom.sun.management.jmxremote -javaagent:/opt/spm/spm-monitor/lib/spm-monitor-es.jar=${SPM_TOKEN}::${ES2} -Des.node.name=${ES2}"
docker run --name $ES2 -h $ES2 -e ES_JAVA_OPTS -d --volumes-from spm-client elasticsearch

Almost done! After a minute or so metrics for the Docker Host,  JVMs and Elasticsearch nodes will arrive to our SPM App.

Please note, because we chose a consistent naming for Elasticsearch Nodes (-Des.node.name), Container Hostname (-h) and JVM Name (-javaagent:/opt/spm/spm-monitor/lib/spm-monitor-es.jar=${SPM_TOKEN}::${ES1}) we can  immediately see in every chart the relevant filters named “ES_1/ES_2” – much better than some random Container IDs or automatically generated Elasticsearch Node-Names!

ES_DOCKER_OVERVIEWSPM overview for monitored Elasticsearch Containers

ES_DOCKER_INDEX_STATSIndex Stats for the Elasticsearch Nodes ES1 and ES2

But what about my Elasticsearch Logs and the Container Metrics?

Simply run SPM for Docker – it collects logs (and it can parse Elasticsearch logs) and stores them in Logsene (not used Logsene before?  See Logsene 1-Click ELK Stack). First we create the SPM App with the type “Docker” and wecreate a Logsene App to receive the logs. Then we use the generated App Tokens to run SPM for Docker.

docker run -d  --name spm-agent --restart=always -e SPM_TOKEN=SPM_DOCKER_APP_TOKEN -e LOGSENE_TOKEN=LOGSENE_APP_TOKEN

After a few minutes you’ll seeHost and Container Metrics together with Events and Logs in SPM.

ES_DOCKER_CONTAINER_METRICS_AND_LOGSContainer Logs and Metrics from Elasticsearch

Please note that logs from the containers are automatically shipped and parsed!
“Wait, there’s no setup for log shippers?” you might be saying.
That’s correct — there is NO complicated setup of syslog, logstash, docker log drivers, etc.  It all gets done bySPM for Docker

For example, each log line has a “node_name” field for the Elasticsearch node. It takes the timestamps, severity and source from the Elasticsearch log and each log is automatically tagged with the container_id and image name. Moving from SPM Metrics to detailed Elasticsearch Logs — including Exceptions and parsed Stack Traces — is just another mouse click!


The filters by the field stats on the right side of the screen make it easy to identify containers with the most logs, by choosing “container_name”.  That’s just a little detail in the Logsene UI, feel free to explore it and create Alerts or use Kibana4 Dashboards for your container logs.

If you’d like to know more about the meaning of all those Elasticsearch metrics check Top 10 Elasticsearch Metrics or the free eBook mentioned above. I believe Docker is a great platform to experiment with multiple nodes, even before you go to production.

Like what you’ve seen here? It’s super easy to get started right now with Docker and Elasticsearch monitoring (and logs!): just sign up for a free 30-day trial (no credit card required) of SPM and Logsene. Interested in monitoring other Apps on Docker with SPM?  Then please contact us by email or hit us on Twitter with questions and feedback. 

See Forrester’s Report, “Vendor Landscape, Application Performance Management” to identify the right vendor to help IT deliver better service at a lower cost, brought to you in partnership with BMC.


Published at DZone with permission of Stefan Thies, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

The best of DZone straight to your inbox.

Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}