Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Open Source Docker Monitoring & Logging

DZone's Guide to

Open Source Docker Monitoring & Logging

Check out this handy comparison of open source tools for logging and monitoring Docker containers. Be sure to click on the MindMap at the bottom for a handy illustration to help with grouping complimentary tools together.

· DevOps Zone
Free Resource

Learn more about how CareerBuilder was able to resolve customer issues 5x faster by using Scalyr, the fastest log management tool on the market. 

Pets ⇒ Cattle ⇒ Orchestration

Docker is growing by leaps and bounds, and along with it, its ecosystem. Being light, the predominant container deployment involves running just a single app or service inside each container. Most software products and services are made up of at least several such apps/services. We all want all our apps/services to be highly available and fault tolerant. Thus, Docker containers in an organization quickly start popping up like mushrooms after the rain. They multiply faster than rabbits.While, in the beginning, we play with them like cute little pets, as their numbers quickly grow we realize we are dealing with a herd of cattle, implying we’ve become cowboys. Managing a herd with your two hands, a horse, and a lasso will only get you so far. You won’t be able to ride after each and every calf that wonders in the wrong direction. To get back to containers from this zoological analogy—operating so many moving pieces at scale is impossible without orchestration—this is why we’ve seen the rise of Docker Swarm, Kubernetes, Mesos, CoreOS, RancherOS, and so on.

Pets ⇒ Cattle ⇒ Orchestration + Operational Insights

Container orchestration helps you manage your containers, their placement, their resources, and their whole life cycle. While containers and applications in them are running, in addition to the whole life cycle management, we need container monitoring and log management so we can troubleshoot performance or stability issues, debug or tune applications, and so on. Just like with orchestration, there are a number of open-source container monitoring and logging tools. It’s great to have choices, but having lots of them means you need to evaluate and compare them to pick the one that best matches your needs.

DevOps Tools Comparison

We’ve open-sourced our Sematext Docker Agent (SDA for short) which works with SPM for monitoring and Logsene for log management (think of it as ELK as a Service), and wanted to provide a high level comparison of SDA and several popular Docker monitoring and logging tools, like CAdvisor, Logspout, and others.  In the following table, we group tools by functionality and include monitoring agents, log collectors and shippers, storage backends, and tools that provide the UI and visualizations. For each functionality, we list in the “Common Tools” column one or more popular open-source tools that provide that functionality.  An empty “Common Tools” cell means there are no popular open-source tools that provide it, or at least we are not aware of it—if we messed something up, please leave a comment or tweet @sematext.

Functionality Common Tools Sematext Tools
Collect Logs from Docker API
(including auto-discovery of new containers)
Logspout Sematext Docker Agent
Log routing Logspout
Routing setup for containers via HTTP API to syslog, redis, kafka, logstash 
Docker Logging Drivers (e.g. syslog, journald, fluentd, etc.)
Sematext Docker Agent
(routing of logs to different indices based on container labels)
Automatic log tagging
(with Docker Compose or Swarm or Kubernetes metadata)
For Kubernetes: fluentd-elasticsearch, assumes Elasticsearch deployed locally Sematext Docker Agent
Collect Docker Metrics CAdvisor Sematext Docker Agent
Collect Docker Events ? Sematext Docker Agent
Logs format detection
(most tools need a static setup per logfile/application)
? Sematext Docker Agent
(out of the box format detection and parsing; the parser and the logagent-js pattern library is open source)
Logs parsing and shipping Fluentd
Logstash
rsyslog
syslog-ng
Sematext Docker Agent
Logs storage and indexing Elasticsearch
Solr
Logsene
(exposes Elasticsearch API)
Logs anomaly detection and alerting ? Logsene
Log search and analytics Kibana
Grafana
Logsene
(Logsene’s own UI or integrated Kibana, or Grafana connected to Logsene via Elasticsearch data source)
Metrics storage and aggregation Graphite
OpenTSDB
KairosDB
Elasticsearch
Influxdb
Prometheus
SPM
Metrics charts and dashboards Grafana
Kibana
SPM
Metrics anomaly detection and alerting Influxdb
Prometheus
SPM
Correlation of Metrics, Logs and Events ?

SPM & Logsene integration

This table shows a few things:

  • Some of the functionality provided by SPM and Logsene is not available in some of the most popular open-source monitoring and logging tools included here.
  • Some of the SPM and Logsene functionality is indeed provided by some of the open-source tools, however, none of them seems to encompass all the features, forcing one to mix and match and head down the tech debt-ridden, Franken-monitoring path.
  • Try it yourself in the MindMap below—pick a few functionalities and see how many different tools you might have to use?.docker_monitoring_oss

 

Again, if we missed something, please leave a comment or tweet @sematext.

If you want to try Sematext Docker Agent sign up for a free trial.

Image title

P.S. Sematext Docker Agent is available in the RancherOS Community Catalog and shows up with our new mascot “Octi” only one more pet—so, if you use RancherOS, search for “sematext” in the RancherOS Catalog and within a few clicks you’ll have the Sematext Docker Agent deployed to your RancherOS clusters!

Find out more about how Scalyr built a proprietary database that does not use text indexing for their log management tool.

Topics:
docker ,containers ,logging ,monitoring ,open source

Published at DZone with permission of Stefan Thies, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}