DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations
Securing Your Software Supply Chain with JFrog and Azure
Register Today

Trending

  • From CPU to Memory: Techniques for Tracking Resource Consumption Over Time
  • RBAC With API Gateway and Open Policy Agent (OPA)
  • Decoding ChatGPT: The Concerns We All Should Be Aware Of
  • Knowing and Valuing Apache Kafka’s ISR (In-Sync Replicas)

Trending

  • From CPU to Memory: Techniques for Tracking Resource Consumption Over Time
  • RBAC With API Gateway and Open Policy Agent (OPA)
  • Decoding ChatGPT: The Concerns We All Should Be Aware Of
  • Knowing and Valuing Apache Kafka’s ISR (In-Sync Replicas)
  1. DZone
  2. Software Design and Architecture
  3. Cloud Architecture
  4. Open Source Docker Monitoring & Logging

Open Source Docker Monitoring & Logging

Check out this handy comparison of open source tools for logging and monitoring Docker containers. Be sure to click on the MindMap at the bottom for a handy illustration to help with grouping complimentary tools together.

Stefan Thies user avatar by
Stefan Thies
·
Jul. 26, 16 · Analysis
Like (7)
Save
Tweet
Share
9.01K Views

Join the DZone community and get the full member experience.

Join For Free

Pets ⇒ Cattle ⇒ Orchestration

Docker is growing by leaps and bounds, and along with it, its ecosystem. Being light, the predominant container deployment involves running just a single app or service inside each container. Most software products and services are made up of at least several such apps/services. We all want all our apps/services to be highly available and fault tolerant. Thus, Docker containers in an organization quickly start popping up like mushrooms after the rain. They multiply faster than rabbits.While, in the beginning, we play with them like cute little pets, as their numbers quickly grow we realize we are dealing with a herd of cattle, implying we’ve become cowboys. Managing a herd with your two hands, a horse, and a lasso will only get you so far. You won’t be able to ride after each and every calf that wonders in the wrong direction. To get back to containers from this zoological analogy—operating so many moving pieces at scale is impossible without orchestration—this is why we’ve seen the rise of Docker Swarm, Kubernetes, Mesos, CoreOS, RancherOS, and so on.

Pets ⇒ Cattle ⇒ Orchestration + Operational Insights

Container orchestration helps you manage your containers, their placement, their resources, and their whole life cycle. While containers and applications in them are running, in addition to the whole life cycle management, we need container monitoring and log management so we can troubleshoot performance or stability issues, debug or tune applications, and so on. Just like with orchestration, there are a number of open-source container monitoring and logging tools. It’s great to have choices, but having lots of them means you need to evaluate and compare them to pick the one that best matches your needs.

DevOps Tools Comparison

We’ve open-sourced our Sematext Docker Agent (SDA for short) which works with SPM for monitoring and Logsene for log management (think of it as ELK as a Service), and wanted to provide a high level comparison of SDA and several popular Docker monitoring and logging tools, like CAdvisor, Logspout, and others.  In the following table, we group tools by functionality and include monitoring agents, log collectors and shippers, storage backends, and tools that provide the UI and visualizations. For each functionality, we list in the “Common Tools” column one or more popular open-source tools that provide that functionality.  An empty “Common Tools” cell means there are no popular open-source tools that provide it, or at least we are not aware of it—if we messed something up, please leave a comment or tweet @sematext.

Functionality Common Tools Sematext Tools
Collect Logs from Docker API
(including auto-discovery of new containers)
Logspout Sematext Docker Agent
Log routing Logspout
Routing setup for containers via HTTP API to syslog, redis, kafka, logstash 
Docker Logging Drivers (e.g. syslog, journald, fluentd, etc.)
Sematext Docker Agent
(routing of logs to different indices based on container labels)
Automatic log tagging
(with Docker Compose or Swarm or Kubernetes metadata)
For Kubernetes: fluentd-elasticsearch, assumes Elasticsearch deployed locally Sematext Docker Agent
Collect Docker Metrics CAdvisor Sematext Docker Agent
Collect Docker Events ? Sematext Docker Agent
Logs format detection
(most tools need a static setup per logfile/application)
? Sematext Docker Agent
(out of the box format detection and parsing; the parser and the logagent-js pattern library is open source)
Logs parsing and shipping Fluentd
Logstash
rsyslog
syslog-ng
Sematext Docker Agent
Logs storage and indexing Elasticsearch
Solr
Logsene
(exposes Elasticsearch API)
Logs anomaly detection and alerting ? Logsene
Log search and analytics Kibana
Grafana
Logsene
(Logsene’s own UI or integrated Kibana, or Grafana connected to Logsene via Elasticsearch data source)
Metrics storage and aggregation Graphite
OpenTSDB
KairosDB
Elasticsearch
Influxdb
Prometheus
SPM
Metrics charts and dashboards Grafana
Kibana
SPM
Metrics anomaly detection and alerting Influxdb
Prometheus
SPM
Correlation of Metrics, Logs and Events ?

SPM & Logsene integration

This table shows a few things:

  • Some of the functionality provided by SPM and Logsene is not available in some of the most popular open-source monitoring and logging tools included here.
  • Some of the SPM and Logsene functionality is indeed provided by some of the open-source tools, however, none of them seems to encompass all the features, forcing one to mix and match and head down the tech debt-ridden, Franken-monitoring path.
  • Try it yourself in the MindMap below—pick a few functionalities and see how many different tools you might have to use?.docker_monitoring_oss

 

Again, if we missed something, please leave a comment or tweet @sematext.

If you want to try Sematext Docker Agent sign up for a free trial.

Image title

P.S. Sematext Docker Agent is available in the RancherOS Community Catalog and shows up with our new mascot “Octi” only one more pet—so, if you use RancherOS, search for “sematext” in the RancherOS Catalog and within a few clicks you’ll have the Sematext Docker Agent deployed to your RancherOS clusters!

Docker (software) Open source Kubernetes

Published at DZone with permission of Stefan Thies, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Trending

  • From CPU to Memory: Techniques for Tracking Resource Consumption Over Time
  • RBAC With API Gateway and Open Policy Agent (OPA)
  • Decoding ChatGPT: The Concerns We All Should Be Aware Of
  • Knowing and Valuing Apache Kafka’s ISR (In-Sync Replicas)

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com

Let's be friends: