Open Source Docker Monitoring & Logging

Pets ⇒ Cattle ⇒ Orchestration

Docker is growing by leaps and bounds, and along with it, its ecosystem. Being light, the predominant container deployment involves running just a single app or service inside each container. Most software products and services are made up of at least several such apps/services. We all want all our apps/services to be highly available and fault tolerant. Thus, Docker containers in an organization quickly start popping up like mushrooms after the rain. They multiply faster than rabbits.While, in the beginning, we play with them like cute little pets, as their numbers quickly grow we realize we are dealing with a herd of cattle, implying we’ve become cowboys. Managing a herd with your two hands, a horse, and a lasso will only get you so far. You won’t be able to ride after each and every calf that wonders in the wrong direction. To get back to containers from this zoological analogy—operating so many moving pieces at scale is impossible without orchestration—this is why we’ve seen the rise of Docker Swarm, Kubernetes, Mesos, CoreOS, RancherOS, and so on.

Pets ⇒ Cattle ⇒ Orchestration + Operational Insights

Container orchestration helps you manage your containers, their placement, their resources, and their whole life cycle. While containers and applications in them are running, in addition to the whole life cycle management, we need container monitoring and log management so we can troubleshoot performance or stability issues, debug or tune applications, and so on. Just like with orchestration, there are a number of open-source container monitoring and logging tools. It’s great to have choices, but having lots of them means you need to evaluate and compare them to pick the one that best matches your needs.

DevOps Tools Comparison

We’ve open-sourced our Sematext Docker Agent (SDA for short) which works with SPM for monitoring and Logsene for log management (think of it as ELK as a Service), and wanted to provide a high level comparison of SDA and several popular Docker monitoring and logging tools, like CAdvisor, Logspout, and others.  In the following table, we group tools by functionality and include monitoring agents, log collectors and shippers, storage backends, and tools that provide the UI and visualizations. For each functionality, we list in the “Common Tools” column one or more popular open-source tools that provide that functionality.  An empty “Common Tools” cell means there are no popular open-source tools that provide it, or at least we are not aware of it—if we messed something up, please leave a comment or tweet @sematext.

This table shows a few things:

• Some of the functionality provided by SPM and Logsene is not available in some of the most popular open-source monitoring and logging tools included here.
• Some of the SPM and Logsene functionality is indeed provided by some of the open-source tools, however, none of them seems to encompass all the features, forcing one to mix and match and head down the tech debt-ridden, Franken-monitoring path.
• Try it yourself in the MindMap below—pick a few functionalities and see how many different tools you might have to use?.

Again, if we missed something, please leave a comment or tweet @sematext.

If you want to try Sematext Docker Agent sign up for a free trial.

P.S. Sematext Docker Agent is available in the RancherOS Community Catalog and shows up with our new mascot “Octi” only one more pet—so, if you use RancherOS, search for “sematext” in the RancherOS Catalog and within a few clicks you’ll have the Sematext Docker Agent deployed to your RancherOS clusters!