Collecting All Docker Logs with Fluentd

Just in case you have been offline for the last two years, Docker is an open platform for distributed apps for developers and sysadmins. By turning your software into containers, Docker lets cross-functional teams ship and run apps across platforms seamlessly.

In a container-centric architecture, applications come and go all the time, which means keeping track of application logs presents operational challenges. Where can logs be archived safely? The host machine is a non-starter since thousands of containers can run on a single host. HDFS or S3 can be a good permanent home for container logs, but how do containers ship their logs to them?

In Version 1.6, Docker added the Logging Driver to solve this problem. By specifying the “–log-driver” option, the Docker user can specify where to send logs to on a per-container basis.

Fluentd as a Docker Logging Driver

As the original creator of Fluentd, an open source data collector for building the unified logging layer, we welcomed this development. We were so excited that one of our engineers and a Fluentd committer, Satoshi Tagomori, sent a pull request to add Fluentd as a Logging Driver.

And today, we are happy to announce that Satoshi’s pull request has been merged. On the master branch, Fluentd is already supported as a Logging Driver and slated to be released in Version 1.8!

So, how would this work? For the truly impatient, I will give a quick tour. The rest of this blog entry has been tested on Ubuntu 14.04 LTS.

Step 1: Getting Docker

Run the following command:

sudo  wget -qO- https://get.docker.com/ | sh

As of July 1, 2015, this installs Docker 1.7. Run “docker –version” to confirm Docker’s version. If it is indeed 1.7 or older, please follow Step 1b. Otherwise, go to Step 2.

Step 1b: Getting Docker from the Master Branch (Until ver 1.8 is released!)

Stop Docker:

sudo service docker stop

Rename the current Docker binary.

sudo mv /usr/bin/docker /usr/bin/docker-orig

Get Docker 1.8

sudo wget -O /usr/bin/dockerhttps://master.dockerproject.org/linux/amd64/docker-1.8.0-dev

Change the permission for Docker 1.8 binary.

sudo chmod 755 /usr/bin/docker

Finally, restart Docker

sudo service docker start

Step 2: Getting Fluentd via td-agent

The easiest way to download Fluentd is via td-agent, the Fluentd package maintained by Treasure Data. Run the following command.

curl -L https://td-toolbelt.herokuapp.com/sh/install-ubuntu-trusty-td-agent2.sh | sh

Step 3: Configuring td-agent

One of the key features of Fluentd is its ability to route events based on their tags. By default, Docker messages are sent with the tag “docker.<CONTAINER_ID>”. This can be configure with Docker’s –log-opt option. For example, if you want to use descriptive container names, you can do so with “–log-opt fluentd-tag=docker.{{.Name}}”.

To test this feature, let’s configure td-agent to output all events with the tag prefixed with “docker” to stdout. Edit /etc/td-agent/td-agent.conf and add the following lines:

<match docker.**>
type stdout
</match>

Then, restart td-agent as follows:

sudo service td-agent restart

Step 4: Launch a Container and Confirm

Finally, let’s launch a container and send logs to the host’s td-agent.

docker run –log-driver=fluentd hello-world

Then, let’s look at td-agent’s stdout log, which can be found in /var/log/td-agent/td-agent.log

tail -n 3 /var/log/td-agent/td-agent.log

2015-07-01 16:56:02 -0400 docker.c61d13c68659: {“container_id”:”c61d13c68659b622a01d8c3825b0bc1186391119d47dbf864d9c3a65c3f2aa79″,”container_name”:”/distracted_bell”,”source”:”stdout”,”log”:””}

2015-07-01 16:56:02 -0400 docker.c61d13c68659: {“source”:”stdout”,”log”:”For more examples and ideas, visit:”,”container_id”:”c61d13c68659b622a01d8c3825b0bc1186391119d47dbf864d9c3a65c3f2aa79″,”container_name”:”/distracted_bell”}

2015-07-01 16:56:02 -0400 docker.c61d13c68659: {“container_name”:”/distracted_bell”,”source”:”stdout”,”log”:” http://docs.docker.com/userguide/”,”container_id”:”c61d13c68659b622a01d8c3825b0bc1186391119d47dbf864d9c3a65c3f2aa79″}

What’s Next?

Recall that Fluentd/td-agent are capable of sending logs to hundreds of backend systems such as Elasticsearch, MongoDB, HDFS and yes, Treasure Data. So, install Fluentd on your Docker host and start shipping your logs to the backend of your choice!