Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Log Management in Containers With Fluentd, Docker Log Drivers, and Kontena

DZone's Guide to

Log Management in Containers With Fluentd, Docker Log Drivers, and Kontena

Read on to learn about these three great tools that can help you better manage your application logs through three key log management processes

· Cloud Zone
Free Resource

Are you joining the containers revolution? Start leveraging container management using Platform9's ultimate guide to Kubernetes deployment.

As we all know, application logs are crucial information in both debugging issues users are facing and also to see how an application is behaving. In multi-server container environments, it is crucial to have one single place to check all the application logs. This blog post summarizes how Kontena manages application logs and what possibilities it offers for further log processing.

Log Capturing

The brilliant 12Factor principles define application logs as a stream of aggregated, time-ordered events. In the principles, it is also stated that each application should not concern how and where the logs are stored, but instead just always dump logs into stdout unbuffered. This is fantastic advice as it really eases up application development from bundling different ways to store logs for each environment the application is used in.

Now with containers in use, this is even more important as container engines (Docker in this case) capture the stdout and stderr streams. In the case of managing your containerized apps with the Kontena Platform, Kontena will also tap into the Docker engine's log streams for each container. While streaming the application logs from each container, Kontena Agent then pushes the logs to be stored in Kontena Master's database. In Kontena Master, since it uses MongoDB as its database, the logs are stored in a capped collection so that they don't consume too much disk space. A capped collection behaves similarly to circular buffers: once the collection fills up, the older entries are overwritten to make room for new entries.

Kontena's log capturing makes it possible to use one single CLI command to see application logs for all the containers in a given service, for example:

$ kontena service logs todo/web
2017-04-27T10:58:48.000Z [5]: `/` is not writable.  
2017-04-27T10:58:48.000Z [5]: Bundler will use `/tmp/bundler/home/unknown' as your home directory temporarily.  
2017-04-27T10:58:49.000Z [5]: Puma starting in single mode...  
2017-04-27T10:58:49.000Z [5]: * Version 3.4.0 (ruby 2.3.1-p112), codename: Owl Bowl Brawl  
2017-04-27T10:58:49.000Z [5]: * Min threads: 0, max threads: 16  
2017-04-27T10:58:49.000Z [5]: * Environment: production  
2017-04-27T10:58:51.000Z [5]: * Listening on tcp://0.0.0.0:9292  
2017-04-27T10:58:51.000Z [5]: Use Ctrl-C to stop  
2017-04-27T10:59:31.000Z [9]: `/` is not writable.  
2017-04-27T10:59:31.000Z [9]: Bundler will use `/tmp/bundler/home/unknown' as your home directory temporarily.  
2017-04-27T10:59:32.000Z [9]: Puma starting in single mode...  
2017-04-27T10:59:32.000Z [9]: * Version 3.4.0 (ruby 2.3.1-p112), codename: Owl Bowl Brawl  
2017-04-27T10:59:32.000Z [9]: * Min threads: 0, max threads: 16  
2017-04-27T10:59:32.000Z [9]: * Environment: production  
2017-04-27T10:59:34.000Z [9]: * Listening on tcp://0.0.0.0:9292  
2017-04-27T10:59:34.000Z [9]: Use Ctrl-C to stop  
2017-04-27T11:03:15.000Z [6]: `/` is not writable.  
2017-04-27T11:03:15.000Z [6]: Bundler will use `/tmp/bundler/home/unknown' as your home directory temporarily.  
2017-04-27T11:03:16.000Z [6]: Puma starting in single mode...  
2017-04-27T11:03:16.000Z [6]: * Version 3.4.0 (ruby 2.3.1-p112), codename: Owl Bowl Brawl  
2017-04-27T11:03:16.000Z [6]: * Min threads: 0, max threads: 16  
2017-04-27T11:03:16.000Z [6]: * Environment: production  
2017-04-27T11:03:19.000Z [6]: * Listening on tcp://0.0.0.0:9292  
2017-04-27T11:03:19.000Z [6]: Use Ctrl-C to stop  

Log Forwarding

While seeing the logs of an application (comprised of multiple containers) is fantastic, it does not offer any real search capabilities. When you have plenty of services and/or a multitude of container instances for a service, real query capabilities are necessary. The following sections will provide some alternatives on how to get your application logs streamed to a system that provides more sophisticated analytics and query capabilities. Setting up one of those systems warrants its own full post and there are plenty of good SaaS services for log management also available.

Fluentd Forwarding

Kontena 1.2.0 came with an option to forward all container logs with Fluentd protocol from the nodes. So while the Kontena Agent captures the container logs from the Docker log streams, it forwards the logs to a specified Fluentd endpoint in addition to sending them to Kontena Master.

When Kontena Agent forwards the logs using Fluentd, it tags all the log events with a pattern hostname.grid_name.stack.service.instance. That makes it possible to do aggregations and other analytics in the log processing system.

Docker Log Drivers

Another option to get logs shipped for further processing is to define log drivers to be used on Docker level.

web:  
  image: nginx
  logging:
    driver: syslog
    options:
      syslog-address: "tcp://192.168.0.42:123"

The downside of this approach is that the logs will not get stored on Kontena Master at all since Docker will send the logs directly from your containers, thus Kontena Agent is not able to capture them.

Streaming Logs From Kontena Master DB

Yet another option to get logs forwarded for further processing is to grab them from the Kontena Master database.

To gather logs from the Master database directly, you need to run the collector somewhere that has access to the Master database. Usually, the database is not exposed to the outside world from the Master Node, so the most natural way is to run it alongside the Master.

For example, with Fluentd, you could use the following configuration to get the logs shipped to AWS S3:

<source>  
  type mongo_tail
  url "#{ENV['MONGODB_URL']}"
  collection container_logs
  tag_key name
  time_key created_at
  id_store_collection container_logs_tail
</source>

<match **>  
  @type s3

  aws_key_id "#{ENV['S3_ACCESS_KEY']}"
  aws_sec_key "#{ENV['S3_SECRET_KEY']}"
  s3_bucket "#{ENV['S3_BUCKET']}"
  s3_region "#{ENV['S3_REGION']}"
  buffer_type memory
  buffer_chunk_limit 256m
  buffer_queue_limit 128
  path logs/

  format json
  include_time_key true
  include_tag_key true

  s3_object_key_format %{path}/ts=%{time_slice}/%{index}_json.%{file_extension}
  time_slice_format %Y%m%d-%H
  time_slice_wait 30m
  utc
</match>  

Log Processing

There's a plethora of different options available for log management and processing functionality. One of the most popular on-premise solutions is the open-source ELK stack. For a commercial option, Splunk offers fantastic capabilities but comes with a rather high price tag.

For hosted options, there's at least logz.io, Papertrail, Loggly, Logentries and AWS CloudWatch to name a few. As usual, the benefit of using a hosted solution is the non-existent maintenance burden which of course comes with a price tag.

I'm planning to do a write-up in the coming weeks on how to setup a private ELK stack with Kontena, so stay tuned.

Using Containers? Read our Kubernetes Comparison eBook to learn the positives and negatives of Kubernetes, Mesos, Docker Swarm and EC2 Container Services.

Topics:
cloud ,log management ,mongodb ,containers ,docker

Published at DZone with permission of Jussi Nummelin, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}