Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Docker Logging with the Logz.io Logging Plugin

DZone's Guide to

Docker Logging with the Logz.io Logging Plugin

Check out how you can use this new Logz.io Docker plugin replace the default JSON logger for shipping your container logs.

· Cloud Zone ·
Free Resource

Insight into the right steps to take for migrating workloads to public cloud and successfully reducing cost as a result. Read the Guide.

We are happy to inform our users that a new Docker logging plugin is available on the Docker Store! Using this plugin, users can easily ship container logs directly to Logz.io, and enjoy the following benefits:

  • Simple installation on the Docker daemon level. Can be configured per container or per host.
  • The plugin operates in isolation, preventing users from accidentally deleting or stopping container log shipping.
  • The plugin gracefully shuts down when the container stops.
  • Data is saved on disk to protect it as much as possible.
  • Can use multiple account tokens for different containers running on the same host, no additional setup process is required (useful for Logz.io users using sub-accounts).

In this article, we will provide some background information on Docker plugins, how we built the Logz.io logging plugin, and of course, explain how to use it to ship container logs to Logz.io. To follow the instructions, you will need a Logz.io account.

Before we start, make sure you have the following prerequisites:

  • Docker Engine version 17.05 or later. If you plan to configure this plugin using daemon.json, you need Docker Community Edition (Docker-ce) 18.03 or later.
  • A Logz.io account.

What are Docker Plugins?

Docker defines plugins as "out-of-process" extensions that are used to add capabilities to the Docker Engine. Docker cannot cover all the custom needs for a Docker container, and plugins are a good place to start if you need to extend Docker functionalities.

You can think of plugins as "swappable batteries", where you are swapping default, or built-in, components with a new component. In this article, we describe how to swap the default JSON logging driver with the new Logz.io logging plugin for shipping Docker container logs directly to Logz.io.

Plugins became first-class citizens as of version 17.03, which means that a full plugin cycle can be managed via the Docker CLI.

Docker currently supports five different interfaces: Authorization, Volume, Network, IPAM, and Logging.

How do Plugins Work?

A plugin is a process that can run on one or multiple hosts. The Docker Engine has a plugin discovery mechanism to automatically detect new plugins on the machine and on a remote host. Currently, the discovery mechanism supports three types of files which can be put in the plugin directory:

  • .sock - files are UNIX domain sockets.
  • .spec - files are text files containing a URL.
  • .json - files are text files containing a full json specification for the plugin.

Docker will first check the /run/docker/plugins folder, and then will continue to look for spec files in /etc/docker/plugins or /usr/lib/docker/plugins.

Writing a Logging Plugin

To write a logging plugin, the plugin needs to implement an HTTP server which can be discovered by the Docker daemon.

The Logz.io logging plugin forwards container logs to Logz.io. It is written in Go, mainly because of go-plugin-helper — a collection of helper packages to extend Docker Engine, but you can write a plugin in any programming language of your choice.

Logging plugins must register as a LogDriver during plugin activation. Once activated, users can specify the plugin as a log driver.

There are two required, and two optional, HTTP endpoints that logging plugins must implement:

/LogDriver.StartLogging (required)

Request: Signals the plugin to support a new container with the plugin as a logging driver. The request contains a file path to the log stream that needs to be consumed and an info struct that holds all the metadata about the container.

Response: A key-value response that specifies any error that occurs. If no error occurs, the value will remain blank.


func (d *Driver) StartLogging(file string, logCtx logger.Info) error


In this section, we implement the registration of a new logger for the container, add it to the appropriate shipper, and allocate on-disk backup mechanism in case the shipping failed etc.

/LogDriver.StopLogging (required)

Request: Signals the plugin that one of the containers that use it as a logging driver stopped running. It contains a file path to the log stream that is going to be removed.

Response: If an error occurred, it will show up as an error. If not, the response will remain empty.


func (d *Driver) StopLogging(file string) error


Here, we erase the container from our container's driver list but not before making sure we ship all the remaining logs in the queue.

/LogDriver.Capabilities (optional)

Request: Empty

Response: A boolean value set to either true if the plugin supports 'ReadLogs' or false if 'ReadLogs' is not supported.

Logz.io plugin supports 'ReadLogs' and thus we return true.

/LogDriver.ReadLogs (optional)

Request: Signals the plugin to read logs back to the client. It contains read configurations struct which is a list of options for reading and an info struct about the requested container.

Response: Log stream


func (d *Driver) ReadLogs(info logger.Info, config logger.ReadConfig) (io.ReadCloser, error)


For this endpoint, the plugin needs to implement the logic of reading the logs back to the client when requested — ' docker logs <container_id> '.

Installing the Logz.io Logging Plugin

So how do we use the Logz.io logging plugin?

Since the plugin is available on the Docker Store, installing the plugin is extremely simple and can be done with one simple docker command:

docker plugin install store/logzio/logzio-logging-plugin:1.0.0 --alias logzio/logzio-logging-plugin


Type "y" and press "Enter" to approve the plugin privileges.

Plugin "store/logzio/logzio-logging-plugin:1.0.0" is requesting the following privileges:
- network: [host]
Do you grant the above permissions? [y/N] y


To verify the plugin is installed, enter:

docker plugin ls

ID NAME DESCRIPTION ENABLED
a5f8e561b726 logzio/logzio-logging-plugin:latest Logz.io logging plugin true


And that's it! The plugin is installed and you can now use the Logz.io logging driver to ship container logs.

Using the Logz.io Docker Logging Plugin

Let's play around with the plugin a bit.

The most basic command you can execute is to run a container while specifying Logz.io as the logging driver.

The Logz.io plugin requires you to supply three parameters:

  • "logzio-url" - Logz.io HTTPs listener (e.g. https://listener.logz.io:8071 )
  • "logzio-token" - Logz.io account token (can be retrieved from the Settings page within the Logz.io UI).
  • "logzio-dir-path" - Logz.io log queue dir path

In addition to the three required parameters, you have a lot of optional parameters you can use. To see a list of these options, visit the plugin's GitHub page .

The full command should look something that is similar to this:

docker run --log-driver=logzio/logzio-logging-plugin:latest --log-opt logzio-token=
<Logz.io_token> --log-opt logzio-url=<Logz.io url> 
--log-opt logzio-dir-path=<path> <container_name>


Additional Tips n' Tricks

The plugin is a container running natively via containerd, the low-level component that implements container execution and provides process management. This means that there are some architectural and functional points that need to be clarified.

Logging the Plugin

The plugin's logs are logged together with the Docker daemon logs. On Ubuntu, for example, enter the following command to see the logs:


sudo journalctl -fu docker.service


You will notice that the logs contain both the Docker daemon logs and the plugin logs, but plugin logs have a plugin=<id> appended to their log lines:

May 24 10:36:36 ip-172-31-94-175 dockerd[26713]: 
time="2018-05-24T10:36:36Z" level=error 
msg="time=\"2018-05-24T10:36:36Z\" level=info msg=\"logzio: Stopping 
logging Driver for closed container 
b0b9ee32374f83545454166e7d7f9458672c779624201bb063680b6bea8723c7.\"" 
plugin=a5f8e561b726aa490b45062f2f59e6a4255ca1b877b8fcdb8dae74b4026360ef


Setting Environment Variables

Because the plugin is a container running via containerd , it won't be visible when typing the common command docker ps -a   .

In addition, if we will try to set an environment variable inside the container that uses the plugin, it won't work because the plugin is not aware of it.

A quick look at docker plugin --help shows the command "set", which we can use to change, for example, the Logz.io drain timeout:

docker plugin set logzio/logzio-logging-plugin 
LOGZIO_DRIVER_LOGS_DRAIN_TIMEOUT=1s


A quick inspection of the plugin shows that the drain timeout changed to one second:

docker plugin inspect logzio/logzio-logging-plugin


"Enabled": true,
        "Id": "a5f8e561b726aa490b45062f2f59e6a4255ca1b877b8fcdb8dae74b4026360ef",
        "Name": "logzio/logzio-logging-plugin:latest",
        "PluginReference": "docker.io/store/logzio/logzio-logging-plugin:1.0.0",
        "Settings": {
            "Args": [],
            "Devices": [],
            "Env": [
                "LOG_LEVEL=info",
                "LOGZIO_DRIVER_LOGS_DRAIN_TIMEOUT=1s",
                "LOGZIO_DRIVER_CHANNEL_SIZE=10000",
                "LOGZIO_DRIVER_DISK_THRESHOLD=70",
                "LOGZIO_MAX_MSG_BUFFER_SIZE=1048576",
                "LOGZIO_MAX_PARTIAL_BUFFER_DURATION=500ms"
            ],


Summing It Up

Logz.io supports various ways of shipping data from a Dockerized environment, including the Docker log collector which can be used for shipping Docker stats and Docker daemon events to Logz.io in addition to container logs.

Each method has its advantages and disadvantages, and we are continuously working on improving these methods. The new plugin introduced here works using native integration with the Docker engine, can be installed per host or per container, and has various running options that can be used when using it.

TrueSight Cloud Cost Control provides visibility and control over multi-cloud costs including AWS, Azure, Google Cloud, and others.

Topics:
cloud ,docker ,docker plugins ,container logging ,containerd ,logz.io

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}