Telemetry Pipelines Workshop: Installing Fluent Bit in Container

This article is part of a series exploring a workshop guiding you through the open source project Fluent Bit, what it is, a basic installation, and setting up the first telemetry pipeline project. Learn how to manage your cloud-native data from source to destination using the telemetry pipeline phases covering collection, aggregation, transformation, and forwarding from any source to any destination. 

The previous article in this series helped with the installation of Fluent Bit on our local machine using the project source code. This time around, we'll learn how to use Fluent Bit in a container on our local machine, including how to run the container while using local configuration files. You can find more details in the accompanying workshop lab.

Let's get started with Fluent Bit in a container.

Installing Fluent Bit using a container image will be demonstrated here using the open-source project Podman. It's assumed you have previously installed the Podman command line tooling. It should also be noted that the following code and command line examples are all based on using an OSX machine.

If you want to use another container tooling, such as Docker, most of the commands are the same with just a substitution of the tooling name (docker instead of podman).

Containerized Fluent Bit

While it's not that difficult to run Fluent Bit in a container, we'll also show you how to do it using local configuration files so that you can use it to build your first telemetry pipelines.

It's pretty straightforward to run Fluent Bit in a container. Just start the container image as follows:

$ podman run --name fb -ti cr.fluentbit.io/fluent/fluent-bit:2.2.2

Let's take a look at what this command is actually doing. First, you see a flag for giving the container a name we can reference (--name fb). Another is assigning the container a console for output and staying interactive (-ti) and finally, it's using the image version supported in this workshop (cr.fluentbit.io/fluent/fluent-bit:2.2.2).

You'll notice the container starts and takes over the console for its output, where Fluent Bit is measuring CPU usage and dumping it to the console (CTRL-C will stop the container):

Fluent Bit v2.2.2
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

____________________
< Fluent Bit v2.2.2 >
 -------------------
          \
           \
            \          __---__
                    _-       /--______
               __--( /     \ )XXXXXXXXXXX\v.
             .-XXX(   O   O  )XXXXXXXXXXXXXXX-
            /XXX(       U     )        XXXXXXX\
          /XXXXX(              )--_  XXXXXXXXXXX\
         /XXXXX/ (      O     )   XXXXXX   \XXXXX\
         XXXXX/   /            XXXXXX   \__ \XXXXX
         XXXXXX__/          XXXXXX         \__---->
 ---___  XXX__/          XXXXXX      \__         /
   \-  --__/   ___/\  XXXXXX            /  ___--/=
    \-\    ___/    XXXXXX              '--- XXXXXX
       \-\/XXX\ XXXXXX                      /XXXXX
         \XXXXXXXXX   \                    /XXXXX/
          \XXXXXX      >                 _/XXXXX/
            \XXXXX--__/              __-- XXXX/
             -XXXXXXXX---------------  XXXXXX-
                \XXXXXXXXXXXXXXXXXXXXXXXXXX/
                  ""VXXXXXXXXXXXXXXXXXXV""

[2024/03/01 10:38:52] [ info] [fluent bit] version=2.2.2, commit=eeea396e88, pid=1
[2024/03/01 10:38:52] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/03/01 10:38:52] [ info] [cmetrics] version=0.6.6
[2024/03/01 10:38:52] [ info] [ctraces ] version=0.4.0
[2024/03/01 10:38:52] [ info] [input:cpu:cpu.0] initializing
[2024/03/01 10:38:52] [ info] [input:cpu:cpu.0] storage_strategy='memory' (memory only)
[2024/03/01 10:38:52] [ info] [sp] stream processor started
[2024/03/01 10:38:52] [ info] [output:stdout:stdout.0] worker #0 started
[0] cpu.local: [[1709289532.997338599, {}], {"cpu_p"=>0.250000, "user_p"=>0.000000, "system_p"=>0.250000, "cpu0.p_cpu"=>0.000000
[0] cpu.local: [[1709289533.996516160, {}], {"cpu_p"=>0.000000, "user_p"=>0.000000, "system_p"=>0.000000, "cpu0.p_cpu"=>0.000000

Should we encounter failures at any time during installation, testing, data population, or build results, don't worry! This can be rerun any time after you fix any problems reported. We might have to remove the Fluent Bit container depending on how far you get before something goes wrong. Just stop, remove, and restart it as follows:

$ podman container stop fb

$ podman container rm fb

$ podman run --name fb -ti cr.fluentbit.io/fluent/fluent-bit:2.2.2

There might come a moment when we are going to want to stop working with Fluent Bit and pause until a later time. To do this, we can shut down our container environment by stopping the running Fluent Bit container and then stopping the Podman virtual machine as follows:

$ podman container stop fb

$ podman machine stop

Let's take a look at building our container images with our specific telemetry pipeline configurations.

Building Container Images

Often you want to set up your specific configuration and add that to the container image. This means you build your own container image and then run that with custom configurations copied into the container image.

For example, let's assume we have the following files in our current directory, all parts of a Fluent Bit telemetry pipeline configuration:

• workshop-fb.conf : The main configuration file importing all other split-out configuration files
• inputs.conf: File containing all input plugin configurations
• outputs.conf: File container all output configurations

To build a container image, we need to provide a Buildfile, which defines what base container image to use and lists where to copy our above configuration files into that base image as shown here:

FROM cr.fluentbit.io/fluent/fluent-bit:2.2.2

COPY ./workshop-fb.conf /fluent-bit/etc/fluent-bit.conf
COPY ./inputs.conf /fluent-bit/etc/inputs.conf
COPY ./outputs.conf /fluent-bit/etc/outputs.conf

Once we have this file, we can now build our container image as follows:

$ podman build -t workshop-fb:v1 -f Buildfile

STEP 1/4: FROM cr.fluentbit.io/fluent/fluent-bit:2.2.2
STEP 2/4: COPY ./workshop-fb.conf /fluent-bit/etc/fluent-bit.conf
--> a379e7611210
STEP 3/4: COPY ./inputs.conf /fluent-bit/etc/inputs.conf
--> f39b10d3d6d0
STEP 4/4: COPY ./outputs.conf /fluent-bit/etc/outputs.conf
COMMIT workshop-fb:v1
--> b06df84452b6
Successfully tagged localhost/workshop-fb:v1
b06df84452b6eb7a040b75a1cc4088c0739a6a4e2a8bbc2007608529576ebeba

The build command uses the flag -t workshop-fb:v1, which tags the container image with a name and version number. This helps us to run the image by name later in the next workshop lab. Furthermore, it makes use of the -f Buildfile we just created to copy in our custom configuration files as shown in the build output.

This process now has us ready to start building our first pipelines, but what if you want to use other versions of Fluent Bit container images?

Other Versions

You might be wondering how to run other versions of Fluent Bit in a container. For example, Fluent Bit v3.0.0 was just released, so the workshop will be updated to follow the releases. Feel free to give this a try by starting the container image as follows:

$ podman run --name fb -ti cr.fluentbit.io/fluent/fluent-bit:3.0.0

This starts the container and takes over the console for its output, whereas Fluent Bit measures CPU usage and dumps it to the console (CTRL-C will stop the container):

Fluent Bit v3.0.0
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

___________.__                        __    __________.__  __          ________  
\_   _____/|  |  __ __   ____   _____/  |_  \______   \__|/  |_  ___  _\_____  \ 
 |    __)  |  | |  |  \_/ __ \ /    \   __\  |    |  _/  \   __\ \  \/ / _(__  < 
 |     \   |  |_|  |  /\  ___/|   |  \  |    |    |   \  ||  |    \   / /       \
 \___  /   |____/____/  \___  >___|  /__|    |______  /__||__|     \_/ /______  /
     \/                     \/     \/               \/                        \/ 

[2024/03/29 15:29:26] [ info] [fluent bit] version=3.0.0, commit=f499a4fbe1, pid=1
[2024/03/29 15:29:26] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/03/29 15:29:26] [ info] [cmetrics] version=0.7.0
[2024/03/29 15:29:26] [ info] [ctraces ] version=0.4.0
[2024/03/29 15:29:26] [ info] [input:cpu:cpu.0] initializing
[2024/03/29 15:29:26] [ info] [input:cpu:cpu.0] storage_strategy='memory' (memory only)
[2024/03/29 15:29:26] [ info] [output:stdout:stdout.0] worker #0 started
[2024/03/29 15:29:26] [ info] [sp] stream processor started
[0] cpu.local: [[1711726167.221360412, {}], {"cpu_p"=>0.000000, "user_p"=>0.000000, "system_p"=>0.000000, "cpu0.p_cpu"=>1.000000, "cpu0.p_user"=>0.000000, "cpu0.p_system"=>1.000000, "cpu1.p_cpu"=>0.000000, "cpu1.p_user"=>0.000000, "cpu1.p_system"=>0.000000}]
[0] cpu.local: [[1711726168.216299447, {}], {"cpu_p"=>1.000000, "user_p"=>0.500000, "system_p"=>0.500000, "cpu0.p_cpu"=>0.000000, "cpu0.p_user"=>0.000000, "cpu0.p_system"=>0.000000, "cpu1.p_cpu"=>0.000000, "cpu1.p_user"=>0.000000, "cpu1.p_system"=>0.000000}]

This puts all available container images at your fingertips.

What's Next?

This article helped us install Fluent Bit on our local machine using the available container builds. The series continues with the next step in this workshop, creating our telemetry pipelines using either the source install or container images. 

Stay tuned for more hands-on material to help you with your cloud-native observability journey.