Mastering Fluent Bit: Developer Guide to Service Section Configuration (Part 5)
This intro to mastering Fluent Bit covers handy tips and tricks for speeding up the inner development loop when integrating telemetry pipelines.
CORE
·
Join the DZone community and get the full member experience.
Join For FreeThis series is a general-purpose getting-started guide for those of us wanting to learn about the Cloud Native Computing Foundation (CNCF) project Fluent Bit. Each article in this series addresses a single topic by providing insights into what the topic is, why we are interested in exploring that topic, where to get started with the topic, and how to get hands-on with learning about the topic as it relates to the Fluent Bit project. The idea is that each article can stand on its own, but that they also lead down a path that slowly increases our abilities to implement solutions with Fluent Bit telemetry pipelines.
Let's take a look at the topic of this article, using Fluent Bit tips and tricks for developers. In case you missed the previous article, check out using a Fluent Bit pipeline on a Kubernetes cluster to take control of all the logs being generated.
This article will be a hands-on tour of the things that help you as a developer testing out your Fluent Bit pipelines. We'll take a look at the services, input, and output sections of your configurations. All examples in this article have been performed on macOS and assume the reader is able to replicate the actions shown here on their own local machines.
Where to Get Started
You should have explored the previous articles in this series to install and get started with Fluent Bit on your developer's local machine. This can be with containers, and it can be from source, either will work. Links at the end of this article will point you to a free hands-on workshop that lets you explore more of Fluent Bit in detail.
You can verify that you have a functioning installation by testing one of the two methods for using Fluent Bit, first the source installation, the second using a container image (with Podman in the following examples):
# For source installation.
$ fluent-bit -i dummy -o stdout
# For container installation.
$ podman run -ti ghcr.io/fluent/fluent-bit:4.0.4 -i dummy -o stdout
...
[0] dummy.0: [[1753105021.031338000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105022.033205000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105023.032600000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105024.033517000, {}], {"message"=>"dummy"}]
...Let's look at a few tips and tricks to help you with your inner development loop.
Service Section: Hot Reload
When first getting started with Fluent Bit, the service section of the configuration does not seem to be that important. There is one thing that is really useful that is often missed by developers, and that's the ability to restart a source installation of Fluent Bit during the inner development loop usage to quickly test configuration changes by configuring hot reloading.
A simple example configuration file named fluent-bit.yaml can be found below:
service:
flush: 1
log_level: info
hot_reload: on
pipeline:
inputs:
# This entry generates a test INFO log level message.
- name: dummy
tag: msg.info
dummy: '{"message":"This is INFO message", "level":"INFO", "color": "yellow"}'
# This entry generates a test ERROR log level message.
- name: dummy
tag: msg.error
dummy: '{"message":"This is ERROR message", "level":"ERROR", "color": "red"}'
outputs:
- name: stdout
match: '*'
format: json_linesNow let's run this as follows:
$ fluent-bit --config fluent-bit.yaml
...
{"date":1753107336.593176,"message":"This is INFO message","level":"INFO","color":"yellow"}
{"date":1753107336.593878,"message":"This is ERROR message","level":"ERROR","color":"red"}
{"date":1753107337.591496,"message":"This is INFO message","level":"INFO","color":"yellow"}
{"date":1753107337.591669,"message":"This is ERROR message","level":"ERROR","color":"red"}
{"date":1753107338.586933,"message":"This is INFO message","level":"INFO","color":"yellow"}
{"date":1753107338.587137,"message":"This is ERROR message","level":"ERROR","color":"red"}
...This will alternate between the two messages and go on forever. In the process of testing and development, we decided to modify the configuration to filter messages with info status by removing the color key. This change means we need to restart the running Fluent Bit instance using the hot reload functionality.
Below are the new configuration changes:
service:
flush: 1
log_level: info
hot_reload: on
pipeline:
inputs:
# This entry generates a test INFO log level message.
- name: dummy
tag: msg.info
dummy: '{"message":"This is INFO message", "level":"INFO", "color": "yellow"}'
# This entry generates a test ERROR log level message.
- name: dummy
tag: msg.error
dummy: '{"message":"This is ERROR message", "level":"ERROR", "color": "red"}'
filters:
- name: modify
match: msg.info
remove: color
outputs:
- name: stdout
match: '*'
format: json_linesThe first method to trigger hot reloading without stopping our instance is done on Linux systems using SIGHUP as follows (note we are using the PIDOF helper tool to identify our running Fluent Bit process ID, but you can find this using PS or your favorite tool):
$ kill -s SIGHUP $(pidof fluent)
...
{"date":1753107336.593176,"message":"This is INFO message","level":"INFO","color":"yellow"}
{"date":1753107336.593878,"message":"This is ERROR message","level":"ERROR","color":"red"}
{"date":1753107337.591496,"message":"This is INFO message","level":"INFO","color":"yellow"}
{"date":1753107337.591669,"message":"This is ERROR message","level":"ERROR","color":"red"}
{"date":1753107338.586933,"message":"This is INFO message","level":"INFO","color":"yellow"}
{"date":1753107338.587137,"message":"This is ERROR message","level":"ERROR","color":"red"}
[2025/07/21 16:20:34] [engine] caught signal (SIGHUP)
... <<<< RESTARTING_FLUENT_BIT_LOG_MESSAGES
{"date":1753107636.966377,"message":"An INFO message","level":"INFO"}
{"date":1753107636.966658,"message":"An ERROR message","level":"ERROR","color":"red"}
{"date":1753107637.96652,"message":"An INFO message","level":"INFO"}
{"date":1753107637.966711,"message":"An ERROR message","level":"ERROR","color":"red"}
{"date":1753107638.966835,"message":"An INFO message","level":"INFO"}
{"date":1753107638.966975,"message":"An ERROR message","level":"ERROR","color":"red"}
...The log output catches the SIGHUP, restarts our Fluent Bit instance in place, and applies the new configuration changes, resulting in the changed output as shown with color removed from all INFO-level events.
Another way to access this hot reloading of your instance is to use the configurable HTTP server and leverage the provided endpoints:
- PUT /api/v2/reload
- POST /api/v2/reload
Add the following to the service section in the configuration file:
service:
flush: 1
log_level: info
http_server: on
http_listen: 0.0.0.0
http_port: 2020
hot_reload: onNow start your instance (still using the same configuration as before, just with the new HTTP server added) and try the commands below to show hot reloading:
$ fluent-bit --config fluent-bit.yaml
...
{"date":1753344624.827537,"message":"An INFO message","level":"INFO"}
{"date":1753344624.827891,"message":"An ERROR message","level":"ERROR","color":"red"}
{"date":1753344625.829556,"message":"An INFO message","level":"INFO"}
{"date":1753344625.829864,"message":"An ERROR message","level":"ERROR","color":"red"}
{"date":1753344626.829288,"message":"An INFO message","level":"INFO"}
{"date":1753344626.829416,"message":"An ERROR message","level":"ERROR","color":"red"}
...
# In a different terminal window.
$ curl -X POST -d '{}' localhost:2020/api/v2/reload
{"reload":"done","status":0}
# Or in a different terminal window.
$ curl -X PUT -d '{}' localhost:2020/api/v2/reload
{"reload":"done","status":0}
# Meanwhile, back in the Fluent Bit terminal window...
#
...
{"date":1753344797.824589,"message":"An INFO message","level":"INFO"}
{"date":1753344797.824942,"message":"An ERROR message","level":"ERROR","color":"red"}
[2025/07/24 10:13:19] [engine] caught signal (SIGHUP)
... <<<< RESTARTING_FLUENT_BIT_LOG_MESSAGES
{"date":1753344801.312548,"message":"An INFO message","level":"INFO"}
{"date":1753344801.31283,"message":"An ERROR message","level":"ERROR","color":"red"}
{"date":1753344802.309874,"message":"An INFO message","level":"INFO"}
{"date":1753344802.310077,"message":"An ERROR message","level":"ERROR","color":"red"}
...If you are using a REST client, it might look something like this:
Note, we can monitor the number of hot reloads that an instance has undertaken by using the following API call:
# The following API call returns the count of reloads done
# on this instance, in the example below its been reloaded twice.
#
$ curl -X GET -d '{}' localhost:2020/api/v2/reload
{"hot_reload_count":2}This wraps up a few handy tips and tricks for developers getting started with Fluent Bit in their solutions. The ability to set up and leverage hot reloading is a big help in speeding up your inner development loop experience.
More in the Series
In this article, you learned a few handy tricks for using the Fluent Bit service section in the configuration to improve the inner developer loop experience. This article is based on this online free workshop.
There will be more in this series as you continue to learn how to configure, run, manage, and master the use of Fluent Bit in the wild. Next up, exploring some of the more interesting Fluent Bit input plugins for developers.
Published at DZone with permission of Eric D. Schabell. See the original article here.
Opinions expressed by DZone contributors are their own.
Comments