Over a million developers have joined DZone.

Musings in YAML: Tips for Configuring Elasticsearch Beats

DZone's Guide to

Musings in YAML: Tips for Configuring Elasticsearch Beats

It ain’t rocket science, but a small and simple mistake can make all the difference between a bad day and an even worse day.

· Performance Zone ·
Free Resource

SignalFx is the only real-time cloud monitoring platform for infrastructure, microservices, and applications. The platform collects metrics and traces across every component in your cloud environment, replacing traditional point tools with a single integrated solution that works across the stack.

The following scenario might sound familiar.

After carefully configuring the prospectors and output settings in your Filebeat configuration, you finally start the service only to open up Kibana and face an empty Discover page.

After hours of searching the documentation and forums online, you finally find your mistake — one of your prospectors was entered in the configuration file with bad indentation!

If only you would have been more attentive during those YAML lessons!

The truth is that there are no lessons teaching us YAML syntax. Elastic has improved the configurations tests that you can run, but some errors still pass these tests. It’s just one of those things we are expected to know. Sadly, when working with the ELK Stack and the Beats family of shippers, especially, we learn by error.

To save you time (and aggravation), here are a few guidelines on writing your YAML configuration files.

Don’t Reinvent the Wheel

There is no need to start from scratch. Default configuration settings are supplied in most, if not all, of the log shippers in the Beats family. Use these settings as your initial reference. In most cases, you will only need to comment or uncomment specific lines.

Some Beats, such as Filebeat, include full example configuration files (/etc/filebeat/filebeat.full.yml). These files include long lists all the available configuration options. Copy the content of these files into a text editor, carefully remove the unnecessary pieces, and paste the content back in the file after validating the formatting (more about validation later).

Spaces vs. Tabs

How the lines are indented is critical in your YAML file, and in the timeless “spaces vs. tabs” debate, YAML creators decided that tabs are forbidden:

“Tabs have been outlawed since they are treated differently by different editors and tools. And since indentation is so critical to proper interpretation of YAML, this issue is just too tricky to even attempt.”

So, rule number one: do not use tabs when indenting your lines — only spaces. Otherwise, you will be reliving the scenario described above time and again.

You can use a code editor to verify if there are any abnormal whitespace characters in the configuration file. In Sublime, for example, simply paste the YAML into the editor and highlight the lines — you will see dots for every space before the line and blank spaces before the line for other whitespace characters. This will help you determine which lines have incorrect characters.

incorrect yaml characters

Preserve a Unified Structure

YAML configuration files for Beats are mostly built the same way, using two spaces for indentation.

Sections are separated by a long line of # (hash) symbols — as much as this sounds funny — don’t mistake these for commented out configuration lines.

The - (dash) character is used for defining new elements. Be sure to preserve their indentations and the hierarchies between sub-constructs.

Validate, Validate, and Then Validate Again

The times in which we needed to pay for text editors has long gone. We now have a wide set of free, online tools that will help you validate your YAML formatting. I recommend YAML Lint, but there are plenty more parsing assistants out there.

Once validated, you can also run a configuration test. You can use Filebeat, for example, once more (you’ll need to run this command from the directory in which the binary is installed): sudo filebeat -c filebeat.yml -configtest.

If there are any additional errors, you will be notified:

yaml lint


It ain’t rocket science, but a small and simple mistake can make all the difference between a bad day and an even worse day.

If you are using Logz.io, we have made it easy to configure a Filebeat YAML with a dedicated wizard. Even if you are not a Logz.io user, it can come in handy. Go through the wizard and simply remove the fields specific to Logz.io.

Happy YAMLing!

SignalFx is built on a massively scalable streaming architecture that applies advanced predictive analytics for real-time problem detection. With its NoSample™ distributed tracing capabilities, SignalFx reliably monitors all transactions across microservices, accurately identifying all anomalies. And through data-science-powered directed troubleshooting SignalFx guides the operator to find the root cause of issues in seconds.

yaml ,configuration ,elasticsearch ,performance

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}