Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Debugging Hints for Logstash

DZone's Guide to

Debugging Hints for Logstash

In order to know a tool’s every nook and cranny, one needs to use and re-use it, be creative, and search on Google… a lot.

· Java Zone
Free Resource

Bitbucket is for the code that takes us to Mars, decodes the human genome, or drives your next car. What will your code do? Get started with Bitbucket today, it's free.

As a Java developer, when you are first shown how to run the JVM in debug mode, attach to it and then set a breakpoint, you really feel like you’ve reached a step on your developer journey. Well, at least I did. Now the world is going full microservice, and knowing that trick means less and less in it every day.

This week, I was playing with Logstash to see how I could send all of an application's exceptions to an Elasticsearch instance, so I could display them on a Kibana dashboard for analytics purpose. Of course, nothing was seen in Elasticsearch at first. This post describes what helped me toward making it work in the end.

The Setup

Components are the following:

  • The application. Since a lot of exceptions were necessary, I made use of the Java Bullshifier. The only adaptation was to wire in some code to log exceptions in a log file.
    public class ExceptionHandlerExecutor extends ThreadPoolExecutor {
    
        private static final Logger LOGGER = LoggerFactory.getLogger(ExceptionHandlerExecutor.class);
    
        public ExceptionHandlerExecutor(int corePoolSize, int maximumPoolSize,
                long keepAliveTime, TimeUnit unit, BlockingQueue<Runnable> workQueue) {
            super(corePoolSize, maximumPoolSize, keepAliveTime, unit, workQueue);
        }
    
        @Override
        protected void afterExecute(Runnable r, Throwable t) {
            if (r instanceof FutureTask) {
                FutureTask<Exception> futureTask = (FutureTask<Exception>) r;
                if (futureTask.isDone()) {
                    try {
                        futureTask.get();
                    } catch (InterruptedException | ExecutionException e) {
                        LOGGER.error("Uncaught error", e);
                    }
                }
            }
        }
    }
  • Logstash
  • Elasticsearch
  • Kibana

Image title

The First Bump

Before being launched, Logstash needs to be configured — especially its input and its output.

There’s no out-of-the-box input focused on exception stack traces. Those are multi-lines messages: hence, only lines starting with a timestamp mark the beginning of a new message. Logstash achieves that with a specific codec including a regex pattern.

Some people, when confronted with a problem, think "I know, I'll use regular expressions."
Now they have two problems.  — Jamie Zawinski

Some are better than others at regular expressions, but nobody learned it as his/her mother tongue. Hence, it’s not rare to have errors. In that case, one should use one of the many available online regex validators. They are just priceless for understanding why some pattern doesn’t match.

The relevant Logstash configuration snippet becomes:

input {
  file {
    path => "/tmp/logback.log"
    start_position => "beginning"
    codec => multiline {
      pattern => "^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}"
      negate => true
      what => "previous"
    }
  }
}

The Second Bump

Now, data found its way into Elasticsearch, but message were not in the expected format. In order to analyze where this problem came from, messages can be printed on the console instead of indexed in Elasticsearch.

That’s quite easy with the following snippet:

output {
  stdout { codec => rubydebug }
}

With messages printed on the console, it’s possible to understand where the issue occurs. In that case, I was able to tweak the input configuration (and add the forgotten negate => true bit).

Finally, I got the expected result:

Image title

Conclusion

With more and more tools with every passing day, the toolbelt of the modern developer needs to increase as well. Unfortunately, there’s no one-size-fits-all solution: in order to know a tool’s every nook and cranny, one needs to use and re-use it, be creative, and search on Google… a lot.

Bitbucket is the Git solution for professional teams who code with a purpose, not just as a hobby. Get started today, it's free.

Topics:
elk stack ,java ,logging ,debugging

Published at DZone with permission of Nicolas Frankel, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

The best of DZone straight to your inbox.

SEE AN EXAMPLE
Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.
Subscribe

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}