Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Lessons Learned While Writing Our New Logback Appender

DZone's Guide to

Lessons Learned While Writing Our New Logback Appender

The trials and tribulations of creating a production ready appender featuring persistance, robustness and scalability out the box.

· Java Zone
Free Resource

Microservices! They are everywhere, or at least, the term is. When should you use a microservice architecture? What factors should be considered when making that decision? Do the benefits outweigh the costs? Why is everyone so excited about them, anyway?  Brought to you in partnership with IBM.

logback appender


Here at Logz.io, we take logs seriously. (If not us, who will?)

We run mainly on Dockerized Java microservices, and until recently, we had another Docker container that collected logs from our microservices.

Then, as part of a move to “self-contained” containers in preparation for another future move to a Docker orchestration solution — more on that in a future blog post! — we created a robust appender that integrates seamlessly into our current Logback setup.

Guess what? You can go here on GitHub to get it too!

Here are some key features:

  • Persistency. Each log line is saved on your local file system in a persistent queue (with a very fast input/output — we checked) using the superb bigqueue library. So, if your application crashes and reboots, you won’t lose a single log line.
  • Robustness. We will make sure that each log that you send will end up in your Logz.io account and return an acknowledgement from our log receivers.
  • Security. All logs sent are encrypted via HTTPS.
  • Light network load. We aggregate your logs into bulks and send them together.
  • Out-of-the-box enrichment. You don’t need to care about simple groking. The log line will retain the timestamp, logger, thread, log level, message, and any other additional fields and environment variables that you have requested.
  • Shading. We know that you love your libraries just the way they are. So, this appender is distributed as an “Uber Jar” that shades everything we need inside so you won’t have to worry about it.

We encourage Logback and Logz.io users to use the appender. (The log4j version is coming soon!)

The Lessons I Learned

When I sat down to write the appender, I encountered some caveats that I wanted to share because I could not find much documentation on writing new appenders at all.

So, here are some useful tips if you ever find yourself in the same situation.

Select the Proper Class to Implement

Your greatest source when starting your appender will be this Logback page. Almost all online searches for answers to questions on Logback appenders will end up there.

It’s a great resource on the options of the common appenders, but it lacks useful information when it come to writing your own.

The beginning states that all appenders must implement the  ch.qos.logback.core.Appender  interface. But if you don’t want to waste time coding unnecessary stuff, you can extend  ch.qos.logback.core.AppenderBase  (which does most of the dirty work for you) or extend any of the other appenders if your appender shares some of the functionality with them.

Don’t code things you don’t have to.

ILoggingEvent

When you start extending the AppenderBase class, you will notice that the one thing you basically need to do is to handle this:

abstractprotectedvoidappend(EeventObject);

The thing that is not documented clearly is that eventObject can come in the form of  ILoggingEvent , which allows you to access the members that you usually need right out of the box.

For example: 

@Override

protectedvoidappend(ILoggingEvent loggingEvent){

  Stringlevel=loggingEvent.getLevel().levelStr;

  StringloggerName=loggingEvent.getLoggerName();

  StringformattedMessage=loggingEvent.getFormattedMessage();

  Stringthread=loggingEvent.getThreadName();

Task Executer

The actual work of this appender is done outside of the “logback ecosystem” and the class that extends the AppenderBase just calls another generic class, so we didn’t want to use logback AsyncAppender and chose to implement our own threading mechanism instead. (AppenderBase calls another generic class so we can integrate more frameworks easily.)

I really love to use  ScheduledExecutorService from java.util.concurrent  instead of just implementing Runnable or any other third-party scheduling library, so I did.

Everything was just awesome until… the appender just did not stop. The main Java program finished its job, but the process was kept going on and on and on.

At first, we considered declaring that to be a feature — an appender that cares so much about logging that it just can’t stop sending them! (Even if there is nothing to send.)

But then we came to our senses and decided to read how Logback handles that issue in their own code. Here was the answer:  context.getScheduledExecutorService() 

You get an executor service from Logback that shuts itself down when the appender dies. That way, we can close ourselves gracefully (while implementing a “no log left behind” policy).

Which leads me to my next point.

Shutdown Hook

For the scheduled service to terminate, we needed to link the JVM termination hook with Logback. Oddly enough, that needed to be explicitly configured. Adding the following line under the <configuration> section of logback does exactly that:  <shutdownHook class="ch.qos.logback.core.hook.DelayingShutdownHook"/> 

Tests

Can’t stress that enough.

It’s obviously important to test all code. But I wanted to mention testing here because this is not a trivial case — the appender, after all, is not a part of our log receivers. I needed to mock it in testing to make sure that I was sending the proper things to it.

In this case, I wrote a Jetty-based mock Logz.io log receiver that accepted logs in the same format as we have and allowed me to assert on it directly based on the specific log line that was sent during the test.

I also added some real-life “features” such as the abilities to get timeouts while sending logs and deliberate exceptions from the server — resulting in connection being closed — so that we can make sure that no log goes to waste when shit hits the fan in real life.

What Do You Think?

So, if you ever need to write an appender yourself, I hope that you will find useful tips here.

And in case you use Logz.io and Logback, this is the appender for you!

As always, we encourage feedback. You can drop us an e-mail, open an issue on GitHub, or — even better — issue a pull request!

Discover how the Watson team is further developing SDKs in Java, Node.js, Python, iOS, and Android to access these services and make programming easy. Brought to you in partnership with IBM.

Topics:
jvm ,java ,orchestration ,log ,microservices ,docker ,logback

Published at DZone with permission of Rol Ravhon, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

THE DZONE NEWSLETTER

Dev Resources & Solutions Straight to Your Inbox

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

X

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}