Over a million developers have joined DZone.

Log4j JSONLayout and JLogIndexer

DZone's Guide to

Log4j JSONLayout and JLogIndexer

· Java Zone
Free Resource

Learn how to stop testing everything every sprint and only test the code you’ve changed. Brought to you by Parasoft.

Last year I was working on building a Solr based log indexing and search.  My idea was simple.  Write a custom appender, use XML Layout, parse the XML into a Document and use SolrJ to push to Solr. Very soon reality struck me :

1)  log4j XML Layout is not a good idea for our requirement.  I really dont want to spend time on parsing about 100 XMLs every second.  That is a lot of time even for a StAX based Woodstox.  So, I just extended log4j layout and over-rid a couple of methods to create a JSON Layout.

2)  Using Jackson Streaming API to parse the JSON string from the Layout opened my eyes.  There were too many external dependency libraries (jackson, solrj and their dependent jars) that I was adding on to the application which ultimately will all go to the host application.  That is totally insane.

So, I figured out that way to go is to use

1) log4j JMS appender to put JSON strings into

2) Active MQ running on a app server running Solr.

I am hoping that this should give us the following benefits :

1) No complications on the host application. (copying libraries, classloader issues with library version incompatibility etc)

2) Non-log4j based java applications and non-java applications could use one of the various connectivity options available with ActiveMQ to put messages into its queue.


3)  Extension and replacing the layout and message parsing.

4)  Solr Document construction and the interface (initial version will still be on SolrJ) can be customized.

Get the top tips for Java developers and best practices to overcome common challenges. Brought to you by Parasoft.


Published at DZone with permission of Arun Manivannan, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}