Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Log4j JSONLayout and JLogIndexer

DZone's Guide to

Log4j JSONLayout and JLogIndexer

· Java Zone
Free Resource

The single app analytics solutions to take your web and mobile apps to the next level.  Try today!  Brought to you in partnership with CA Technologies

Last year I was working on building a Solr based log indexing and search.  My idea was simple.  Write a custom appender, use XML Layout, parse the XML into a Document and use SolrJ to push to Solr. Very soon reality struck me :

1)  log4j XML Layout is not a good idea for our requirement.  I really dont want to spend time on parsing about 100 XMLs every second.  That is a lot of time even for a StAX based Woodstox.  So, I just extended log4j layout and over-rid a couple of methods to create a JSON Layout.

2)  Using Jackson Streaming API to parse the JSON string from the Layout opened my eyes.  There were too many external dependency libraries (jackson, solrj and their dependent jars) that I was adding on to the application which ultimately will all go to the host application.  That is totally insane.

So, I figured out that way to go is to use

1) log4j JMS appender to put JSON strings into

2) Active MQ running on a app server running Solr.

I am hoping that this should give us the following benefits :

1) No complications on the host application. (copying libraries, classloader issues with library version incompatibility etc)

2) Non-log4j based java applications and non-java applications could use one of the various connectivity options available with ActiveMQ to put messages into its queue.

http://activemq.apache.org/connectivity.html

3)  Extension and replacing the layout and message parsing.

4)  Solr Document construction and the interface (initial version will still be on SolrJ) can be customized.

CA App Experience Analytics, a whole new level of visibility. Learn more. Brought to you in partnership with CA Technologies.

Topics:

Published at DZone with permission of Arun Manivannan, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

THE DZONE NEWSLETTER

Dev Resources & Solutions Straight to Your Inbox

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

X

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}