Log4j JSONLayout and JLogIndexer

Last year I was working on building a Solr based log indexing and search.  My idea was simple.  Write a custom appender, use XML Layout, parse the XML into a Document and use SolrJ to push to Solr. Very soon reality struck me :

1)  log4j XML Layout is not a good idea for our requirement.  I really dont want to spend time on parsing about 100 XMLs every second.  That is a lot of time even for a StAX based Woodstox.  So, I just extended log4j layout and over-rid a couple of methods to create a JSON Layout.

2)  Using Jackson Streaming API to parse the JSON string from the Layout opened my eyes.  There were too many external dependency libraries (jackson, solrj and their dependent jars) that I was adding on to the application which ultimately will all go to the host application.  That is totally insane.

So, I figured out that way to go is to use

1) log4j JMS appender to put JSON strings into

2) Active MQ running on a app server running Solr.

I am hoping that this should give us the following benefits :

1) No complications on the host application. (copying libraries, classloader issues with library version incompatibility etc)

2) Non-log4j based java applications and non-java applications could use one of the various connectivity options available with ActiveMQ to put messages into its queue.

http://activemq.apache.org/connectivity.html

3)  Extension and replacing the layout and message parsing.

4)  Solr Document construction and the interface (initial version will still be on SolrJ) can be customized.