Feeding Solr With its own Logs
Join the DZone community and get the full member experience.
Join For Freei always looked for a simple way to visualize our log data e.g. from solr. at that time i had in mind a combination of gnuplot and some shellscripts but this session from the lucene revolution changed my idea. (look here for all videos from lucene revolution.)
i thought: “hey thats it! just put the logs into solr!” so i coded something which simply reads the log files and named it sogger. without sharding, without message queues, … but it should work on real systems without any changes to your system (but probably to sogger).
i hope sogger doesn’t suck, but it does not come with any warranty, so use it with care! and: it is only a proof of concept – nothing comparable to the guys from loggly.com
to get your logs sogged:
-
download the ‘sogger’ code via:
hg clone http://timefinder.hg.sourceforge.net/hgroot/timefinder/sogger sogger-code
-
download the solr from trunk.
sogger doesn’t necessarily need the trunk version but i didn’t tested it for others yet
- compile solr and sogger with ant
- cd solr-code/solr/example/
- copy solrconfig.xml, schema.xml from sogger into solr/conf
- copy the *.vm files from sogger into the files at solr/conf/velocity/
-
start solr
java -jar start.jar -
start feeding your logs
cd sogger-code/ java -jar dist/sogger.jar url=http://localhost:8983/solr logfile=data/solr.2010-10-25.log.gz
-
-
to search your logs do:
http://localhost:8983/solr/browse?q=twitter
now you should see something like this
sogger has several advantages over simple “grep-ing” or scripting with your solr logs:
-
full text search. near real time: ~1min
- performance. i hope commiting every minute does not make solr a lot slower
- filtering by log level: quickly find warnings and exceptions
- filtering by webapp: if you have multiple apps or solr cores which are logging into the same file filtering is really easy with solr (with grep too, but you’ll have to re-grep the whole log …)
- open source: you can change the feeding method i used and take care of your special needs. tell me if you need assistance!
- new log lines will be detected and commited ala tail -f
- besides text files sogger accepts and detects compressed (zip, gzip/gz) files ala zgrep. so you don’t need to change your log handlers or preprocess the files.
to do’s:
-
make the log format customizable within a property file:
line1=regular expression pattern1
line2=regular expression pattern2 - read and monitor multiple log files
- make it a solr plugin via special updatehandler?
- a xy plot (or barchart) in velocity for some facets or facet queries would be nice. something like i had done before with wicket.
- i don’t like velocity … althought it is sufficient for this … but should we use wicket!?
from http://karussell.wordpress.com/2010/10/27/feeding-solr-with-its-own-logs/
Opinions expressed by DZone contributors are their own.
Comments