DZone
Java Zone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
  • Refcardz
  • Trend Reports
  • Webinars
  • Zones
  • |
    • Agile
    • AI
    • Big Data
    • Cloud
    • Database
    • DevOps
    • Integration
    • IoT
    • Java
    • Microservices
    • Open Source
    • Performance
    • Security
    • Web Dev
DZone > Java Zone > Feeding Solr With its own Logs

Feeding Solr With its own Logs

Peter Karussell user avatar by
Peter Karussell
·
Oct. 28, 10 · Java Zone · Interview
Like (0)
Save
Tweet
8.53K Views

Join the DZone community and get the full member experience.

Join For Free

i always looked for a simple way to visualize our log data e.g. from solr. at that time i had in mind a combination of gnuplot and some shellscripts but this session from the lucene revolution changed my idea. (look here for all videos from lucene revolution.)

i thought: “hey thats it! just put the logs into solr!” so i coded something which simply reads the log files and named it sogger. without sharding, without message queues, … but it should work on real systems without any changes to your system (but probably to sogger).

i hope sogger doesn’t suck, but it does not come with any warranty, so use it with care! and: it is only a proof of concept – nothing comparable to the guys from loggly.com

to get your logs sogged:

  • download the ‘sogger’ code via:
    hg clone http://timefinder.hg.sourceforge.net/hgroot/timefinder/sogger sogger-code
  • download the solr from trunk.
    svn co -r  1023329 https://svn.apache.org/repos/asf/lucene/dev/trunk solr-code

    sogger doesn’t necessarily need the trunk version but i didn’t tested it for others yet

  • compile solr and sogger with ant
  • cd solr-code/solr/example/
  • copy solrconfig.xml, schema.xml from sogger into solr/conf
  • copy the *.vm files from sogger into the files at solr/conf/velocity/
  • start solr
    java -jar start.jar
  • start feeding your logs
    cd sogger-code/
    java -jar dist/sogger.jar url=http://localhost:8983/solr logfile=data/solr.2010-10-25.log.gz
  • to search your logs do:

    http://localhost:8983/solr/browse?q=twitter

now you should see something like this

sogger has several advantages over simple “grep-ing” or scripting with your solr logs:

  • full text search. near real time: ~1min ;-)
  • performance. i hope commiting every minute does not make solr a lot slower
  • filtering by log level: quickly find warnings and exceptions
  • filtering by webapp: if you have multiple apps or solr cores which are logging into the same file filtering is really easy with solr (with grep too, but you’ll have to re-grep the whole log …)
  • open source: you can change the feeding method i used and take care of your special needs. tell me if you need assistance!
  • new log lines will be detected and commited ala tail -f
  • besides text files sogger accepts and detects compressed (zip, gzip/gz) files ala zgrep. so you don’t need to change your log handlers or preprocess the files.

to do’s:

  • make the log format customizable within a property file:
    line1=regular expression pattern1
    line2=regular expression pattern2
  • read and monitor multiple log files
  • make it a solr plugin via special updatehandler?
  • a xy plot (or barchart) in velocity for some facets or facet queries would be nice. something like i had done before with wicket.
  • i don’t like velocity … althought it is sufficient for this … but should we use wicket!?

from http://karussell.wordpress.com/2010/10/27/feeding-solr-with-its-own-logs/

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • DZone's Article Submission Guidelines
  • How to Determine if Microservices Architecture Is Right for Your Business
  • Debugging Deadlocks and Race Conditions
  • Java: Why Core-to-Core Latency Matters

Comments

Java Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • MVB Program
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends:

DZone.com is powered by 

AnswerHub logo