Solr 4.0: Realtime GET

the next functionality i decided to look at, from the upcoming solr 4.0, is the so called “realtime get”. it allows you to see the data even though it was not yet added to the index, thus before the commit operation being sent to solr. let’s see how it works.

some theory

data update in lucene and solr has one disadvantage – when you submit index updates they can’t be seen until commit operation is run. the problem is that commit is costly in terms of performance and intense commiting may cause performance problems. so, when you need your data to be visible right after being change you may be forced to choose – either performance, or fast updates. in order to address that lucene and solr are working towards enabling near real time (nrt) searching. in lucene we have that possibility, in solr 4.0 we will also be able to use that and not only that.

configuration

in order to use realtime get functionality we need to configure the following solr features:

transaction log

the first thing to configure is the transaction log writing. in order to do that you need to add the following to your updatehandler configuration:

<updatelog>
  <str name="dir">${solr.data.dir:}</str>
</updatelog>

realtime get handler

the second thing that needs to be done, to see the realtime get in action, is the appropriate handler configuration (or adding component to your already defined handler). to do that add the following to your solrconfig.xml file :

<requesthandler name="/get" class="solr.realtimegethandler">
  <lst name="defaults">
    <str name="omitheader">true</str>
  </lst>
</requesthandler>

the above entry it’s nothing unusual – it just add a new request handler implementing solr.realtimegethandler class, which enables checking the transaction log.

action

to check how realtime get works i decided to do a simple test. the first thing i did is indexing one file (from the ones that are available in the exampledocs directory) with the use of the following bash command:

curl 'http://localhost:8983/solr/update' -d @hd.xml -h 'content-type:application/xml'

of course i did not send the commit operation after indexing. as we could expect the following query:

http://localhost:8983/solr/select?q=*:*

didn’t return search results. so let’s check, if the handler registered as /get will be able to get us some results. in order to do that i send the following query:

http://localhost:8983/solr/get?id=sp2514n

and in result i got the following document:

<?xml version="1.0" encoding="utf-8"?>
<response>
<doc name="doc">
  <str name="id">sp2514n</str>
  <str name="name">samsung spinpoint p120 sp2514n - hard drive - 250 gb - ata-133</str>
  <str name="manu">samsung electronics co. ltd.</str>
  <str name="manu_id_s">samsung</str>
  <arr name="cat">
    <str>electronics</str>
    <str>hard drive</str>
  </arr>
  <arr name="features">
    <str>7200rpm, 8mb cache, ide ultra ata-133</str>
    <str>noiseguard, silentseek technology, fluid dynamic bearing (fdb) motor</str>
  </arr>
  <float name="price">92.0</float>
  <int name="popularity">6</int>
  <bool name="instock">true</bool>
  <date name="manufacturedate_dt">2006-02-13t15:26:37z</date>
  <str name="store">35.0752,-97.032</str></doc>
</response>

so solr returned the result that wasn’t added to the index – nice !

usage possibilities

you probably noticed, that in order to fetch a document with /get handler i needed to provide it’s unique identifier (or identifiers list). that’s true, realtime get doesn’t support searching, because it was not created to support full searching. this functionality is able to show us the updates of the documents which identifiers are known (so for example the ones in the index) – in example by adding the component used in solr.realtimegethandler to any of your defined handler. and the good news is – you don’t have to worry update performance – solr.realtimeget is very fast. so, if one of your problems is frequent updated you can look in the future with a smile

last few words

the realtime get functionality brings new possibilities when it comes to solr and also on the road to the solrcloud. with the use of transaction log one can implement automatic cluster node restore or instance nrt instance updates. as you can see solr 4.0 is not only about search, but also about data store and bringing solr closer to nosql solutions.


source: http://solr.pl/en/2012/01/09/solr-4-0-realtime-get-2/