apache lucene is the leading open source search engine and is used in many businesses, projects and products. lucene has sub-projects which provide additional functionality such as the nutch web crawler and the solr search service. this article gives an introduction to lucene, a tutorial on three grails lucene plugins and a comparison between them.

this article originally appeared in the september 2011 edition of groovymag .

lucene

apache lucene core provides a java-based indexing and search implementation. in essence, the index is made up of documents that are comprised of fields.

indexing

when you index a document, the fields are processed through a tokenizer to break it into terms. figure 1 shows the effect a whitespace tokenizer would have on splitting “mary had a little lamb” into the tokens: mary, had, a, little, lamb.

figure 1: whitespace tokenization

the terms may undergo further analysis through tokenfilter classes. figure 2 shows a stop words filter removing common terms that would skew relevancy.

figure 2: stopwords

figure 3 shows the porterstemfilter applying the (english) porter stemming algorithm to reduce tokens to their word stems so that they are equated. this tokenfilter needs to work on lower case input – so the lowercasefilter / tokenizer should be used before the stemming.

figure 3: word stemming

lastly, the terms are then mapped to their documents as shown in figure 4.

figure 4: term index

index updates

updates to index documents are handled as a delete operation followed by an add operation. over time the index segments can become fragmented – this can be cured by running an optimization operation to pack the index.

querying

the queries in lucene need to pass through the same analyzers as were used during indexing – otherwise identical terms might not match.
a single word query (termquery) requires a lookup in the term index to return the matching documents.

e.g. querying the term index shown in figure 4 for ‘consignment’ would return documents 1, 4 and 7.

a two word query (booleanquery) requires lucene to perform two lookups in the term index then filter the resulting documents on either an and or an or basis (from an explicit or default operator). e.g. querying the figure 4 term index for ‘consign and ship’ would return document 4, whereas ‘consign or ship’ would return documents 1, 4 and 7.

a phrase query is denoted by double quotes (e.g. “new york”) and matches documents containing a particular sequence of terms. this relies on positional information in the index. phrase slop, or proximity, is specified using the ~ operator and an integer to specify how close the terms in the phrase need to be together. e.g. “big banana”~5 would match documents containing “big banana”, “big green banana” and “big straight yellow banana”

by default results are returned in relevancy order, and the score is calculated by a formula (if you’re really interested it is in the javadoc for the similarity class – http://lucene.apache.org/java/3_3_0/api/core/org/apache/lucene/search/similarity.html ). the score can be influenced by boosting terms. e.g. the query ‘subject:lucene or author:bramley^2′ would boost the score contribution of the author field two-fold for documents whose author fields contained ‘bramley’.

tools

before we get onto the practical application it is worth mentioning that luke, the lucene index toolbox, is an invaluable tool for allowing inspection, searching and browsing of lucene indices.

it is available from http://code.google.com/p/luke/ – but be aware that you need to use the right version to match the version of lucene that created your indices. this has been made easier and more obvious as the version numbers of luke now correspond directly with lucene versions (e.g. luke 3.3.0 is for lucene 3.3.0, however luke 0.9.9.1 is based on lucene 2.9.1).

implementing in grails

rather than a twitter-style application, we’ll build a simple to-do list application with one domain class to represent the to-do item, shown in listing 1, which will use a generated controller and views (you may choose to enable scaffolding instead).

we’ll use this along with the figure 5 test data in 3 similar applications to showcase the different lucene-related grails plugins. all the sample applications are available on github under https://github.com/rbramley/groovymaglucene

package com.rbramley.todo
 
class item {
    date datecreated
    string subject
    date startdate
    date duedate
    string status
    string priority
    boolean completed = false
    int percentcomplete = 0
    string body
 
    static constraints = {
        subject(blank:false, nullable:false)
        startdate()
        duedate()
        status(inlist:["not started","in progress","completed","waiting on someone else","deferred"])
        priority(inlist:["low","normal","high"])
        completed()
        percentcomplete(range:0..100)
        body(nullable:true, maxsize:1000)
    }
}

figure 5: test data

searchable plugin

the searchable plugin makes use of compass::gps to index hibernate domain objects using gorm/hibernate lifecycle events. the plugin also provides a default search controller and view. the aim, which lives up to marc palmer’s recommendations, is to make full text searching of your domain objects as simple as possible.

the first step is to install the plugin using grails install-plugin searchable.

once we’ve created the domain class (grails create-domain-class com.rbramley.todo.item) and populated it with the code from listing 1, we then add a static searchable property to the domain class: static searchable = [except: 'datecreated']

running the application, select the com.rbramley.todo.itemcontroller link from the home page, and enter the test data from figure 5.

return to the home page and then select the grails.plugin.searchable.searchablecontroller link.

enter the query ‘timesheet’ in the search box then click the ‘search’ button – this will give you the results as shown in figure 6.

figure 6: searchable results

there you have it – quick and easy full text search on your domain model. whilst the style may not match with our application, at least it gives us something to work from.

note that the plugin stores the indices under ~/.grails/projects/ <project-name> /searchable-index/ <environment>

solr plugin

version 0.2 of the plugin was released in january 2010, is compatible with grails 1.1+ and bundles an old 1.4 release of solr and solrj. it would be nice to see a new version of this plugin with a current solr release and updated for the newer dependency resolution mechanisms – if you’ve got time to help out you can fork it on github at http://github.com/mbrevoort/grails-solr-plugin

once you’ve installed the plugin using ‘grails install-plugin solr‘, the ‘grails start-solr‘ script will start up a default solr instance using jetty. this is located in ‘solr-home‘ under the project working directory (based on the grails build settings) using the example solr schema and configuration. whilst this example configuration is fine for evaluation, i (and lucid imagination) would strongly recommend you don’t use this configuration in production. for instance, you’ll need to modify this if you want to use the dismax handler (at which point you’re probably better off with a separate installation and version controlled configuration).

the plugin makes good use of convention (leveraging solr dynamic fields) and meta-programming to add methods to domain classes.

how does it do auto-indexing?

if you check out the plugin dowithapplicationcontext closure, you will see that a listener is registered for the post- insert/update/delete events.

what about search?

this is handled by the solrservice which uses the solrj client and constructs a simple query (here it would be nice to leverage dismax). however the plugin author (mike brevoort) has also helpfully provided the ability to construct your own advanced query and supply that to the solrservice.

now we’ve uncovered how it works out of the box – it’s time to write some code so we can take it for a test drive.

the plugin can be installed by grails install-plugin solr.

again we’ll use the domain class from listing 1, but this time adding the two properties, static enablesolrsearch = true and static solrautoindex = true

once the domain class is completed we can generate the controllers & views:

grails generate-all com.rbramley.todo.item

we’ll add a search box to the main.gsp layout. this is shown in listing 2 and styling has been left as an exercise for the reader (note you can supply an image for the search button). this submits to the search controller (created using grails create-controller com.rbramley.todo.search – and populated by listing 3) with the search.gsp displaying the results (listing 4 shows a fragment).

<div>
     <strong>quick search</strong>
    <g:form url='[controller: "search", action: "search"]' id="searchform" name="searchform" method="get">
         <input type="text" name="query" value="${query ?: 'keywords'}" onclick="javascript:if (this.value=='keywords') { this.value='' }">
        <input type="image" src="${resource(dir:'images',file:'go_quick_search.png')}">
    </g:form>
</div>

listing 2: main.gsp search box

package com.rbramley.todo
 
class searchcontroller {
    def solrservice
 
    def search = {
        def res = solrservice.search("${params.query}")
        [query:params.query, total:res.total, searchresults:res.resultlist]
    }
}

listing 3: search controller code

<g:if test="${haveresults}">
    <g:each var="result" in="${searchresults}">
        <g:set var="classname" value="${classutils.getshortname(result.doctype_s)}" />
        <li><solr:resultlink result="${result}">${classname}: ${result.id}</solr:resultlink></li>
        ${result.subject?.encodeashtml()}
    </g:each>
</g:if>

listing 4: results page fragment

before you start the grails application, you need to start solr using grails start-solr.

once solr and then grails have started, we can create an item using the figure 5 test data, then search for it using ‘timesheet’.

why are there no results?

the plugin doc states that it will search against the default solr field (which by default is called ‘text’).

trying again: subject_s:timesheet also returns no results…

so let’s try a phrase query: subject_s:”complete timesheet” as shown in figure 7 – this one works because we’ve supplied the whole string to match, this is due to
“the strfield type is not analyzed, but indexed/stored verbatim.”

figure 7: solr with fully specified query

figure 8 shows the result of inspecting the subject field using solr’s luke request handler ( http://localhost:8983/solr/admin/luke ).

figure 8: solr luke analysis of subject field

so to get the desired results we’ll need to do some solr configuration, and we have the following options:

1. map the fields to text type so that they are analyzed (can also be forced to text through annotations) and use copy field directives to copy them to the default text field
2. configure a dismax handler (beyond the scope of this article) to have more flexible control over the fields that are queried by default

if we change the solr schema, we’ll need to do a full re-index – or in our case with a test application in development mode, we can just delete the index files (typically in ~/.grails/ <grails-version> /projects/ groovymagsolr /solr-home/solr/data).

first we’ll modify item.groovy to add import org.grails.solr.solr and then add the solr annotation to the string subject field e.g. @solr(astext=true) – this will cause the field to be indexed as subject_t. we’d now be able to search for subject_t:timesheet and get a match – but this still doesn’t meet our usability requirements as it requires knowledge of the underlying document fields. we could use an explicit <copyfield source="subject_t" dest="text"/> in the solr schema.xml, however if you inspect the schema.xml you will see that the first 3000 characters of all ‘_t’ fields are copied to the ‘text’ field.

now a search for ‘timesheet’ gives the results shown in figure 9.

figure 9: solr simple query result

elasticsearch plugin

elasticsearch ( http://www.elasticsearch.org/ ) is a new lucene-based distributed, restful search engine. it was created by shay banon, who created compass (used by the searchable plugin) and has worked on data grid technologies.

version 0.2 of the grails plugin uses elasticsearch 0.15.2 – you can start with an embedded instance in development mode which stores the indices in the project source directory under ‘data’.

the first step (within a clean application) is to install the plugin:

grails install-plugin elasticsearch

then create the domain class (grails create-domain-class com.rbramley.todo.item) and fill in using the listing 1 domain class code with the addition of listing 5.

 static searchable = {
        except = 'datecreated'
    }

listing 5: elasticsearch mapping

once that is done and the controller and views generated (grails generate-all com.rbramley.todo.item), we can then create our search controller (grails create-controller com.rbramley.todo.search) and complete it using listing 6.

package com.rbramley.todo
 
class searchcontroller {
    def elasticsearchservice
 
    def search = {
        def res = elasticsearchservice.search("${params.query}")
        [query:params.query, total:res.total, searchresults:res.searchresults]
    }
}

listing 6: elasticsearch controller

let’s re-use the main.gsp from the groovymagsolr application (listing 2) and we’ll adapt the search.gsp fragment from listing 4 to get listing 7.

<g:if test="${haveresults}">
    <g:each var="result" in="${searchresults}">
        <g:set var="classname" value="${classutils.getshortname(result.class)}" />
        <g:set var="link" value="${createlink(controller: classname[0].tolowercase() + classname[1..-1], action: 'show', id: result.id)}" />
        <li><a href="${link}">${classname}: ${result.id}</a></li>
        ${result.subject?.encodeashtml()}
    </g:each>
</g:if>

listing 7: search results page for elasticsearch

having started up the application and entered the test data as shown in figure 5, i hit a problem in that the database record was created but the index record wasn’t and the console was filling up with repeated log entries until i stopped the application:

2011-08-20 22:33:51,630 [elasticsearch[index]-pool-2-thread-1] error index.index
requestqueue - failed bulk item: nullpointerexception[null]
2011-08-20 22:33:51,633 [elasticsearch[index]-pool-2-thread-2] error index.index
requestqueue - failed bulk item: nullpointerexception[null]

when i looked at the indexrequestqueue source, the javadoc says “if indexing fails, all failed objects are retried. still no support for max number of retries (todo)”.

the nullpointerexception itself is elasticsearch bug 795 – but the underlying cause seems to be that the json document didn’t meet the expectations of elasticsearch for that type!

i got the plugin working by changing the domain class searchable mapping to only = 'subject' – then the results look identical to figure 9.

in the time available i didn’t manage to track down the initial issue (i also tried with an external elasticsearch instance and upgrading the elasticsearch dependencies to 0.16). i’ll be in communication with the plugin authors…

plugin comparison

so how do they compare? well this article has given a basic introduction to their usage for english text and hasn’t demonstrated advanced features or the distributed capabilities of the latter two.

the criteria for comparison is partially inspired by marc palmer’s views on plugins – a distilled form is “make it work, make it simple, make it magic”; we’ll also add scalability to this list.

searchable

this is a well established plugin and works very well out of the box with simple domain classes and even relationships. it is simple to use, works as expected and provides a basic search page and controller which includes some administrative actions such as the ability to re-index all searchable domain classes.

i’ve occasionally encountered older versions throwing exceptions on start-up that could be rectified by removing the indices and restarting the application.

on the downside, due to the embedded nature of the indices, it is only really suitable for single-instance applications.

solr

this plugin requires some configuration to get the best out of it and it could benefit from some attention to update it. however, it does have reasonable documentation and some powerful features such as faceted-search, spatial search and taglibs for facets and result links.

also it should be possible to index domain classes that use mongo through the use of the metaclass added indexsolr() method.

elasticsearch

this plugin has great promise – although i encountered some issues relating to the elasticsearch expectation of the index document structure, i should mention that the plugin documentation currently contains the warning “you should only use this plugin for testing” .

the main attraction of elasticsearch is the real time search with a distributed and scalable nature.

as per the solr plugin, it should also be possible to index mongo-mapped domain classes using a metaclass added index() method.

references

for further reading the dzone refcardz provide good overviews:

• http://refcardz.dzone.com/refcardz/lucene
• http://refcardz.dzone.com/refcardz/solr-essentials