Solr and Autocomplete (part 2)

in the previous part i showed how the faceting mechanism can be used to achieve the autocomplete functionality. today i’ll show you how to use a component called suggester to implement autocomplete functionality.

the begining

there is one thing that you must know – suggest component is not available in solr version 1.4.1 and below. to start using this component you need to download 3_x or trunk version from lucene/solr svn.

configuration

before we get into the index configuration we need to define an search component. so let’s do it:

<searchcomponent name="suggest" class="solr.spellcheckcomponent">
 <lst name="spellchecker">
  <str name="name">suggest</str>
  <str name="classname">org.apache.solr.spelling.suggest.suggester</str>
  <str name="lookupimpl">org.apache.solr.spelling.suggest.tst.tstlookup</str>
  <str name="field">name_autocomplete</str>
 </lst>
</searchcomponent>

it is worth mentioning that suggest component is based on solr.spellcheckcomponent and that’s why we can use the above configuration. we have three important attributes in the configuration:

• name - name of the component.
• lookupimpl – an object that will handle the search. at this point we have two possibilities to use – jasperlookup or tstlookup. this second one characterizes greater efficiency.
• field – the field on the basis of which suggestions are generated.

now let’s add the appropriate handler:

<requesthandler name="/suggest" class="org.apache.solr.handler.component.searchhandler">
 <lst name="defaults">
  <str name="spellcheck">true</str>
  <str name="spellcheck.dictionary">suggest</str>
  <str name="spellcheck.count">10</str>
 </lst>
 <arr name="components">
  <str>suggest</str>
 </arr>
</requesthandler>

quite simple configuration, which defines a handler with an additional search component and tell solr that the maximum number of suggestions returned is 10, this it should use dictionary named suggest (which is actually a suggest component) which is exactly the same as our defined component.

index

let us assume that our document consists of three fields: id, name and description. we want to generate suggestions on the field that hold the name of the product. our index could look like this:

<field name="id" type="string" indexed="true" stored="true" multivalued="false" required="true"/>
<field name="name" type="text" indexed="true" stored="true" multivalued="false" />
<field name="name_autocomplete" type="text_auto" indexed="true" stored="true" multivalued="false" />
<field name="description" type="text" indexed="true" stored="true" multivalued="false" />

in addition, there is the following copy field definition:

<copyfield source="name" dest="name_autocomplete" />

suggesting single words

in order to achieve individual words suggestions text_autocomplete type should be defined as follows:

<fieldtype class="solr.textfield" name="text_auto" positionincrementgap="100">
 <analyzer>
  <tokenizer class="solr.whitespacetokenizerfactory"/>
  <filter class="solr.worddelimiterfilterfactory" generatewordparts="1" generatenumberparts="1" catenatewords="1" catenatenumbers="1" catenateall="0" splitoncasechange="1"/>
  <filter class="solr.lowercasefilterfactory"/>
 </analyzer>
</fieldtype>

suggesting phrases

to implement the entire phrase suggestions our text_autocomplete type should be defined as follows:

<fieldtype class="solr.textfield" name="text_auto">
 <analyzer>
  <tokenizer class="solr.keywordtokenizerfactory"/>
  <filter class="solr.lowercasefilterfactory"/>
 </analyzer>
</fieldtype>

if you want to use phrases you may want to define your own query converter.

dictionary building

before we start using the component, we need to build its index. to this send the following command to solr:

/suggest?spellcheck.build=true

queries

now we come to use of the component. in order to show how the use the component, i decided suggest whole phrases. the example query could look like that:

/suggest?q=har

after running that query i got the following suggestions:

<?xml version="1.0" encoding="utf-8"?>
<response>
 <lst name="responseheader">
  <int name="status">0</int>
  <int name="qtime">0</int>
 </lst>
 <lst name="spellcheck">
  <lst name="suggestions">
   <lst name="dys">
    <int name="numfound">4</int>
    <int name="startoffset">0</int>
    <int name="endoffset">3</int>
    <arr name="suggestion">
     <str>hard drive</str>
     <str>hard drive samsung</str>
     <str>hard drive seagate</str>
     <str>hard drive toshiba</str>
    </arr>
   </lst>
  </lst>
 </lst>
</response>

the end

in the next part of the autocomplete functionality i’ll show how to modify its configuration to use static dictionary into the mechanism and how this can helk you get better suggestions. the last part of the series will be a performance comparison of each method in which i’ll try to diagnose which method is the fastest one in various situations.