Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Using Guava's Multimap to Improve Solr's Autocomplete Suggester

DZone's Guide to

Using Guava's Multimap to Improve Solr's Autocomplete Suggester

· Java Zone
Free Resource

Check out this 8-step guide to see how you can increase your productivity by skipping slow application redeploys and by implementing application profiling, as you code! Brought to you in partnership with ZeroTurnaround.

Context-less, multi-term autocomplete is difficult.

Given the term "di", we can look at our index and rank terms starting with "di" by frequency and return the n most frequent terms. Solr's TSTLookup and FSTLookup do this very well.

However, given the term "walt di", we can no longer do what we did above for each term and not look silly, especially if the corpus in question is a list of US companies (hint: think mickey mouse". There's little excuse to suggesting "walt discovery" or "walt diners" when our corpus does not contain any documents with that combination of terms.

In the absence of a large number of historical user queries to augment the autocomplete, context is king when it comes to multi-term queries.

The simplest way I can think of doing this, if it is feasible to do so memory-wise, is to store a list of terms and the term that immediately follows it. For example, given the field value "international business machines", mappings would be created for

international=>business
business=>machines

Out-of-order queries wouldn't be supported with this system, nor would term skips (e.g. international machines).

Here's a method fragment that does just this:

HashMultimap<String, String> map = HashMultimap.create();
for (int i = 0; i < reader.numDocs(); ++i) {
  Fieldable fieldable = reader.document(i).getFieldable(field);
  if(fieldable == null) continue;
  String fieldVal = fieldable.stringValue();
  if(fieldVal == null) continue;
  TokenStream ts = a.tokenStream(field, new StringReader(fieldVal));
  String prev = null;
  while (ts.incrementToken()) {
    CharTermAttribute attr = ts.getAttribute(CharTermAttribute.class);
    String v = new String(attr.buffer(), 0, attr.length()).intern();
    if (prev != null) {
      map.get(prev).add(v);
    }
    prev = v;
  }
}

Guava's Multimap is perfect for this, and Solr already has a Guava dependency, so we might as well make full use of it.

The Java Zone is brought to you in partnership with ZeroTurnaround. Check out this 8-step guide to see how you can increase your productivity by skipping slow application redeploys and by implementing application profiling, as you code!

Topics:

Published at DZone with permission of Kelvin Tan. See the original article here.

Opinions expressed by DZone contributors are their own.

The best of DZone straight to your inbox.

SEE AN EXAMPLE
Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.
Subscribe

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}