Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Using Guava's Multimap to Improve Solr's Autocomplete Suggester

DZone's Guide to

Using Guava's Multimap to Improve Solr's Autocomplete Suggester

· Java Zone
Free Resource

Build vs Buy a Data Quality Solution: Which is Best for You? Gain insights on a hybrid approach. Download white paper now!

Context-less, multi-term autocomplete is difficult.

Given the term "di", we can look at our index and rank terms starting with "di" by frequency and return the n most frequent terms. Solr's TSTLookup and FSTLookup do this very well.

However, given the term "walt di", we can no longer do what we did above for each term and not look silly, especially if the corpus in question is a list of US companies (hint: think mickey mouse". There's little excuse to suggesting "walt discovery" or "walt diners" when our corpus does not contain any documents with that combination of terms.

In the absence of a large number of historical user queries to augment the autocomplete, context is king when it comes to multi-term queries.

The simplest way I can think of doing this, if it is feasible to do so memory-wise, is to store a list of terms and the term that immediately follows it. For example, given the field value "international business machines", mappings would be created for

international=>business
business=>machines

Out-of-order queries wouldn't be supported with this system, nor would term skips (e.g. international machines).

Here's a method fragment that does just this:

HashMultimap<String, String> map = HashMultimap.create();
for (int i = 0; i < reader.numDocs(); ++i) {
  Fieldable fieldable = reader.document(i).getFieldable(field);
  if(fieldable == null) continue;
  String fieldVal = fieldable.stringValue();
  if(fieldVal == null) continue;
  TokenStream ts = a.tokenStream(field, new StringReader(fieldVal));
  String prev = null;
  while (ts.incrementToken()) {
    CharTermAttribute attr = ts.getAttribute(CharTermAttribute.class);
    String v = new String(attr.buffer(), 0, attr.length()).intern();
    if (prev != null) {
      map.get(prev).add(v);
    }
    prev = v;
  }
}

Guava's Multimap is perfect for this, and Solr already has a Guava dependency, so we might as well make full use of it.

Build vs Buy a Data Quality Solution: Which is Best for You? Maintaining high quality data is essential for operational efficiency, meaningful analytics and good long-term customer relationships. But, when dealing with multiple sources of data, data quality becomes complex, so you need to know when you should build a custom data quality tools effort over canned solutions. Download our whitepaper for more insights into a hybrid approach.

Topics:

Published at DZone with permission of Kelvin Tan. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}