Developing Your Own Solr Filter - Part 2

public class reversefilterfactory extends tokenfilterfactory {
 @override
 public tokenstream create(tokenstream ts) {
  return new reversefilter(ts);
 }
}

public final class reversefilter extends tokenfilter {
 private chartermattribute chartermattr;
 private positionincrementattribute posincattr;
 private queue<char[]> terms;

 protected reversefilter(tokenstream ts) {
  super(ts);
  this.chartermattr = addattribute(chartermattribute.class);
  this.posincattr = addattribute(positionincrementattribute.class);
  this.terms = new linkedlist<char[]>();
 }

 @override
 public boolean incrementtoken() throws ioexception {
  if (!terms.isempty()) {
   char[] buffer = terms.poll();
   chartermattr.setempty();
   chartermattr.copybuffer(buffer, 0, buffer.length);
   posincattr.setpositionincrement(1);
   return true;
  }

  if (!input.incrementtoken()) {
   return false;
  } else {
   // we reverse the token
   int length = chartermattr.length();
   char[] buffer = chartermattr.buffer();
   char[] newbuffer = new char[length];
   for (int i = 0; i < length; i++) {
    newbuffer[i] = buffer[length - 1 - i];
   }
   // we place the new buffer in the terms list
   terms.add(newbuffer);
   // we return true and leave the original token unchanged
   return true;
  }
 }
}

implementation description

let’s talk about the differences between the above filter and the version shown in the previous blog post:

• line 4 – a list, that will be used to hold term buffer that needs to be written to token stream.
• line 9 – we add the attribute that is responsible for setting the token position in the token stream.
• line 10 – initialization of the list we’ve defined in line 4.
• line 15 – 21 – condition that checks if we have tokens in the lists that we need to process. if there are such tokens we take the first token from the list (and we remove it), we set the term buffer, we set its position in the token stream abd we return true in order to inform that processing should be continued. it is worth noting that we didn’t fetch a new token from the token stream, because we didn’t call the input.incrementtoken() method.
• lines 23 – 25 – we check if there are tokens left for processing. if there are not we just return false and we end processing.
• lines 27 – 36 – we reverse the token and we add it to the list declared on line 4. we didn’t modify the token that is actually present in the token and we return true . by doing this we inform solr that we want to continue processing and solr should call the incrementtoken() method of our filter. after the next call we will end up executing the code from lines 15 – 21 because our list will contain a new token.

there is one more thing that is worth noting in our opinion – the position increment attribute. in the above implementation, each reversed token will be written in the next position in the token stream comparing to the original token. to put it simple – those tokens will be treated as single words. in a few we will check what will happen when the reversed tokens will be put on the same positions as the original ones.

does it work ?

the work of the above filter can be illustrated by using solr administration panel:

as you can see, both original values: ala , ma and kota and the reversed ones ala , am and atok were put on in a new positions (the position attribute). so it works as intended

let’s change the position increment

so let’s check what will happen when we change the following line:

to the following one:

how it works

again, let’s illustrate how it works by looking at solr administration panel:

as you can see, after the change, both original token as well as its reversed version were put on the same position, which is what we wanted to achieve. because of that we can now run queries like q=”ala am kota” (look at the am word). what we gain is the ability to use original or reversed tokens in the phrase queries.

to sum up

as you can see creating your own filters is not a rocket science, at least when it comes to lucene and solr part. what we get from lucene and solr is a nice set of features which we can use to control what will be finally put in the token stream, for example thanks to token stream attributes. of course the complexity of the code will depend on you business logic, but this is far beyond the scope of this post