Solr filters: PatternReplaceCharFilter

Continuing the overview of the filters included in Solr today we look at the PatternReplaceCharFilter.

As you might guess the task of the filter is to change the matching input stream parts that match the given regular expression.

You have the following parameters:

• pattern (required) – the value to be changed (regular expressions)
• replacement (default: “”) – the value that will be used as a replament for the fragment that matched the regular expression
• blockDelimiters
• maxBlockChars (default: 10000, must be greater than 0) – buffer used for comparison
  

Use examples

The use of a filter is simple – we add its definition to the type definition in schema.xml file, for example:

<fieldType name="textCharNorm" class="solr.TextField">
  <analyzer>
    <charFilter class="solr.PatternReplaceCharFilterFactory" …/>
    <charFilter class="solr.MappingCharFilterFactory" mapping="mapping-ISOLatin1Accent.txt"/>
    <tokenizer class="solr.WhitespaceTokenizerFactory"/>
  </analyzer>
</fieldType>

Below are examples of definitions for different cases.

Cut pieces of text

You just need to specify, in the pattern attribute, what we want to cut. Example:

Cut pieces of text

You just need to specify, in the pattern attribute, what we want to cut. Example:

<charFilter class="solr.PatternReplaceCharFilterFactory" pattern="#TAG" />

which will suppress the content of the data elements: “#TAG”

Text fragments replacement

A similar case to the one above, but we want to convert text to another.

<charFilter class="solr.PatternReplaceCharFilterFactory" pattern="#TAG" replacement="[CENZORED]"/>

Changing patterns

The two above cases were trivial. What is the strength of this filter is handling regular expressions. (You use regular expressions, right?) The following example is simple – it hides all the numbers by turning them into stars. It also handles the numbers separated by hyphens, treating them as a single number.

<charFilter class="solr.PatternReplaceCharFilterFactory" pattern="(\d+-*\d+)+" replacement="*"/>

Text Manipulation

The replacement doesn’t have to be plain text. This filter supports references which allow you to refer to parts of the matched pattern. For details, refer to the documentation of regular expressions. In the following example, all multiplied characters are replaced with a single sign.

<charFilter class="solr.PatternReplaceCharFilterFactory" pattern="(.)\1" replacement="$1"/>

Advanced Parameters

So far I have not mentioned the following parameters: blockDelimiters and maxBlockChars. If you look at the source code you would see that those parameters are related to the way the filter is implemented. CharFilter operates on a single character, and pattern matching requires an internal buffer to read more characters. MaxBlockChars allows you to specify the size of the buffer. You do not have to worry about it, if the pattern you defined, does not match piece of text larger than 10k characters). BlockDelimiters can further optimize filling of the buffer. It can be used if the information in the analyzed field is somehow divided into sections (eg, it is a CSV, sentences, etc.). It is a text that informs the scanner, that a new section starts, therefore, parts matched in the previous section are no longer useful.

Limits

An important limitation of the filter is that it directly manipulates the input data and does not keep information related to the original text. This means that if the filter removes a portion of the string, or add a new fragment, tokenizer will not notice that and the location of tokens in the original box will not be saved properly. You should be aware of that when using queries that operate on the relative positions of tokens or if you use highlighting.