DZone
Java Zone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
  • Refcardz
  • Trend Reports
  • Webinars
  • Zones
  • |
    • Agile
    • AI
    • Big Data
    • Cloud
    • Database
    • DevOps
    • Integration
    • IoT
    • Java
    • Microservices
    • Open Source
    • Performance
    • Security
    • Web Dev
DZone > Java Zone > Custom tokenization and Lucene’s FastVectorHighlighter

Custom tokenization and Lucene’s FastVectorHighlighter

Itamar Syn-hershko user avatar by
Itamar Syn-hershko
·
Jun. 23, 12 · Java Zone · Interview
Like (1)
Save
Tweet
4.21K Views

Join the DZone community and get the full member experience.

Join For Free

Perhaps you have tackled this before: you wanted to use Lucene's FastVectorHighlighter (aka FVH), but since you have a custom CharFilter in your analysis chain, the highlighter fails to produce valid fragments.

In my particular case, I used HTMLStripCharFilter (available to Lucene.Net through my pet contrib project) to extract text content from HTML pages, and then pass it through the rest of the analysis process. This confused FVH, since it was taking the full content from store, where HTML was still present, and token positions were not taking that into account. And any other custom CharFilter that is added to the analysis chain is going to cause the same troubles.

To overcome this, I needed to make sure FVH is aware of all content stripping operations that are made before or while tokenization is happening. All I had to do was to implement a custom FragmentsBuilder, looking as follows (.Net code; a Java version would look almost identical):

public class HtmlFragmentsBuilder : BaseFragmentsBuilder
{
    /// <summary>
    /// a constructor.
    /// </summary>
    public HtmlFragmentsBuilder()
        : base()
    {
    }
 
    /// <summary>
    /// a constructor.
    /// </summary>
    /// <param name="preTags">array of pre-tags for markup terms</param>
    /// <param name="postTags">array of post-tags for markup terms</param>
    public HtmlFragmentsBuilder(String[] preTags, String[] postTags)
        : base(preTags, postTags)
    {
    }
 
    /// <summary>
    /// do nothing. return the source list.
    /// </summary>
    public override List<WeightedFragInfo> GetWeightedFragInfoList(List<WeightedFragInfo> src)
    {
        return src;
    }
 
    protected override String GetFragmentSource(StringBuilder buffer, int[] index, Field[] values, int startOffset, int endOffset)
    {
        string fieldText;
        while (buffer.Length < endOffset && index[0] < values.Length)
        {
            fieldText = GetFilteredFieldText(values[index[0]]);
            if (index[0] > 0 && values[index[0]].IsTokenized() && fieldText.Length > 0)
                buffer.Append(' ');
            buffer.Append(fieldText);
            ++(index[0]);
        }
        var eo = buffer.Length < endOffset ? buffer.Length : endOffset;
        return buffer.ToString().Substring(startOffset, eo - startOffset);
    }
 
    /// <summary>
    /// Gets the field text, after applying custom filtering
    /// </summary>
    /// <param name="field"></param>
    /// <returns></returns>
    protected string GetFilteredFieldText(Field field)
    {
        var theStream = new MemoryStream(Encoding.UTF8.GetBytes(field.StringValue()));
        var reader = CharReader.Get(new StreamReader(theStream));
        reader = new HTMLStripCharFilter(reader);
 
        int r;
        var sb = new StringBuilder();
        while ((r = reader.Read()) != -1)
        {
            sb.Append((char)r);
        }
        return sb.ToString();
    }
}

FVH will then need to be configured to use it:

var fvh = new FastVectorHighlighter(FastVectorHighlighter.DEFAULT_PHRASE_HIGHLIGHT,                                             FastVectorHighlighter.DEFAULT_FIELD_MATCH,
                    new SimpleFragListBuilder(), new HtmlFragmentsBuilder());
// ...
var fq = fvh.GetFieldQuery(query);
var fragment = fvh.GetBestFragment(fq, searcher.GetIndexReader(), hits[i].doc, "Content", 300);

 If you're using Lucene.Net, you'll have to make sure this patch is applied to your FVH before this could compile.

That was the easiest way to get this working, and fast. Perhaps I could make it more generic, or change the original implementation to allow that and submit it as a patch. Maybe I'll do it someday. Or you could...

Lucene

Published at DZone with permission of Itamar Syn-hershko, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • How to Test JavaScript Code in a Browser
  • What Is Lean Software Development
  • Building a Login Screen With React and Bootstrap
  • Message Queuing and the Database: Solving the Dual Write Problem

Comments

Java Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • MVB Program
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends:

DZone.com is powered by 

AnswerHub logo