Peeking into Lucene indexing

continuing my trip into the lucene codebase, now i’m looking into the process indexing as they are happening. interestingly enough, that is something that we never really had to look at before.

it is quite clear that lucene is heavily meant for utilizing additional threads for improving overall indexing speed. you can see it in the number of per thread state that exist all over the indexing code. that is also meant to reduce memory consumption, as far as i can see.

the real stuff happens in the processdocument() where we have a chain of docfieldconsumerperthread and termshashconsumerperthread which actually do the work.

then the real work is happening on a per field level in docinverterperfield, where the analyzer is actually called. the process in which the analyzers return values for the text is interesting. there are “attributes” that are added to the stream, per token, and they are used to get the relevant values. i assume that this allows to have different levels of analyzers.

this way, you can have analyzers that don’t deal with offesets or positions, etc. and the actual processing of this is done in a chain that appears to be:

• freq prox
• term vectors

but that isn’t something that i really care about now. i want to see how lucene writes the actual terms. this is being written in the termsinfowriter, which took some time to find.

terms are stored in a prefix compressed mode (sorted, obviously), and there is the actual terms, and an index into the terms, which allows for faster seeking into the file. this is actually done here:

this is a single term written to the file. a lot of the stuff lucene does (prefixes, vint, etc) are done in the name of conserving space, and it reminds me greatly of leveldb’s sst. in fact, the way terms are stored is pretty much an sst, except that this happens to be on multiple files. pretty much the entire behavior and all the implications are the same.

it also means that searching on this is fast, because the data is sorted, but pretty complex. and it also explains a lot about the actual exposed api that it has. i think that i have a pretty good idea on how things work now. i want to now go back up and look at specific parts of how it works… but that is going to be the next post.