An Exploration Into Lucene Disk Format

i realized lately that i wanted to know a lot more about exactly how lucene is storing data on disk. oh, i know the general stuff about segments and files, etc. but i wanted to know the actual bits & bytes. so i started tracing into lucene and trying to figure out what it is doing.

and, by the way, the only thing that the lucene.net codebase is missing is this sign:

at any rate, this is how lucene writes the segment file. note that this is done in a crc32 signed file:

and the info write method is:

today, i would probably use a json file for something like that (bonus point, you know if it is corrupted and it is human readable), but this code was written in 2001, so that explains it.

this is the format of the format of a segment file, and the segments.gen file is generated using:

moving on to actually writing data, i created ten lucene documents and wrote them. then just debugged through the code to see what will happen. it started by creating _0.fdx and _0.fdt files. the .fdt is for fields, the fdx is for field indexes.

both of those files are used when writing the stored fields. this is the empty operation, writing an unstored field.

this is how fields are actually stored:

and then it ends up in:

note that this particular data goes in the fdt file, while the fdx appears to be a quick way to go from a known document id to the relevant position in the fdx file.

as i was going through the code, i did some searches, and found a very detailed explanation of the actual file format in the docs . that is really nice and quite informative, however, just seeing how the “let us take the documents and make them searchable” part is quite interesting. lucene has a lot of chains of responsibilities going through. and it is also quite interesting to see the design choices that were made.

unfortunately, lucene is very much wedded to its file format, and making changes to it isn’t going to be possible, which is a shame, since it impacts quite a lot of the way lucene works in general.