RavenDB–Indexing Performance Timeline

we get people asking us about the details of ravendb indexes, why a certain index is busy, what is costly, etc. this is a pretty common set of questions, and while we were always able to give the customer a good answer, i was never truly happy with that. a lot of that was about understanding the system to a very deep level and being able to read fairly obscure to an outsider. like most things that aren’t obvious and easy to use, that bugged me, deeply. therefor, we spent a few weeks adding internal metrics and exposing everything in a nice and easy fashion.

let me show you the details from one of our production databases:

the y axis is for different indexes. on the y axis, you have time. note that the top part is for map operations, and the bottom part is for reduce operations, and that they are running in parallel. you can also visually see that those the map & reduce are actually running in parallel.

what is even better, you can visually see each individual indexing batch, let us focus on the long indexing batch in the middle:

as you can see, we are indexing 512 documents, but it took us a really long time to index them, why is that? we can see that the reason that this batch took so long is that the orders/stats and products/stats index were very long. clicking on them, we can see:

the orders/stats index accepted a 105 documents, and outputted 1,372 map values. and the costly part of this index was to actually commit those (along with the products/stats, which also output a similar amount) to the map storage.

another slow index is exposed here:

in this case, we have an index that is big enough that we need to flush it to disk from ram. that explains things.

i’m really excited about this feature.