Architecture and Design Guidance: Make the Machine Do Your Work for You

i am currently playing around with persistent b+ trees. this is done using memory-mapped files and pointer arithmetic in c#. i consider this a non-trivial exercise, and certainly quite outside of my usual haunts. obviously, as you can imagine, there are bugs. those can be pretty hard to figure out because they can arise from any number of issues (usually it is me, but still …).

after fighting for a few hours with a set of bugs that just wouldn’t let me be, i decided that i had enough of that and set out to design the debug hooks i need to actually figure it out. because i am dealing with pointers, i couldn’t just use the normal methods that i would use when debugging c# code. to top that, i think that most of you would agree that explaining a data structure ( any data structure) without a whiteboard is nigh impossible. in the same manner, trying to figure out a bug in a b+ tree by following the member references was too hard. i wanted to actually visualize it.

i could throw it into a treeview in a few minutes, but i wanted to take the time to do it right. i have a feeling that this is going to be useful for the future as well, so i learned how to use graphviz’s dot language and wrote the code to dump the structure of the b+tree into a properly formatted dot file, after which i could ask dot.exe to turn that into an image. the piece of code that i was having trouble with was the page split function.

here is the before snapshot. note that we have nearly run out of room in this page.

and here is the output of the after snapshot. as you can see, this really makes no sense at all. we have a page split, but it all went into the same page?

what actually happened is that i had a bug where i wired both the left and right pages in the split to the original page. once that was fixed (and once i actually saw the data), it was very easy to see what was wrong. it was a matter of changing a single parameter. after the fix, i could visually see that everything was fine.

leaving aside the details about the actual bug and fix, the lesson that i wish to impart today is that building those sort of capabilities into your systems is important. now that i have this ability, i fully expect to be start using it when i need to debug the next issue.

in fact, i ran into the next issue quite quickly. after inserting quite a lot of data, i messed up the pointers and returned what looked very much like corrupted data. it was pretty hard to isolate (as usual with pointer issues) and, since the actual error occurred at some point during the process, i wasn’t really clear on where and how to figure it out. i ended up just dumping the tree content into a file after every operation, and then checking the operations until i found the one where we started to have corruption.

that allowed me to figure out that the corruption began at step 98 and iteration four.

as it turned out, that was pretty hard to figure out because i didn’t realize that when i was splitting the page i needed to compact the data on the original page. too much thinking in list<t> and not enough thinking about the layout of the problem in memory.