Reviewing Lightning Memory-Mapped Database Library: Going Deeper

okay, i now have a pretty rough idea about how the codebase actually works. i still think that the codebase is quite ugly. for example, take a look at this:

the len parameter for createfile is whatever to open or create or just open (read only). but why is it in a parameter called len?

probably because it was just there, and it would be a sin to create another local variable just for this, i guess (in a codebase where a method had > 18 local variables!).  to make things more interesting, in the rest of this method, this is actually a string len variable, sigh.

at any rate, let us actually dig deeper now. the following structure is holding data about a db.

this is actually somewhat misleading, at least with regards to how i would think about a db. this is the entry point for all the pages that belong to a specific db. but a db in lmdb is not really the same thing as a db in sql server or ravendb. it all reside in the same file, and you always have at least two. the first one is the free db, which is used to track all the free pages. the second one is the main db. then you have additional, named databases.

this is used here:

this define the metadata for the entire environment. note that we have the two dbs there in mm_dbs. the mm_txnid denotes the last committed transaction id. this value is what gives lmdb its mvcc support.  the mm_last_pg value is the last used page, any transaction that wants to write will start writing at that value.

let us see how we deal with pages here, shall we?

the first part find try to find a dirty page if we are in a read/write transaction and we haven’t specify that we can write directly to memory. this is done by doing a binary search on the list of dirty pages.

otherwise, we can just hand the user the actual page by accessing it directly.

next, let us look where this is actually used. searching for a page with a specific key in it. this is done mostly in mdb_node_search.

this seems to be doing a binary search for the keys inside a specific page (in this case, the page that is at the top of the stack on the cursor). that leads to the conclusion that pages internally have data internally stored as sorted arrays.

and this leads me to another pet peeve with this code base. take a look at this line:

sure, this is a well known trick to cheaply divide  a number by half, but are you seriously telling me that the compiler isn’t going to optimize  (low + high) / 2 ? to my knowledge, no c compiler updated in the last 10 – 15 years managed to miss this optimization. so why write code that is going to be harder to read?

okay, so now we know how we search for a specific key inside a page, but how do we get to the actual page that we want to search on? this happens on mdb_page_search_root. let me see if i can take it apart.

when this method is called, the cursor is setup so the first page on the pages stack is the root page.

and… that is enough for me. up until now, i have been trying to just read the code. not execute it, not debug through it, nothing .just go over the code one line at a time and figure out what is going on. i actually think that i have a really good grasp about what is going on in there, but i think that this is pretty much all i can do at that point from just reading the code. so i am going to stop now and setup an debug environment so i can work with it, and report my finding from stepping through the code.