DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations
Building Scalable Real-Time Apps with AstraDB and Vaadin
Register Now

Trending

  • Building a Flask Web Application With Docker: A Step-by-Step Guide
  • 5 Key Concepts for MQTT Broker in Sparkplug Specification
  • File Upload Security and Malware Protection
  • From On-Prem to SaaS

Trending

  • Building a Flask Web Application With Docker: A Step-by-Step Guide
  • 5 Key Concepts for MQTT Broker in Sparkplug Specification
  • File Upload Security and Malware Protection
  • From On-Prem to SaaS
  1. DZone
  2. Data Engineering
  3. Databases
  4. Reviewing Lightning Memory-Mapped Database Library: Going Deeper

Reviewing Lightning Memory-Mapped Database Library: Going Deeper

Oren Eini user avatar by
Oren Eini
·
Jul. 25, 13 · Interview
Like (0)
Save
Tweet
Share
3.36K Views

Join the DZone community and get the full member experience.

Join For Free

okay, i now have a pretty rough idea about how the codebase actually works. i still think that the codebase is quite ugly. for example, take a look at this:

image

the len parameter for createfile is whatever to open or create or just open (read only). but why is it in a parameter called len?

probably because it was just there, and it would be a sin to create another local variable just for this, i guess (in a codebase where a method had > 18 local variables!).  to make things more interesting, in the rest of this method, this is actually a string len variable, sigh.

at any rate, let us actually dig deeper now. the following structure is holding data about a db.

image

this is actually somewhat misleading, at least with regards to how i would think about a db. this is the entry point for all the pages that belong to a specific db. but a db in lmdb is not really the same thing as a db in sql server or ravendb. it all reside in the same file, and you always have at least two. the first one is the free db, which is used to track all the free pages. the second one is the main db. then you have additional, named databases.

this is used here:

image

this define the metadata for the entire environment. note that we have the two dbs there in mm_dbs. the mm_txnid denotes the last committed transaction id. this value is what gives lmdb its mvcc support.  the mm_last_pg value is the last used page, any transaction that wants to write will start writing at that value.

let us see how we deal with pages here, shall we?

image

the first part find try to find a dirty page if we are in a read/write transaction and we haven’t specify that we can write directly to memory. this is done by doing a binary search on the list of dirty pages.

otherwise, we can just hand the user the actual page by accessing it directly.

next, let us look where this is actually used. searching for a page with a specific key in it. this is done mostly in mdb_node_search.

image

this seems to be doing a binary search for the keys inside a specific page (in this case, the page that is at the top of the stack on the cursor). that leads to the conclusion that pages internally have data internally stored as sorted arrays.

and this leads me to another pet peeve with this code base. take a look at this line:

image

sure, this is a well known trick to cheaply divide  a number by half, but are you seriously telling me that the compiler isn’t going to optimize  (low + high) / 2 ? to my knowledge, no c compiler updated in the last 10 – 15 years managed to miss this optimization. so why write code that is going to be harder to read?

okay, so now we know how we search for a specific key inside a page, but how do we get to the actual page that we want to search on? this happens on mdb_page_search_root. let me see if i can take it apart.

when this method is called, the cursor is setup so the first page on the pages stack is the root page.

and… that is enough for me. up until now, i have been trying to just read the code. not execute it, not debug through it, nothing .just go over the code one line at a time and figure out what is going on. i actually think that i have a really good grasp about what is going on in there, but i think that this is pretty much all i can do at that point from just reading the code. so i am going to stop now and setup an debug environment so i can work with it, and report my finding from stepping through the code.

Database Lightning Memory-Mapped Database Lightning (software) Library

Published at DZone with permission of Oren Eini, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Trending

  • Building a Flask Web Application With Docker: A Step-by-Step Guide
  • 5 Key Concepts for MQTT Broker in Sparkplug Specification
  • File Upload Security and Malware Protection
  • From On-Prem to SaaS

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com

Let's be friends: