DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations
Securing Your Software Supply Chain with JFrog and Azure
Register Today

Trending

  • Part 3 of My OCP Journey: Practical Tips and Examples
  • Unlocking Game Development: A Review of ‘Learning C# By Developing Games With Unity'
  • How To Use an Automatic Sequence Diagram Generator
  • File Upload Security and Malware Protection

Trending

  • Part 3 of My OCP Journey: Practical Tips and Examples
  • Unlocking Game Development: A Review of ‘Learning C# By Developing Games With Unity'
  • How To Use an Automatic Sequence Diagram Generator
  • File Upload Security and Malware Protection

Raven's Storage: Memtables are Tough

Oren Eini user avatar by
Oren Eini
·
May. 06, 13 · Interview
Like (0)
Save
Tweet
Share
3.55K Views

Join the DZone community and get the full member experience.

Join For Free

memtables are conceptually a very simple thing. you have the list of values that you were provided, as well as a skip list for searches.

complications:

  • memtables are meant to be used concurrently.
  • we are going to have to have to hold all of our values in memory. and i am really not sure that i want to be doing that.
  • when we switch between mem tables (and under write conditions, we might be doing that a lot), i want to immediately clear the used memory, not wait for the gc to kick in.

the first thing to do was to port the actual skiplist from the leveldb codebase. that isn’t really hard, but i had to make sure that assumptions made for the c++ memory model are valid for the clr memory model. in particular, .net doesn’t have atomicpointer, but volatile.read / volatile.write are a good replacement, it seems. i decided to port the one from leveldb because i don’t know what assumptions other list implementations have made. that was the first step in order to create a memtable. the second was to decide where to actually store the data.

here is the most important method for that part:

public void add(ulong seq, itemtype type, slice key, stream val)

the problem is that we cannot just reference this. we have to copy those values into memory that we control. why is that? because the use is free to change the stream contents or the slice’s array as soon as we return from this method. by the same token, we can’t just batch this stuff in memory, again, because of the loh. the way this is handled in leveldb never made much sense to me, so i am going to drastically change that behavior.

in my implementation, i decided to do the following:

  • copy the keys to our own buffer, and keep them inside the skip list. this is what we will use for actually doing searches.
  • change the skiplist to keep track of values, as well as the key.
  • keep the actual values in unmanaged memory, instead of managed memory. that avoid the whole loh issue, and give me immediate control on when the memory is disposed.

this took some careful coding, because i want to explicitly give up on the gc for this. that means that i need to make damn sure that i don’t have bugs that would generate memory leak.

each memtable would allocate 4mb of unmanaged memory, and would write the values to it. note that you can write over 4mb (for example, by writing a very large value, or by writing a value whose length exceed the 4mb limit. at that point, we would allocate more unmanaged memory, and hand over the memory table to compaction.

the whole thing is pretty neat, even if i say so myself smile .

Memory (storage engine)

Published at DZone with permission of Oren Eini, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Trending

  • Part 3 of My OCP Journey: Practical Tips and Examples
  • Unlocking Game Development: A Review of ‘Learning C# By Developing Games With Unity'
  • How To Use an Automatic Sequence Diagram Generator
  • File Upload Security and Malware Protection

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com

Let's be friends: