DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
View Events Video Library
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Integrating PostgreSQL Databases with ANF: Join this workshop to learn how to create a PostgreSQL server using Instaclustr’s managed service

Mobile Database Essentials: Assess data needs, storage requirements, and more when leveraging databases for cloud and edge applications.

Monitoring and Observability for LLMs: Datadog and Google Cloud discuss how to achieve optimal AI model performance.

Automated Testing: The latest on architecture, TDD, and the benefits of AI and low-code tools.

Related

  • Java and Low Latency
  • Detecting and Solving Segmentation Faults in Linux Containers
  • What Is Segmentation in Operating System?
  • All You Need To Know About Garbage Collection in Java

Trending

  • Memory Management in Java: An Introduction
  • Best Plugins For JetBrains IDEs
  • Parallelism in ConcurrentHashMap
  • Send Your Logs to Loki

Linux Transparent Huge Pages, JEMalloc and NuoDB

Seth  Proctor user avatar by
Seth Proctor
·
May. 20, 14 · Interview
Like (0)
Save
Tweet
Share
4.67K Views

Join the DZone community and get the full member experience.

Join For Free

This article was originally written by Adam Abrevaya, VP of Engineering at NuoDB

With our latest release cycle, we faced a problem that impacted extremely load-intensive long running (multi-day) tests. The symptom was a slow but steady increase in Resident Set Size (RSS) that impacted both our Transaction Engines (TEs) and Storage Managers (SMs).  On large machines unless you're looking at "ps" stats you wouldn't notice it, but on typical cloud hardware the OOM killer would take out the NuoDB processes.

Since our system is written in C++, the obvious thought was that we had a memory leak.  That turned out not to be the case after we went through a number of various attempts to find a leak using Valgrind and JEMalloc heap profiler.  Then investigation turned to memory fragmentation which potentially was causing more pages to be held in memory with lots of holes that the memory allocator couldn't use.  JEMalloc does a great job of minimizing memory fragmentation, and we confirmed that fact.  So memory fragmentation wasn't the issue.

Our engineering team discovered (thanks Oleg) was that there were cases where JEMalloc was releasing pages back to the operating system and that didn't seem to have any effect. The problem turned out to be due to a somewhat new feature in the Linux kernel called Transparent Huge Pages (THPs).  THPs prevented pages marked with madvise(...,..., MADV_DONTNEED) from being purged from resident memory.  The quick description of THPs is that Linux will automatically create a "huge" page when a virtual memory allocation is above a certain size.  By doing this there is less bookkeeping for a single huge page resource vs lots of little 4K pages - this bookkeeping impacts the performance of the virtual memory translation lookaside buffer (TLB).  Many more details about THPs can be found here.

JEMalloc uses madvise(...,..., MADV_DONTNEED), to discard pages it doesn't need anymore.  Since that doesn't work with THPs, our engineering team (thanks Tommy) patched JEMalloc to turn off huge page allocations using madvise(...,..., MADV_NOHUGEPAGE). Doing that fixed the memory consumption issues we were seeing without any noticeable impact on our performance.  Tommy is submitting the change back to the JEMalloc community.

Unfortunately, the story doesn't end there.  Kernel versions prior to 2.6.38  don't support madvise(...,..., MADV_NOHUGEPAGE).  For that case, our TEs and SMs (in the upcoming 2.0.4 release) are producing warnings that say to turn off Transparent Huge Pages.  To turn off THPs, as root, you need to do the following:

  echo never > /sys/kernel/mm/transparent_hugepage/enabled
  echo never > /sys/kernel/mm/transparent_hugepage/defrag
Note: On some systems (CentOS 6.3), the name of directory will be redhat_transparent_hugepage.  Also note that while THPs were introduced in the Linux kernel, they are turned off by default in the Ubuntu kernel (so no worries there!).

If you run NuoDB on a Linux kernel with Transparent Huge Pages enabled, we strongly recommend turning them off.  Anyone running things like Centos 6.3, 6.4 and 6.5, which run kernel revisions below 2.6.38, needs to pay attention to this. 


Linux kernel Memory (storage engine) operating system

Published at DZone with permission of Seth Proctor, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Java and Low Latency
  • Detecting and Solving Segmentation Faults in Linux Containers
  • What Is Segmentation in Operating System?
  • All You Need To Know About Garbage Collection in Java

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends: