Solving the Mystery: Why Java RSS Grows in Docker on M1 Macs

The Problem

You're running a Java application in a Docker container on your M1 Mac. Everything works fine, but you notice something strange: The resident set size (RSS) keeps growing, even though your heap usage is stable. After hours of investigation, you find mysterious rwxp memory regions, each exactly 128 MB, accumulating in your process memory map.

What's causing this? Is it a memory leak? A JVM bug? Something else entirely?

The Investigation

Our journey began with monitoring RSS growth in a Java 17 application deployed on Docker-backed Minikube. Despite stable heap usage and no obvious memory leaks, RSS continued to grow by hundreds of megabytes over time.

Initial Observations

• RSS growth: ~500-700 MB over 11 hours
• Heap usage: Stable and within limits
• Thread count: Stable
• Native memory tracking: No obvious leaks

Deep Dive Into Memory Maps

Using /proc/PID/maps and /proc/PID/smaps, we discovered the growth was coming from anonymous executable memory regions:

$ cat /proc/1/maps | grep rwxp
efffd1d7c000-efffd9d7c000 rwxp 00000000 00:00 0 
efffdb185000-efffe3185000 rwxp 00000000 00:00 0 
efffe3d85000-efffebd85000 rwxp 00000000 00:00 0 
...

Each region was exactly 128 MB, in the 0xefff* address range, with read-write-execute permissions. But what was in them?

The Discovery

Reading the memory content revealed something unexpected: ARM64 machine code instructions. But wait, the Java binary was x86-64, and the process reported x86_64 architecture. What was ARM64 code doing there?

The "Aha!" Moment

The answer: Rosetta 2 translation cache.

When running x86-64 containers on ARM64 M1 Macs via Docker Desktop, Rosetta 2 translates x86-64 instructions to ARM64. The translated code is cached in executable memory regions-those mysterious RWXP regions we were seeing!

The Root Cause

Here's what was happening:

1. JIT compilation: Java's JIT compiler generates x86-64 native code for hot methods
2. Rosetta 2 intercepts: When x86-64 code executes, Rosetta 2 translates it to ARM64
3. Translation cache: Translated ARM64 code is stored in 128 MB RWXP memory regions
4. Growth: More JIT-compiled methods = more translations = more RWXP regions

Evidence

The Proof

To definitively prove JIT was the trigger, we disabled JIT compilation using the -Xint flag:

-Xint  # Run in interpreter-only mode

Results

Result: With JIT disabled, RWXP growth completely stopped. Monitoring over 1+ hour confirmed zero growth.

Why This Happens

The Perfect Storm

• ARM64 host: M1 Mac (Apple Silicon)
• x86-64 container: Docker image built for AMD64
• Rosetta 2 enabled: Docker Desktop uses Rosetta 2 for emulation
• Dynamic code generation: Java JIT compiler

When all four conditions are met, Rosetta 2 must translate every JIT-compiled method from x86-64 to ARM64, storing the translations in executable memory regions that count toward process RSS.

The Solution

Option 1: Use Native ARM64 Images (Recommended)

The best solution is to use ARM64-native Docker images:

# Build for ARM64
docker build --platform linux/arm64 ...

# Or use multi-arch images
docker pull --platform linux/arm64 your-image:tag

Benefits:

• No Rosetta 2 translation needed
• No RWXP growth
• Better performance (native execution)
• Lower memory usage

Option 2: Deploy to x86-64 Infrastructure

If ARM64 images aren't available, deploy to x86-64 servers or cloud instances where Rosetta 2 isn't needed.

Option 3: Accept and Monitor

If you must use x86-64 containers on M1 Macs:

• Increase container memory limits
• Monitor RWXP growth
• Plan for periodic restarts if needed

Not Recommended

Don't disable JIT in production (-Xint). While it stops RWXP growth, it dramatically reduces performance. Use it only for testing/debugging.

Key Takeaways

1. Rosetta 2 translation cache causes RWXP memory growth in x86-64 containers on ARM64 Macs
2. JIT compilation is the primary trigger; each compiled method needs translation
3. Native ARM64 images eliminate the problem entirely
4. This is expected behavior, not a bug-it's the cost of emulation

Conclusion

What started as mysterious RSS growth turned out to be Rosetta 2's translation cache storing ARM64 translations of JIT-compiled Java code. By understanding the mechanism and testing with JIT disabled, we proved the root cause and identified the best solution: use native ARM64 images.

If you're experiencing similar RSS growth in Java applications on M1 Macs, check for RWXP regions in your process memory map. If you see them, Rosetta 2 translation is likely the culprit.

How to Check

# Check for RWXP regions
cat /proc/PID/maps | grep rwxp

# Count RWXP regions
cat /proc/PID/maps | grep rwxp | wc -l

# Check if Rosetta 2 is active
cat /proc/PID/maps | grep rosetta

Have you encountered similar issues? Share your experience in the comments below!