Native Memory May Cause Unknown Memory Leaks

Recently I came across a strange case: the memory usage of my program exceeded the maximum value intended for the heap. Even after running GC, part of the memory was not free. I already knew that a part of  JVM memory would be allocated to native memory and part of native memory allocated to C code, but I did not have even one line of native code in my program. After reviewing and profiling my code several times, I found an interesting issue. Before diving into the problem, let's look at Java memory concepts.

Memory Management in Java

 JVM divides memory into two major spaces, heap and native memory. Heap spaces are used for allocating Java objects whereas native memory is the memory available to the OS. There is a key difference between Java 7 and 8 in the memory management model. Java 7 has PermGen; PermGen is the memory area in the heap that is used by the JVM to store class metadata, static content, primitive variables. Java 8 has eliminated PermGen and added Metaspace; actually, Metaspace and PermGen do the same thing. The main difference is that PermGen is part of the Java heap while Metaspace is NOT part of the heap. Rather Metaspace is part of native memory, which is only limited by the Host Operating System.

Native Memory

Native memory is a memory area outside the normal JVM heap but generally is a part of total memory spared by OS for the JVM process. Part of native memory is assigned to the C heap; the C heap is space that used by native C programs in Java programs.

Native Memory Tracking (NMT)

NMT is a memory tracking tool used to monitor the native memory usage of the program. You can access NMT data using jcmd utility. NMT is not enabled by default; you can enable it by adding the following parameters to JVM options:

-XX:NativeMemoryTracking=detail-XX:+UnlockDiagnosticVMOptions -XX:+PrintNMTStatistics

For monitoring and tracking memory changes during execution you should use the following command: 

jcmd <pid> VM.native_memory detail.diff

<pid> is the exact process id of your Java program in Linux. In my case my <pid> is '92165' so I executed this command:

jcmd 92165 VM.native_memory detail.diff.

Consider that  monitoring is a time-based approach it means that you should compare memory state in time T2 with the state in time T1, so  you need to mark the state of T1 as a baseline by using the following command:

jcmd 92165 VM.native_memory baseline

I also executed these commands in chronological order:

DEC 19, 2020 9:07:10 AM IRS :jcmd 92165 VM.native_memory baseline

DEC 19, 2020 12:07:10 PM IRS:jcmd 92165 VM.native_memory detail.diff. 

The output of that was something like this:

 reserved 3892 KB for Thread Stack        
        from [Thread::record_stack_base_and_size()+0xca]      

[0x9f586000 - 0x9f791000] reserved 1680KB for Thread Stack        
        from [Thread::record_stack_base_and_size()+0xca]     

I tried to execute the command jcmd 92165 VM.native_memory detail.diff at different times, and I was getting different results every moment, which means that my native memory usage is growing over time.

So, What's the Problem?!

In the previous part, I said my native memory grows over time, and I was sure I have no native code in my program, but then what is the problem? I started tracing the program at the same time as monitoring, and I found an interesting thing: there was a usage of java.util.zip code in the program. Once I reached this code the usage of native memory increased significantly. The problem was clear; this package classes internally and uses native codes. If you take a look at JDK source at jdk8 source, you can find interesting codes like this, implementations of certain classes based on native codes and C:

xxxxxxxxxx

src/share/native/java/util/zip/ZipFile.c 

JNIEXPORT jlong JNICALL

Java_java_util_zip_ZipFile_open(JNIEnv *env, jclass cls, jstring name,

                                        jint mode, jlong lastModified,

                                        jboolean usemmap)

{

    const char *path = JNU_GetStringPlatformChars(env, name, 0);

    char *msg = 0;

    jlong result = 0;

    int flag = 0;

    jzfile *zip = 0;

    if (mode & OPEN_READ) flag |= O_RDONLY;

    if (mode & OPEN_DELETE) flag |= JVM_O_DELETE;

    if (path != 0) {

        zip = ZIP_Get_From_Cache(path, &msg, lastModified);

        if (zip == 0 && msg == 0) {

            ZFILE zfd = 0;

#ifdef WIN32

            zfd = winFileHandleOpen(env, name, flag);

            if (zfd == -1) {

                /* Exception already pending. */

                goto finally;

            }

#else

            zfd = JVM_Open(path, flag, 0);

            if (zfd < 0) {

                throwFileNotFoundException(env, name);

                goto finally;

            }

#endif

            zip = ZIP_Put_In_Cache0(path, zfd, &msg, lastModified, usemmap);

        }

        if (zip != 0) {

            result = ptr_to_jlong(zip);

        } else if (msg != 0) {

            ThrowZipException(env, msg);

            free(msg);

        } else if (errno == ENOMEM) {

            JNU_ThrowOutOfMemoryError(env, 0);

        } else {

            ThrowZipException(env, "error in opening zip file");

        }

finally:

        JNU_ReleaseStringPlatformChars(env, name, path);

    }

    return result;

}

JNIEXPORT jint JNICALL

Java_java_util_zip_ZipFile_getTotal(JNIEnv *env, jclass cls, jlong zfile)

{

    jzfile *zip = jlong_to_ptr(zfile);

    return zip->total;

}

JNIEXPORT jboolean JNICALL

Java_java_util_zip_ZipFile_startsWithLOC(JNIEnv *env, jclass cls, jlong zfile)

{

    jzfile *zip = jlong_to_ptr(zfile);

    return zip->locsig;

}

JNIEXPORT void JNICALL

Java_java_util_zip_ZipFile_close(JNIEnv *env, jclass cls, jlong zfile)

{

    ZIP_Close(jlong_to_ptr(zfile));

}

Remember that native codes are not just in these classes. There are a lot of components, like components that interact with devices or communication components, that potentially use native codes.