Troubleshoot OutOfMemoryError: Unable to Create New Native Thread

There are 8 flavors of java.lang.OutOfMemoryError. In these 8 flavors java.lang.OutOfMemoryError: unable to create new native thread is one of the commonly occurring flavor. This type of OutOfMemoryError is generated when an application isn’t able to create new threads. This error can surface because of following two reasons:

1. There is no room in the memory to accommodate new threads.
2. The number of threads exceeds the Operating System limit.

Solutions

There are 6 potential solutions to address this ‘java.lang.OutOfMemoryError: unable to create new native thread’ error. Depending on what event is triggering this error, either one or a combination of the below-mentioned solutions can be applied to resolve the problem

1. Fix Thread Creation Rate

When you see ‘java.lang.OutOfMemoryError: unable to create new native thread’, you should diagnose whether the application has started to create more threads. You can use online thread dump analyzer tool such as http://fastthread.io/ (which I would highly recommend), to see how many threads are created? What is the stack trace of those excessively created threads? Who is creating them? Once you know to these questions, it’s easy to solution them. Check out the ‘real world example’ section of this article, which walks through a ‘java.lang.OutOfMemoryError: unable to create new native thread’ problem experienced a major B2B travel application and how http://fastthread.io/ tool was used to diagnose the problem.

2. Increase the Thread Limits Set at Operating System

The Operating System has limits for the number of threads that can be created. The limit can be found by issuing “ulimit –u” command. In certain servers, I have seen this value set to a low value such as 1024. It means totally only 1024 threads can be created in this machine. So if your application is creating more than 1024 threads, it’s going to run into java.lang.OutOfMemoryError: unable to create new native thread.In such circumstances increase this limit.

3. Allocate More Memory to the Machine

If you don’t see a high number of threads created and “ulimit –u” value is well ahead then it’s indicative that your application has grown organically and needs more memory to create threads. In such circumstance, allocate more memory to the machine. It should solve the problem.

4. Reduce Heap Space

One very important point that even seasoned engineers forget is: threads are not created within the JVM heap. They are created outside the JVM heap. So if there is less room left in the RAM, after the JVM heap allocation, application will run into java.lang.OutOfMemoryError: unable to create new native thread. So let’s consider this example:

As per this configuration 5.5 GB (i.e. 5 GB heap + 512 MB Perm Gen) is used by the JVM Heap and it leave only 0.5GB (i.e. 6 GB – 5.5GB) space. Note in this 0.5 GB space - kernel processes, other user processes and threads has to run. It may not be sufficient, and most likely the application will start to experience java.lang.OutOfMemoryError: unable to create new native thread. To mitigate this problem, you can consider reducing the Heap Size from 5GB to 4GB (if your application can accommodate it without running into other memory bottlenecks).

5. Reduce Number of Processes

This solution is quite similar to ‘Reduce Heap Space’. Let’s looks into this scenario, where you are running multiple processes in a server which is constrained by memory. Say:

It means in total all of the java processes heap is occupying 30 GB (i.e. 5 processes X 6 GB) of memory. It leaves only 2 GB for kernel processes, other user processes and threads to run. It may not be sufficient, and most likely the application will start to experience java.lang.OutOfMemoryError: unable to create new native thread. In this circumstance, it’s better to run only 4 java processes on one server. So that only 24 GB is occupied (4 processes X 6GB) and it leaves 8 GB (i.e. 32 GB – 24 GB) of memory. It might leave enough room for threads and run other processes to run.

6. Reduce Thread Stack Size (-Xss)

A thread occupies memory in RAM. So if each thread has high memory allocation then overall memory consumption will also go higher. The default value of a thread’s memory size depends on the JVM provider. In some cases it's 1mb. So if your application has 500 threads then threads alone is going to occupy 500mb of space.

However, you can use the java system property –Xss to set the thread’s memory size. Using this property you can throttle down the memory size. Example if you configure -Xss256k, your threads will only consume 125mb of space (i.e. 500 threads X 256k). So by lowering –Xss size also, you might be able to eliminate java.lang.OutOfMemoryError: unable to create new native thread.

CAUTION: However if you configure –Xss to a very low value, you will start to experience java.lang.StackOverflowError. If you configure to even lower value, JVM will not even start.

Real World Example

Now let me walk through a real world example of java.lang.OutOfMemoryError: unable to create new native thread, which I diagnosed recently. This error was experienced by a major B2B travel application in North America. No recent production deployments were made to this application, but all of a sudden it started to throw java.lang.OutOfMemoryError: unable to create new native thread.

Step 1: As an initial step, we captured the thread dump from the application when it was experiencing this java.lang.OutOfMemoryError: unable to create new native thread. Then we uploaded the thread dump into online thread dump analyzer http://fastthread.io/.

Step 2: http://fastthread.io/ tool reported that application had 3216 threads alive and rightly pointed that it can result in java.lang.OutOfMemoryError: unable to create new native thread. 3000+ threads was a very high thread count for this application, which is at least 6 times more than the regular period.

Fig 1: http://fastthread.io/ tool showing thread count summary

Step 3: Since now it’s confirmed that excessive threads are causing java.lang.OutOfMemoryError: unable to create new native thread, the next step is to identify what are those excessively created threads? And who is creating them? http://fastthread.io/ tool has a section “Repeating Stack traces”, in which threads with same stack traces are grouped together. In that section, we noticed that 2319 threads (i.e. 72%) are exhibiting same stack trace as shown in Fig 2.

Fig 2: http://fastthread.io/ tool showing group of threads which has same stack trace

Fig 3: Individual Thread’s stack trace as reported by the http://fastthread.io/ tool

From the stack trace, we inferred that these threads created are by the Datastax driver. This application uses DataStax driver for connecting with Apache Cassandra NoSQL Database. So now the question becomes all of sudden why Datastax driver started to create so many threads? No upgrades were made to this driver. No recent deployments were made to the application. Why all of a sudden this problem started?

Root cause: Apparently the problem turned out that Apache Cassandra NoSQL DB was running into disk space issue on one of its nodes. This issue caused the Datastax driver to spawn thousands of threads. Thus it cascaded as java.lang.OutOfMemoryError: unable to create new native thread error on the JVM side. When more space was allocated to Apache Cassandra NoSQL DB nodes, the problem got resolved.