Simulating and Troubleshooting OOMError in Scala

As we continue in this series of simulating and troubleshooting performance problems in Scala, now let’s discuss how to simulate the java.lang.OutOfMemoryError: Java Heap space problem. java.lang.OutOfMemoryError: Java Heap space will be thrown by the Scala application when it generates more objects than the maximum configured heap size.

Scala OutOfMemoryError Program

Here is a sample Scala program, which generates the java.lang.OutOfMemoryError: Java Heap space problem:

package com.yc 
import java.util 
class OOMApp { 
}
object OOMApp {   
   var myMap = new util.HashMap[String,String]()      
      def main(args: Array[String]): Unit = {         
       var counter = 0;            
         while (true) {               
           System.out.println("Inserting large String")               
           myMap.put("key"+counter, "Large stringgggggggggggggggggggggggggggg"               
           + "ggggggggggggggggggggggggggggggggggggggggggggggggggggg"               
           + "ggggggggggggggggggggggggggggggggggggggggggggggggggggg"               
           + "ggggggggggggggggggggggggggggggggggggggggggggggggggggg"               
           + "ggggggggggggggggggggggggggggggggggggggggggggggggggggg"               
           + "ggggggggggggggggggggggggggggggggggggggggggggggggggggg"               
           + "ggggggggggggggggggggggggggggggggggggggggggggggggggggg"               
           + "ggggggggggggggggggggggggggggggggggggggggggggggggggggg"               
           + "ggggggggggggggggggggggggggggggggggggggggggggggggggggg"               
           + "ggggggggggggggggggggggggggggggggggggggggggggggggggggg"               
           + "ggggggggggggggggggggggggggggggggggggggggggggggggggggg"               
           + "ggggggggggggggggggggggggggggggggggggggggggggggggggggg"               
           + counter)               
           counter = counter+(1);            
        }      
    } 
}

This sample Scala program contains an OOMApp class. Since this class contains the while(true) loop, it keeps on inserting records into the HashMap infinitely. When HashMap grows beyond the maximum heap size (i.e., -Xmx), the java.lang.OutOfMemoryError: Java Heap space will be thrown. The diagram below illustrates the records present in the HashMap.

HashMap causing OutOfMemoryError

When we executed the program above, as expected, java.lang.OutOfMemoryError: Java heap space was thrown within a few seconds.

Even though this is a hypothetical example that simulates java.lang.OutOfMemoryError, this is how a typical memory leak happens in enterprise applications. When records get inserted into a data structure (like HashMap, ArrayList, Set, etc.) and never get removed, java.lang.OutOfMemoryError will be thrown.

How To Troubleshoot OutOfMemoryError

You can diagnose OutOfMemoryError either through a manual or automated approach. 

Manual Approach

In the manual approach, you will need to capture a heap dump as the first step. A heap dump is a snapshot of memory that will show all the objects in memory, the values contained by those objects, and their references. You can capture a heap dump using one of the 7 approaches given here, but an important criterion is that you need to capture the heap dump right before OutOfMemoryError is thrown. If you are going to capture a heap dump after OutOfMemoryError has occurred, then leaking objects can get garbage collected, and it will become hard (or even impossible) to diagnose the problem. Once heap dumps are captured, you need to import the heap dumps from your production servers to your local machine. From your local machine, you can use heap dump analysis tools like jHat and HeapHero to analyze the heap dumps.

Automated Approach

On the other hand, you can also use the yCrash open source script, which would capture 360-degree data (GC log, 3 snapshots of thread dump, heap dump, netstat, iostat, vmstat, top, top -H, etc.) from your application stack within a minute and generate a bundle zip file. You can then either manually analyze these artifacts or upload them to the yCrash server for automated analysis. 

We used the automated approach. Once the captured artifacts were uploaded to the yCrash server, it instantly generated the below root cause analysis report highlighting the source of the problem.  

Heap dump analysis report generated by the yCrash tool 

yCrash tool pointing out the root cause of OutOfMemoryError

The heap dump analysis report above is from the tool which precisely points out that the HashMap (i.e., myMap) data structure present in the com.yc.OOMApp$.myMap to be the root cause of the memory leak. Also, the tool is reporting that this HashMap is holding 99% of memory. 

Equipped with this information one can easily go ahead and fix the problematic code.

Video

To see the visual walk-through of this post, click below: