The Leprechaun Trap: A Thread Dump Analysis Pattern

The world is full of patterns.

Starting from a fine grain of salt to complex cosmos, there are patterns everywhere. A true pattern withstands time. They are changeless in the ever-changing world. It requires a lot of hard labor, rich experience (both good and bad), laser focus, and perseverance to crystallize and create patterns from the noise. Fortunately, our computing world has been blessed to have such wonderful patterns created.

One of the classic examples is software design patterns: singleton, factory, visitor, observer, memento, etc. created by Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. These four engineers have encapsulated their years and years of hard-learned lessons, refined them, and passed them onto us as easy-to-understand patterns. These patterns are universally applicable, irrespective of what programming language you use (Java, C, C++, PHP, Ruby, etc.), irrespective of what technology stack you run (JEE, .NET, LAMP, etc.), and irrespective of what type of application you build (mobile, web, SOA, microservices, batch, etc.).

Thread Dump Analysis Patterns

Inspired by those four great engineers in my limited capacity, in a humble way, I have crystalized, refined, and created, through my years and years of production battles, thread dump analysis patterns.

Thread dumps are vital components of RCA (Root Cause Analysis). Did your application become unresponsive all of sudden? Did your application’s CPU start to spike without an increase in traffic, without making any code changes, or any environmental changes? Did your application’s response start to degrade? Did your application start to experience memory problems after running for multiple days/weeks? Answers to several such complex problems are present in the thread dumps. But they are buried inside a mountain of details. To shed light on those hidden answers, I have created thread dump analysis patterns.

In this article, let me introduce you to the ‘Leprechaun trap’ pattern. Objects that have finalize() methods are treated differently during garbage collection than those that don't don’t have those methods. During garbage collection, objects with finalize() aren’t immediately evicted from the memory. Instead, as a first step, those objects are added to an internal queue of the java.lang.ref.Finalizer object.

There is a low priority JVM thread by the name of ‘Finalizer’ that executes the finalize() method of each object in the queue. Only after the execution of finalize() do objects become eligible for GC. Because of poor implementation of finalize(), if the Finalizer thread gets blocked, then it will have a severe, detrimental, cascading effect on the JVM.

If Finalizer gets blocked, then the internal queue of java.lang.ref.Finalize will start to grow. It would cause the JVM’s memory consumption to grow rapidly. It would then result in an OutOfMemoryError, jeopardizing the entire JVM’s availability. Thus, when analyzing thread dumps, it’s highly recommended to study the stack trace of the Finalizer thread.

Real-world example

Here is a sample stack trace of a Finalizer thread that got blocked:

"Finalizer" daemon prio=10 tid=0x00007fb2dc32b000 nid=0x7a21 waiting for monitor entry [0x00007fb2cdcb6000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at net.sourceforge.jtds.jdbc.JtdsConnection.releaseTds(JtdsConnection.java:2024)
- waiting to lock 0x00000007d50d98f0 (a net.sourceforge.jtds.jdbc.JtdsConnection)
at net.sourceforge.jtds.jdbc.JtdsStatement.close(JtdsStatement.java:972)
at net.sourceforge.jtds.jdbc.JtdsStatement.finalize(JtdsStatement.java:219)
at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:101)
at java.lang.ref.Finalizer.access$100(Finalizer.java:32)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:178)

The above stack trace was captured from a JVM that was using one of the older versions of the JTDS JDBC driver. Apparently, this version of the driver had an issue; you can see the finalize() method in the net.sourceforge.jtds.jdbc.JtdsStatement object calling the JtdsConnection#releaseTds() method. Apparently, this method got blocked and never returned back. Thus, the Finalizer thread got stuck indefinitely in the JtdsConnection#releaseTds() method. Due to that, Finalizer wasn’t able to work on the other objects that had finalize() methods. Due to that, the application started to suffer from an OutOfMemoryError . In the latest version of the JTDS JDBC driver, this issue was fixed. The point is that when you are implementing finalize() methods, be very careful.

Why named as Leprechaun Trap?

Kids in some western countries build Leprechaun traps as part of celebrating St. Patrick’s Day. Leprechauns are fairy characters — basically a very tiny old man wearing a green coat and hat who hoards and searches for gold coins. Kids build creative traps for Leprechauns, luring them with gold coins. Similarly, the anxious Finalizer thread is always in search of objects that have finalize() methods to execute them. If a finalize() method is wrongly implemented, it can trap the Finalizer thread. Because of this similarity, we have named it a Leprechaun Trap.

Tool

You can learn more such patterns here. On the other hand, to make it easy for the users, we have built a universal thread dump analysis tool: fastthread.io in which all the thread dump analysis patterns are incorporated. You just have to upload your thread dump to this online tool, and automatically, all the thread dump analysis patterns are applied and the root cause of the problem will be reported to you in flash.