Simulating and Troubleshooting Blocked Threads in Scala [Video]

In this series of simulating and troubleshooting performance problems in Scala, let’s discuss how to make threads go into a blocked state. A thread will enter into a blocked state when it cannot acquire a lock on an object because another thread already holds the lock on the same object and doesn’t release it.

Scala Blocked Thread Program

Here is a sample program, which would make threads go into a blocked state.

package com.yc 
class BlockedApp {
}
object BlockedApp {   
    def main(args: Array[String]): Unit = {      
         for (counter <- 1 to 10) {         
            new AppThread().start()      
         }   
     }  
     def action(): Unit = {      
         this.synchronized {         
             while (true)         
             Thread.sleep(600000)      
        }   
     }   
     class AppThread extends Thread {      
       override def run(): Unit = BlockedApp.action()   
     } 
}

The sample program contains the BlockedApp class. This class has a start() method. In this method, 10 new threads are created. In the AppThread class there is a run() method that invokes BlockedApp.action(). In this BlockedApp.action() method, the thread is put to continuous sleep (i.e., the thread is repeatedly sleeping for 10 minutes again and again). 

But if you notice, the action() method is a synchronized method. Synchronized methods can be executed by only one thread at a time. If any other thread tries to execute the action() method while the previous thread is still working on it, then the new thread will be put in the blocked state. 

In this case, 10 threads are launched to execute the action() method. However, only one thread will acquire the lock and execute this method, and the remaining 9 threads will be put in a blocked state.

NOTE: If threads are in a BLOCKED state for a prolonged period, then the application may become unresponsive.

How To Diagnose Blocked Threads

You can diagnose blocked threads either through a manual or automated approach.

Manual Approach

In the manual approach, you need to capture thread dumps as the first step. A thread dump shows all the threads that are in memory and their code execution path. You can capture a thread dump using one of the 8 options mentioned here. But an important criterion is as follows: you need to capture the thread dump right when the problem is happening (which might be tricky to do). Once the thread dump is captured, you need to manually import the thread dump from your production servers to your local machine and analyze it using thread dump analysis tools like fastThread or samurai.

Automated Approach

On the other hand, you can also use the yCrash open-source script, which would capture 360-degree data (GC log, 3 snapshots of thread dump, heap dump, netstat, iostat, vmstat, top, top -H, etc.) right when the problem surfaces in the application stack and analyze them instantly to generate root cause analysis report. 

We used the automated approach. Below is the root cause analysis report generated by the yCrash tool highlighting the source of the problem.

yCrash reporting transitive dependency graph of 9 BLOCKED threads

yCrash prints a transitive dependency graph that shows which threads are getting blocked and who is blocking them. In this transitive graph, you can see ‘Thread-0’ blocking 9 other threads. If you click on the thread names in the graph, you can see the stack trace of that particular thread.

yCrash reporting the stack trace of 9 threads that are in BLOCKED state

Here is the screenshot that shows the stack trace of the 9 threads which are in a blocked state. From the stack trace, you can observe that the thread is stuck on the ‘com.yc.BlockedApp$.action(BlockedApp.scala:17)’ method.

Equipped with this information, one can easily identify the root cause of the blocked state threads.

Video

To see the visual walk-through of this post, click below: