Find JVM Memory Leaks with Instrumentation and Phantom References

Over the past year or so there has been quite a lot of attention on finding memory leaks in a JVM. Memory leaks can cause havoc in a JVM. They can be unpredictable and result in costly performance degradation or even down time during server restarts. Ideally, memory leaks are found long before an application runs in production. Unfortunately, lower-state testing is often not adequate enough to cause performance degradation or OutOfMemoryErrors. Sometimes, a leak slowly steals memory over weeks or months of up-time. This type of leak is difficult to detect before production, but not impossible. This article will outline one approach to finding such leaks without costing you a penny!

The nuts and bolts of this approach rely on PhantomReferences and instrumentation. PhantomReferences are not commonly used, so let’s take a closer look. In Java, there are three types of Reference: WeakReference, SoftReference, and PhantomReference. Most developers are familiar with WeakReferences. They simply do not prevent garbage collection of the objects they reference. SoftReferences are similar to WeakReferences, but will sometimes prevent their referent objects from being garbage collected. SoftReferences will likely prevent an object from being garbage collected if the available memory is deemed plentiful during garbage collection. Lastly, PhantomReferences are almost nothing like their weak and soft siblings in that they are not intended for an application to directly hold these references. PhantomReferences are used as a notification mechanism of when an object is about to be garbage collected. The Javadoc says “Phantom references are most often used for scheduling pre-mortem cleanup actions in a more flexible way than is possible with the Java finalization mechanism.”. We are not interested in performing any clean-up, but we will record when an object is garbage collected.

Instrumentation is the other integral functionality. Instrumentation is the process of altering the bytecode of a class before it is loaded by the VM. It is a powerful feature of Java and can be used for monitoring, profiling, and in our case, event logging. We will use instrumentation to modify application classes such that any time an object is instantiated, we will create a PhantomReference for it. The design for this memory leak detection mechanism should be shaping up now. We’ll use instrumentation to coerce classes into telling us when they create objects and we’ll use PhantomReferences to record when they are garbage collected. Lastly, we’ll use a data store to record this data. This data will be the basis for our analysis to determine if objects are being leaked.

Before we go any further, let’s skip to a screen shot of what we’ll be able to do at the end of this article.

This graph displays the number of objects that have been allocated, but not garbage collected over time. The running code is Plumbr’s own memory leak demo application. Plumbr added a couple of memory leaks to Spring framework‘s Pet Clinic sample application. The graph will make more sense when you see the relevant leaky code:

public class LeakingInterceptor extends HandlerInterceptorAdapter {

  static List<ImageEntry> lastUsedImages = 
      Collections.synchronizedList(new LinkedList<ImageEntry>());

  private final byte[] imageBytes;

  public LeakingInterceptor(Resource res) throws IOException {
    imageBytes = FileCopyUtils.copyToByteArray(res.getInputStream());
  }

  @Override
  public boolean preHandle
      (HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
    byte[] image = new byte[imageBytes.length];
    System.arraycopy(imageBytes, 0, image, 0, image.length);
    lastUsedImages.add(new ImageEntry(image));
    return true;
  }
}

With every request, this interceptor leaks an ImageEntry object, which references a byte array. Hence, after only 10 refreshes of the PetClinic’s front page, you can already see the leaky trend in the graph.

Now, let’s start writing some code. The first class we are going to need is an interface that the instrumented bytecode will call when a class is instantiated. Let’s call this class “Recorder” and create a static method for receiving objects:

class Recorder {
    public static void record(Object o) {
       ...
    }
}

The implementation details of this method are out of the scope of this article. Now that we have an interface, we can work on our instrumentation code. Instrumentation is a broad topic, but we’ll keep the scope narrowed to our project. We’ll use ASM (from ObjectWeb) to perform the bytecode manipulation. This guide assumes you are already familiar with ASM. If you are not, you may want to take some time to brush-up first.

Simply put, we want to modify any application code that instantiates a new object such that it will call our Recorder.record(…) method with the new object as a parameter. To identify “application code”, we’ll allow the user to provide a set of regular expressions that will specify the set of classes to be included and classes to be excluded. We’ll create a class called Configuration to load a configuration properties file, which contains this information. This class will be used to determine if a class should be instrumented. Later, we’ll use it to define some other properties as well.

Instrumentation occurs at runtime as classes are loaded. Instrumentation is performed by “agents”, which are packaged in a jar file. If you are not familiar with agents, you can check out the java.lang.instrument package’s javadoc documentation. The entry point into an agent is the agent class. Here are the possible method signatures for the agent entry points:

public static void premain(String agentArgs, Instrumentation inst);

public static void agentmain(String agentArgs, Instrumentation inst);

The premain method is invoked when the agent is specified using the “-javaagent” parameter of the JVM command line at start up. The agent main method is invoked if the agent is attached to an existing JVM. Our agent will be most useful if it is applied at start up. Conceivably, you could attach the agent after you have discovered a memory leak, but it can only provide memory leak data that it has recorded since it was attached. Further it will only instrument classes that get loaded after it has attached. It is possible to force the JVM to redefine classes, but our component will not provide this functionality.

Let’s call our agent class HeapsterAgent and give the above methods a body each:

public static void premain(String agentArgs, Instrumentation inst) {
    configure(agentArgs, inst);
}

public static void agentmain(String agentArgs, Instrumentation inst) {
    configure(agentArgs, inst);
}

The initialization process for both entry points will be identical, so we’ll cover them in a single configuration method. We’ll skip over the most of the Configuration implementation details to focus on instrumentation. We’ll want our class to implement java.lang.instrument.ClassFileTransformer interface. When a ClassFileTransformer is registered with the JVM, it is given the opportunity to modify classes as they are loaded. Our HeapsterAgent class now has this signature:

public class HeapsterAgent implements ClassFileTransformer

The configure method needs to register an instance of HeapsterAgent with the JVM in order to intercept the classes being loaded. Here is the code:

inst.addTransformer(new MemoryTraceAgent());

“inst” is the Instrumentation parameter of the configure(…) method.

Clever readers might already be thinking "How will the application classloaders discover the new Recorder class?". There are a couple of solutions to this problem. We could use the Instrumentation class’ appendToBootstrapClassLoaderSearch(JarFile jarfile) method to append the relevant classes to the boot classpath, where classes should be discoverable by the application classloaders. However, in order to discover leaked classes, the ClassLoader class itself must be instrumented. This can only be done effectively by creating your own jar containing java.lang.ClassLoader and superseding the JRE’s own java.lang.ClassLoader using the -Xbootclasspath/p parameter. Hence, we may as well pack the other supporting classes in the same jar.

Let’s now move on to the transform method. This method is provided in the ClassFileTransformer interface. Here is the full signature:

public byte[] transform(ClassLoader loader, String className,
            Class<?> classBeingRedefined, ProtectionDomain protectionDomain,
            byte[] classfileBuffer) throws IllegalClassFormatException;

This is where the instrumentation magic starts happening. This method is invoked every time the JVM loads a class. The most important parameters for us are the className and the classfileBuffer. className will help us determine if the class is an application class and classfileBuffer is the byte array of bytecode that we may wish to modify. Let’s take a look at how we’ll eliminate which classes to modify. Obviously we only want to modify application classes, so we’ll compare the className parameter to our inclusions and exclusions. Keep in mind that className is in its internal format and for name separators uses slashes (/) instead of dots (.). We also don’t want to instrument our agent code. We’ll be able to control that by comparing the package path of className with our own code base. Lastly, while developing this code, I isolated several Oracle classes that should simply never be instrumented (there are probably more). However, in general, if you are looking for leaks in your application, you can probably ignore java.*, javax.*, sun.* etc. I have hard coded some of these into the transform method. If you think there is a bug in Oracle code, you can always disable this filtering. However, I recommend that you be sparing with the Oracle code that you instrument from core packages like java.lang. It is very unlikely that you are the first to find a bug in these classes and you can send your JVM into an unrecoverable tailspin.

The last part of the transform method is the actual transformation. Here is the important code:

public byte[] transform(ClassLoader loader, String className,
    Class<?> classBeingRedefined, ProtectionDomain protectionDomain,
    byte[] classfileBuffer) throws IllegalClassFormatException {

    if (className.startsWith("java") || className.startsWith("sun")) return null;

    if (!isAgentClass(dotName) && configuration.isIncluded(dotName)) {
        ClassWriter writer = new ClassWriter(ClassWriter.COMPUTE_MAXS);
        Transformer transformer = new Transformer(writer);
        ClassReader reader = new ClassReader(classfileBuffer);
        reader.accept(transformer, ClassReader.EXPAND_FRAMES);
        return writer.toByteArray();
    }
    else return null;
}

If the fully qualified class name starts with “java” or “sun”, we return null. Returning null is your agent’s way of saying “I am not interested in transforming this class”. Next we check to see if className matches an agent class by calling isAgentClass(…). Here is the implementation:

boolean isAgentClass(String className) {
    return className.startsWith("ca.discotek.rebundled.org.objectweb.asm") || 
           className.startsWith("ca.discotek.heapster");
}

You’ll notice in the above code snippet that I have changed the base package name for the ASM classes from org.objectweb.asm to ca.discotek.rebundled.org.objectweb.asm. The agent classes will be made available on the boot classpath. If I had not changed the package namespace of the agent’s ASM classes, other tools or applications running in the JVM might unintentionally use the agent’s ASM classes.

The rest of the transform method is fairly basic ASM operations. However, we now need to take a close look at how the HeapsterTransformer class works. As you might guess, HeapsterTransformer extends the ClassVisitor class and overrides the visit method:

public void visit(int version, int access, String name, String signature, String superName, String[] interfaces) {
    super.visit(version, access, name, signature, superName, interfaces);
    this.className = name;
    this.superName = superName;
}

It records the class name and the super class name for later use.

The visitMethod method is also overridden:

public MethodVisitor visitMethod
    (int access, String name, String desc, String signature, String[] exceptions) {
    return new HeapsterMethodVisitor
        (name, access, desc, super.visitMethod(access, name, desc, signature, exceptions));
}

It forces our own MethodVisitor called HeapsterMethodVisitor to be visited. HeapsterMethodVisitor will need to add some local variables so it subclasses LocalVariableSorter. The constructor parameters include the method name, which it records for later use. The other methods that HeapsterMethodVisitor override are: visitMethodInsn, visitTypeInsn, and visitIntInsn. One might think we can get it all done in visitMethodInsn by adding code when we see an invocation to a constructor (<init>), but unfortunately, it is just not that simple. First, let’s review what we are trying to accomplish. We want to record each time an application object is instantiated. This can happen in a number of ways. The most obvious one is via a “new” operation. But what about Class.newInstance() or when an ObjectInputStream is deserialized via a readObject method? These methods do not use the “new” operator. Also, what about arrays? Creating an array is not a visitMethodInsn instruction, but we’ll want to record them too. Needless to say, assembling the code to capture all these events is tricky.

Let’s first take a look at the visitMethodInsn method. Here is the first statement:

if (opcode == Opcodes.INVOKESPECIAL && name.equals("<init>") &&  
    !isIgnorableConstructorCall(className, methodName, owner, superName)) {

Opcodes.INVOKESPECIAL indicates that either a constructor is being called or a static initializer. We only care about calls to constructors. Additionally, we don’t care about all constructor calls. Specifically, we only care about the call to the first constructor, not the chain of constructors calls from a constructor to its super class constructor. This is why it was important to record the superClass name earlier. We use a method called isIgnorableConstructorCall to determine if we want to instrument or not:

boolean isIgnorableConstructorCall(String className, String containingMethodName, String owner, String superName) {
    if (owner.equals(className) && containingMethodName.equals("<init>")) return true;
    else if (owner.equals("java/lang/Object")) return true;
    else return superName.equals(owner);    
}

The first if-statement checks if a constructor is calling another constructor within the same class (e.g. this(…)). The second line checks if the constructor call is being called on type java.lang.Object. When using ASM, any class whose super class is java.lang.Object, the super class will be specified as null. This prevents a NullPointerException from happening in the third line where we check that a contructor call to a super class is being called (e.g. super(…)). An object of type java.lang.Object will have a null super class type.

Now that we have established which constructors we can ignore, let’s get back to visitMethodInsn. Once the constructor invocation has been completed, we can record the object:

mv.visitMethodInsn(opcode,  owner,  name,  desc);
addRecorderDotRecord();

The first line is identical to the original bytecode instruction. The second line calls addRecorderDotRecord(). This method contains the bytecode to call ourRecorder class. We’ll reuse this several times, so it is in its own method. Here is the code:

void addRecorderDotRecord() {
    mv.visitMethodInsn
        (Opcodes.INVOKESTATIC, Recorder.class.getName().replace('.', '/'), "record", "(Ljava/lang/Object;)V");
}

This should all appear fairly straightforward if you understand ASM, but there is one unexplained omission that should be obvious to a bytecode expert. Java is stack based. When we called the original method:

mv.visitMethodInsn(opcode,  owner,  name,  desc);

…it popped the new object off the stack. But addRecorderDotRecord's instruction expects the new object to still be on the stack. And when it completes, it will pop off the new object. This doesn’t make sense because we haven’t examined the rest of the overridden methods. Let’s skip down to visitTypeInsn(…). Here is the first half:

public void visitTypeInsn(int opcode, String type) {
    mv.visitTypeInsn(opcode,  type);
    if (opcode == Opcodes.NEW) {
        addDup();
    }
    ...

A visitTypeInsn with Opcode.NEW as a parameter will immediately precede a call to the object’s constructor. Furthermore, the JVM specification disallows you from calling others method before an object is initialized. By using visitTypeInsn first and visitMethodInsn second, we are able to add an extra reference to the object on the stack which can be used as the parameter to our Recorder.record(…) method.

Now let’s take a look at the else-if-statement of visitMethodInsn. The methods newInstance, clone, and readObject are special methods that can instantiate an object without using the “new” operator. When we come across these methods, we create a duplicate of the object reference on the stack (using addDup()) and then call our Recorder.record(…) method, which will pop our duplicate object reference off the stack. Here is the addDup() method:

void addDup() {
    mv.visitInsn(Opcodes.DUP);
}

We have already partially examined the the visitTypeInsn method, but let’s now review its entirety:

public void visitTypeInsn(int opcode, String type) {   
    mv.visitTypeInsn(opcode,  type);
    if (opcode == Opcodes.NEW) {
        addDup();
    }
    else if (opcode == Opcodes.ANEWARRAY) {
        addDup();
        addRecorderDotRecord();
    }
}

The first line of this method ensures the original instruction is executed. We have already discussed the if-statement, which is used to duplicate an object instantiated with the “new” operator before we add the call toRecorder.record(…). The else-if-statement handles creating 1-dimensional arrays of non-primitive types. In this case we add a duplicate array object reference on the stack and then call Recorder.record(…) which pops it off.

Next we have visitMultiANewArrayInsn:

mv.visitMultiANewArrayInsn(desc,  dims);
addDup();
addRecorderDotRecord();

This method is fairly simple to understand. The first line creates a new array of multiple dimensions. The second line pushes a duplicate reference on the stack and the third line calls our Recorder.record(…) method which pops the duplicate off the stack.

Lastly, we have visitIntInsn:

public void visitIntInsn(int opcode, int operand)  {
    mv.visitIntInsn(opcode,  operand);

    if (opcode == Opcodes.NEWARRAY || opcode == Opcodes.MULTIANEWARRAY) {
        addDup();
        addRecorderDotRecord();
    }
}

This method handles the byte code operation of creating an array of primitives and a multi-dimensional array. The if-statement identifies these operations and its body ensures the original instruction is executed, then duplicates array object reference on the stack and then we call Recorder.record(…) which pops it off.

Let’s now change gears and review the ASM code to generate our custom java.lang.ClassLoader method. As mentioned earlier, we need to define our own java.lang.ClassLoader to record classes as they are defined. There is a ClassLoaderGenerator class, which does the grunt work of extracting the java.lang.ClassLoader class from the rt.jar file of our target JRE, but let’s drill in to the ASM code in ClassLoaderClassVisitor. Much of the code in this class is not particularly interesting. Let’s go right to the visitMethodInsn method of its MethodVistor class:

public void visitMethodInsn
    (int opcode, String owner, String name, String desc, boolean isInterface) {
    mv.visitMethodInsn(opcode,  owner,  name,  desc, isInterface);

    if (opcode == Opcodes.INVOKEVIRTUAL &&
        includedMethodNameList.contains(name)) {
        logger.info
            ("Instrumenting method invocation: " + owner + "." + name + ": " + desc);

        int variableIndex = newLocal(Type.getType(Class.class));
        visitVarInsn(Opcodes.ASTORE, variableIndex);
        visitVarInsn(Opcodes.ALOAD, variableIndex);
        addRecorderDotRecord();
        visitVarInsn(Opcodes.ALOAD, variableIndex);
    }
}

Line 3 invokes the original instruction. The if-statement of lines 5-6 identify the instruction as a defineClass method. The defineClass methods (namely, defineClass0, defineClass1, defineClass2) are native methods that return thejava.lang.Class object. By capturing these calls, we can capture when classes are created. Lines 10-13 create a local variable to store the java.lang.Class object, create the call to Recorder.record(…), and place Class back on the stack. FYI In other ASM code, I used a dup instruction to duplicate the reference, but when I ran the code it didn’t cooperate, which lead me to use a local variable instead.

We have now covered all the required instrumentation. The other main concept to document is the use of PhantomReferences. We have already discussed how PhantomReferences can inform us when an object is garbage collected, but how does this help us track memory leaks? If we use PhantomReferences to reference each application object, we can eliminate objects as leaky if they are regularly garbage collected. The remaining set of objects become our candidate set. If we can observe a trend of increasing object counts for a particular type, it is quite likely that we have identified a leak. You should note that, these trends that persist beyond major garbage collections are even more likely to be leaky. However, this code does not consider garbage collections at this time.

We’ll now return to the Recorder class to examine the PhantomReference functionality. The record method has the following code:

long classId = dataStore.newObjectEvent(o, System.currentTimeMillis());
PhantomReference<Object> ref = new PhantomReference<Object>(o, queue);
map.put(ref, classId);

The first line references a variable called dataStore. The data store is an implementation detail. I have implemented an in-memory data store, but I want to focus on PhantomReferences, so we’ll ignore these details for now. dataStore is an instance of BufferedDataStore, which has the following method signature:

public long newObjectEvent(Object o, long time) throws NameNotFoundException;

This method takes the newly instantiated object as a parameter and the time the object was created. The method returns the a long value representing a unique identifier for the object’s type. The next step is to create the PhantomReference. We pass a ReferenceQueue into the PhantomReference's constructor which registers it as an object we want to be notified of when it is garbage collected. Lastly, we store the reference and its associated class id in a map. These lines will make more sense after looking at the code that listens to the queue

static class RemoverThread extends Thread {
    public RemoverThread() {
        setDaemon(true);
    }

    public void run() {
        while (alive) {
            PhantomReference ref = (PhantomReference) queue.remove();
            Long classId = map.remove(ref);
            dataStore.objectGcEvent(classId, System.currentTimeMillis());

        }
    }
}

This is a class that is defined inside the Recorder class. It is a daemon thread, which means that it will not prevent the JVM from exiting even if it is still alive. The run method contains a while loop that will run forever unless the stop method is called to change the alive property. The ReferenceQueue.remove()method blocks until there is a PhantomReference to remove. Once a PhantomReference appears, we look up the classId from the map. Then we record the event by calling the dataStore's objectGcEvent method.

We have now covered how to instrument application classes to insert theRecorder.record(…) method, how to create PhantomReferences for these objects, and how to respond to their garbage collection events. We now have the ability to record when objects are created and when they are garbage collected. Once this core functionality is established, you can implement your leak detector in a variety of ways. This code base uses an in-memory data store. This type of data source consumes the same heap space as your application to store data, so it is not recommended for long term leak detection (in other words, it is a memory leak itself!). A more sensible option for long term detection would be to store the data in a real database.

Another aspect of this leak detector is the approach to identifying a leak. Some leak detectors may tell you “I found a leak!”, but this one does not. It provides you with a graph of the top leak candidates and allows the user to evaluate which objects are in fact leaky. This means you have to be proactive about discovering leaks. However, this code could easily be improved upon. You could develop an algorithm for isolating memory leaks that informs the user reactively. There are also other possible improvements. The user interface for this tool is a line graph of the objects with the most potential for leaking. It is quite normal for object counts to climb when there is plenty of memory. Hence, one improvement would be be to record and draw the major garbage collections on the graph. Knowing that potentially leaky objects survive garbage collections is a very good indicator of a memory leak.

There is a lot of code in this project we have not covered. For instance, the code to find trends, to graph the data, or how to query the data was not covered. The intent of this article was to demonstrate how memory leak data can be collected, which makes unrelated code outside of the scope. However, there is one more topic we’ll cover, which is how to configure and run this software.

The type of data store used by the agent in the target JVM is determined by a data-store-class property in the configuration file. For now, there is only an in-memory implementation, ca.discotek.heapster.datastore.MemoryDataStore, for storing leak data. As is, it is a terrible idea because it is a leak itself. It does not have an eviction policy and will eventually cause an OutOfMemoryError. When the MemoryDataStore initializes, it sets up a server socket, which clients can use to request data. The MemoryDataStore uses the configuration file to get the server port number. It also uses it to set the log level (which you probably don’t need to adjust, but valid values are trace, info, warn, and error.). The inclusion property is a Java language regular expression used to specify your application classes to be instrumented for memory leak detection. You can also specify an exclusion property to exclude name spaces from the inclusion property.

To connect to your server, you’ll need to run a client. There is a generic ca.discotek.heapster.client.gui.ClientGui class, which looks up the client-class property in a configuration file. It instantiates an instance and uses it to communicate with the server. Since our agent is configured to use the MemoryDataStore class, we want our client to the ca.discotek.heapster.client.MemoryClient to connect to the MemoryDataStore server. The MemoryClient class looks up the server port in the configuration file. To keep my configuration simple, I put both the server and client properties in one test.cfg configuration file. If your target JVM is on a different machine than your client, you’d have to have separate configuration files. Here is what I have been using:
I created a ca.discotek.heapster.client.MemoryClient class. This client uses the configuration file to look up the port

client-class=ca.discotek.heapster.client.MemoryClient
data-store-class=ca.discotek.heapster.datastore.MemoryDataStore
log-level=info
server-port=8888
inclusion=ca\.discotek\.testheapster\..*

The inclusion property specify the namespace of my test app called LeakTester, which can create various types of objects in various ways. Here’s a screen shot:

In order to override the JVM’s java.lang.ClassLoader, we’ll generate our own bootstrap jar and use the -Xbootclasspath/p JVM flag to insert our bootstrap jar at the end of bootclasspath. We will have to perform this task for ever different JRE version of target JVM. There may be internal API changes between versions that would break compatibility if you used the generated ClassLoader class from JRE X with JRE Y.

Let’s assume you have downloaded the memory leak distribution and extracted it to /temp/heapster. Let’s also assume that your target JVM JRE version is 1.6.0_05. First, will create the directory/temp/heapster/1.6.0_05 to house the jar we are about to generate. Next we’ll run the following command:

java -jar /temp/heapster/heapster-bootpath-generator.jar /java/jdk1.6.0_05/jre/lib/rt.jar /temp/heapster/1.6.0_05

The second and third program arguments specify the location of the rt.jar of the target JVM and the location where you want to store the generated jar. This command will generate a heapster-classloader.jar in /temp/heapster/1.6.0_05.

Assuming you want to run the LeakTester app bundled with this project, you would run the following command:

/java/jdk1.6.0_05/bin/java -Xbootclasspath/p:/temp/heapster/1.6.0_05/heapster-classloader.jar -javaagent:/temp/heapster/discotek-heapster-agent.jar=/temp/heapster/config/test.cfg ca.discotek.testheapster.LeakTester

Next up, let’s run the client:

/java/jdk1.6.0_05/bin/java -classpath /temp/heapster/discotek-heapster-client.jar;/temp/heapster/discotek-graph-1.0.jar ca.discotek.heapster.client.gui.ClientGui /temp/heapster/config/test.cfg

You should now see a window similar to the screen shot above. However, that screen shot uses the Plumbr sample application, not my LeakTester app. If you’d like to see a graph of the Plumbr sample app, you can do the following:

1. Get the Plumbr sample app running as per their instructions.
2. Open the demo/start.bat file in an editor.
3. In the Java command line toward the bottom, place -agentlib:plumbr -javaagent:..\..\plumbr.jar with -Xbootclasspath/p:/temp/heapster/1.6.0_05/heapster-classloader.jar -javaagent:/temp/heapster/discotek-heapster-agent.jar=/temp/heapster/config/test.cfg
4. Save your changes.
5. Open /temp/heapster/config/test.cfg in an editor.
6. Change the inclusion property to inclusion=.*petclinic.*
7. Save your changes.
8. Run the Plumbr sample app as you did before.
9. Start the ClientGui using the exact same command line as we used for the LeakTester scenario.

Note, you can generate traffic with the Plumbr demo two ways: 1. Use the create_usage.bat to drive traffic with JMeter, or 2. open the app in a browser (http://localhost:18080). I recommend you use a browser so you can control the traffic and observe the consequences of each page refresh.

This article has been an introduction into how one might find memory leaks with instrumentation and PhantomReferences. It is not meant to be a complete product. The following features could be added to improve the project:

1. Indicate major garbage collections on the graph
2. Allow for stack traces to be collected when leaky objects are instantiated to reveal the buggy sourc code
3. Store the instantiation and garbage collection data in a database to avoid the application becoming a memory leak itself
4. ClientGui could graph the available heap and permgen (similar to JConsole) (which might be useful to cross reference against the object graphs)
5. Provide a mechanism to expunge instantiation and garbage collection data

If you liked this article and would like to read more, see other byte code engineering articles at Discotek.ca or follow Discotek.ca on twitter to be notified when my next article is ready on how to instrument classes to gather performance statistics!

Resources

• Binaries and source for the leak detector code.