No More Leaks: Detecting Memory Leaks in Cloud Applications with Plumbr and Stackato

[This article was written by John Wetherill]

At JavaOne I had the pleasure of meeting Nikita Salnikov-Tarnovski, founder of Plumbr, an exciting startup from Estonia. Nikita and his team have created a tool which I believe should be categorized as “essential” for cloud developers that build apps based on the JVM. This article describes Plumbr, and shows how to quickly enable Plumbr memory leak detection in any Stackato-deployed Java or JVM cloud application.

Why Avoid Leaks?

First let’s talk about leaks. Memory leaks are: 

• unpredictable, often showing up at the worst possible time.
• evasive, difficult to track down, and even more difficult to reproduce
• costly, resulting in countless hours of wasted productivity, lost revenue, and decreased moral
• damaging to reputation, brand, inter-team and intra-team relationships, revenue
• dangerous: the wrong leak at the wrong time and place can sink a ship or a company.

So leaks are bad, but they’re also very common. According to Nikita, 47% of Java applications contain memory leaks.

We are fortunate that memory leaks, like most chronic problems in the software world, have attracted the attention of The People Who Like To Fix Things, who over the years have generously given us many ingenious tools and practices to help combat leaks.

Detecting Leaks

Tools and practices to detect and prevent leaks have been around for a long time. By necessity a cloud developer needs a selection of tools to combat leaks, and they come in many forms, including:

Static code analysers

Klocwork or JetBrain’s On-the-fly code analyser included with IntelliJ IDEA.

Platform Capabilities

Java’s garbage collection (GC) is often considered as a platform capability that can help reduce memory leaks by automating the release of unreferenced objects. But GC is not a silver bullet. In fact it can be argued that GC distances the developer from thinking about leaks, which then causes leaks. Certainly Java’s GC doesn’t prevent leaks, which is the whole reason behind tools like Plumbr.

Language Constructs and Idioms

Consider this Ruby code snippet:

File.open(“textfile”, ‘w’) {|f| f.write(“hi”) }

This ensures that the allocated resource (a file in this case) is immediately and automatically released when its reference goes out of scope. Clojure and many other languages provide similar constructs that help developers “remember” to close or free resources.

Coding conventions and practices

When coding, a good habit to get into is to type the code to close a resource immediately after opening it, before typing any code that consumes the resource.

Testing frameworks, tools and libraries

Loads of tools and libraries are available to help detect and minimize leaks. jMeter and apache bench are handy ways to stress-test an application scare out any leaks that might be lurking.

Monitoring Tools, Instrumentation Libraries

Products like New Relic, Nagios, Riemann, and facilities like JMX are invaluable for leak and performance monitoring.

Developer Tools

The popular profiler YourKit can help discover and fix leaks in Java and .NET code.

Logging

Logging, the old standby, can be helpful to help discover and fix leaks. To use logging effectively in the cloud, a log aggregation capabilities and tools like Logyard, PaaS, and Loggly are invaluable.

Assembly Required

Each of the above tools/practices takes some investment to employ, such as the time to learn, or configure, or implement, or educate, or automate the use of such tools.

Not so Plumbr. This amazing tool asks so little yet provides so much. It installs in seconds, and will monitor any Java or JVM-based app with ease. When a leak is detected it notifies the deployer, providing general or detailed information about the leak. I’ll cover steps to use Plumbr with Stackato below, but first it’s illustrative to look at the inner workings.

How does Plumbr work?

Plumbr makes use of the handy, but not widely used, java.lang.instrument package which tweaks the bytecode of compiled Java and JVM apps to enable instrumentation and measurement of various app characteristics.

Plumbr uses this package to observe the lifecycle of all of the objects created by an app, including when/where they were created, when they were freed by GC, and their memory consumption characteristics over time. But Plumbr is much more than just tracking objects and memory: they also have a sophisticated cloud-based analytics system which learns over time the signature and patterns of common leaks across large classes of applications, then uses this information to further refine their leak-recognition. In other words, Plumbr gets better over time.

After a leak is detected they then provide a detailed root-cause analysis of the leak, with specifics on how to fix it.
Nikita made it clear to me that Plumbr doesn’t show false negatives: i.e. Plumbr will detect all leaks. It might also report leaks in code where the increase in memory usage is by design (i.e., false positives) but these are usually easily recognized by the application developer. Also, as mentioned above, Plumbr is getting smarter over time so presumably the false positives are decreasing.

Leak Detection in the Cloud With Stackato

Cloud apps that scale are particularly sensitive to leaks, and therefore cloud app developers will want to use any tools at their disposal to detect and be aware of any leaks before they disrupt the application.

Fortunately this process is trivially simple with Plumbr and Stackato. Here are the steps. Please try this at home.

1. Sign up to the Plumbr portal. Only two things are required: an email address and your choice of password.
2. Download the agent. The important part of this is the “plumbr.jar” file, which conveniently includes your specific auto-generated Plumbr API keys.
3. This is one of several ways that Plumbr strives to simplify life for the developer. Many other APIs and services rely on API keys, but require copying/pasting the key into a script or app to use it. With Plumbr auto-generates your key when you signup, and embeds this key in the jarfile, so the key creation is completely seamless and hidden.
4. Add the jarfile from step 2 to your application directory structure.
5. As with any Stackato-deployed app, create a stackato.yml file, and add this directive to enable Plumbr

env: JAVA_OPTS: -javaagent:plumbr.jar

stackato push -n