Memory Leak Protection in Tomcat 7

 the last time dzone spoke with mark thomas, a senior software engineer with springsource and committer for apache tomcat, he said that tomcat has had a long history of memory leaks when reloading web applications.  this topic could easily fill its own article, he said.  in his  overview of the features  coming in tomcat 7, thomas said that the new version, which is expected to have an alpha release next month, will address many of the memory leaks caused by web applications, their libraries, and even the jvm.  he said that a small percentage of the leaks were caused by tomcat, but those bugs were fixed several years ago.  now the challenge has been to come up with workarounds for the vast array of memory leak sources.  in this exclusive interview with dzone, thomas elaborates on these fixes coming in the next version of apache tomcat. 

 dzone:  what typically causes a memory leak on web application reload and what symptoms might a developer see? 

 mark thomas:  the usual symptom observed is an outofmemoryerror (oome) indicating that the permanent generation (permgen) is exhausted when reloading a web application.

permgen is where the jvm stores classes. classes are stored in permgen using class name and class loader as the unique identifier. each web application has its own class loader; this allows different versions of the same class (with the same name) to be used in different web applications without conflict. a web application also gets a new class loader when it is reloaded. classes are removed from permgen when the class loader is collected by the garbage collector. this normally happens after a web application stops. however, if something retains a reference to the web application class loader, it won't be garbage collected and the classes it loaded will remain in permgen. since permgen is limited in size, this usually only has to happen a few times before all the permgen is used. that is when the oome occurs.

to prevent the oome, it is necessary to ensure that nothing retains a reference to the web application class loader.

 dzone:  tell me about the tomcat's history with memory leaks on web application reload.

 mark:  memory leaks on web application reload have been a problem for as long as i have been involved in the tomcat project. where the problem has been traced to a bug in the tomcat code base, it has been fixed. there were a handful of such bugs fixed around the time of tomcat 4.1.x / tomcat 5.5.x that regularly caused problems. since then there have been a few edge case bugs that occasionally caused problems and, again, these were fixed as they were found.

historically, tomcat has been blamed for causing these memory leaks but when investigated by the tomcat developers, it usually emerged that the root cause was a bug in the web application or the library rather than tomcat.

around twelve months ago i was giving a presentation on tomcat tuning and made an off-hand comment that the root cause of memory leaks on web application reload was application and/or library bugs rather than a tomcat bug. this comment generated rather more feedback than i was expecting and i was soon working with a number of the attendees debugging their memory leaks.

the leaks that were traced to application or library bugs weren't unexpected. however, we also traced some leaks to usage of particular standard java apis and this was a surprise. as we worked to develop fixes for these, i realised that these fixes would be best placed as part of the standard tomcat codebase. that became the starting point for the new memory leak protection.

the memory leak protection was enhanced by taking a number of web applications known to have memory leaks on reload and developing fixes and workarounds for each of the memory leaks found until the applications could be reloaded without triggering a memory leak. the combination of these fixes and workarounds became the new memory leak protection features that will be included in tomcat 7 and has recently been back-ported to tomcat 6.0.x (6.0.24 onwards).

 dzone:  so have all of the bugs causing memory leaks on tomcat's end been fixed? 

 mark:  all the ones we know about!  there are probably a few rarely encountered bugs remaining. these too, will get fixed as they are reported.

 tomcat's jvm memory profile

 dzone:  what are some of the bugs in web applications, their libraries, and the jvm that have caused memory leaks? 

 mark:  the memory leaks all fit a common pattern. the web application creates an object of a class loaded by the web application class loader and then places this object in a registry that has been loaded by a different

class loader. this is fine, as long as the object is removed from the registry when the web application stops. if the object is not removed, the registry retains a reference to the object, the object retains a reference to the class and the class retains a reference to the class loader. this pins the class loader in memory. this chain of

references prevents the class loader from being garbage collected which in turn means that the permgen used to store classes loaded by this class loader cannot be freed.

application or library code that can trigger this situation include:

• jdbc driver registration
• some logging frameworks
• storing objects in threadlocals and not removing them
• starting threads and not stopping them

there are also a number of places that using the standard java api can trigger a similar issue. these include:

• using the javax.imageio api (the google web toolkit can trigger this)
• using java.beans.introspector.flushcaches() (tomcat does this to prevent memory leaks caused by this caching)
• using xml parsing (the root cause is unknown due to a bug in the  jre  )
• using rmi (somewhat ironically, causes a leak related to the garbage collector)
• reading resources from jar files

 dzone:  tell me about the work-arounds that tomcat 7 will have for these bugs. 

 mark:  for the application or library code problems, tomcat fixes problems where it can by de-registrating the object(s) that were registered by the application or library. tomcat will also log any action taken so the web application developer can fix the root cause of the problem, rather than relying on tomcat to fix the leak. this is especially important for threads that are not stopped since tomcat can not do this safely.

the details of how tomcat fixes these problems may be found in the clearreferences() method of the  webappclassloader class  .

for the java api usages that can trigger a memory leak, these are all triggered if the first call to that api is made by a web application. tomcat, therefore, prevents these leaks by ensuring that the core tomcat code is the first to call these apis, thereby making them safe for web applications to call.

the details of this protection can be in the  jreleakpreventionlistener class  .

the one exception to this is the java.util.logging implementation (juli). the logging framework provided by the jre is not class loader aware and can quickly lead to memory leaks in a servlet container environment. to address this, juli replaces the standard logmanager with one that is class loader aware. this allows java.util.logging to be used without triggering a memory leak.

 dzone:  how much memory leak improvement do you think tomcat 7 will have overthe current version?

 mark:  the new features in tomcat 7 and 6.0.24 onwards should offer significant improvements. however, it is likely that there are other causes of memory leaks since the investigations so far have only used a small number of web applications for testing. the new features in tomcat should make it easier to track own these additional sources of memory leaks and, as they are identified, protection against them can be added to tomcat.

 dzone:  how are things progressing on tomcat 7 development and where is the community at right now as far as the roadmap is concerned?  is there a ballpark for it's release date? 

 mark:  things are progressing well. the jsp & el 2.2 implementation is complete and the servlet 3.0 implementation is close to completion. i had hoped that tomcat 7 would be ready for a first release by the end of january but it now looks as if it will happen in february.

 dzone:  is there anything else you'd like to mention about tomcat 7? 

 mark:  there has been lots of clean-up and other tweaks in tomcat 7 but the one thing that i think is worth mentioning is improved support for embedding tomcat. whilst tomcat has always been embeddable, it has been rather more complex than it needs to be. with tomcat 7 we will be providing a simple api and a download option that provides all the tomcat features but with a fewer number of jars. together, these changes will make it much easier to embed tomcat in other applications.

finally, the tomcat community is always looking for new contributors. whether it is answering questions on the users list, providing translations, fixing bugs, developing new features, improving the web site or something else, contributions are always welcome. the first step for people interested in contributing is to join the user and/or dev  mailing list  and let the tomcat community know how they can help.