DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

How does AI transform chaos engineering from an experiment into a critical capability? Learn how to effectively operationalize the chaos.

Data quality isn't just a technical issue: It impacts an organization's compliance, operational efficiency, and customer satisfaction.

Are you a front-end or full-stack developer frustrated by front-end distractions? Learn to move forward with tooling and clear boundaries.

Developer Experience: Demand to support engineering teams has risen, and there is a shift from traditional DevOps to workflow improvements.

Related

  • Troubleshooting Memory Leaks With Heap Profilers
  • All You Need To Know About Garbage Collection in Java
  • Performance Engineering Management: A Quick Guide
  • Java Memory Management

Trending

  • Supercharge Your Java Apps With AI: A Practical Tutorial
  • What They Don’t Teach You About Starting Your First IT Job
  • How I Built an AI Portal for Document Q and A, Summarization, Transcription, Translation, and Extraction
  • How to Improve Software Architecture in a Cloud Environment
  1. DZone
  2. Coding
  3. Languages
  4. OpenJPA: Memory Leak Case Study

OpenJPA: Memory Leak Case Study

By 
Pierre - Hugues Charbonneau user avatar
Pierre - Hugues Charbonneau
·
Mar. 21, 13 · Interview
Likes (0)
Comment
Save
Tweet
Share
9.0K Views

Join the DZone community and get the full member experience.

Join For Free
This article will provide the complete root cause analysis details and resolution of a Java heap memory leak (Apache OpenJPA leak) affecting an Oracle Weblogic server 10.0 production environment.
This post will also demonstrate the importance to follow the Java Persistence API best practices when managing the javax.persistence.EntityManagerFactory lifecycle.
Environment specifications
  • Java EE server: Oracle Weblogic Portal 10.0
  • OS: Solaris 10
  • JDK: Oracle/Sun HotSpot JVM 1.5 32-bit @2 GB capacity
  • Java Persistence API: Apache OpenJPA 1.0.x (JPA 1.0 specifications)
  • RDBMS: Oracle 10g
  • Platform type: Web Portal
Troubleshooting tools
  • Quest Foglight for Java (Java heap monitoring)
  • MAT (Java heap dump analysis)
Problem description & observations
The problem was initially reported by our Weblogic production support team following production outages. An initial root cause analysis exercise did reveal the following facts and observations:
  • Production outages were observed on regular basis after ~2 weeks of traffic.
  • The failures were due to Java heap (OldGen) depletion e.g. OutOfMemoryError: Java heap space error found in the Weblogic logs.
  • A Java heap memory leak was confirmed after reviewing the Java heap OldGen space utilization over time from Foglight monitoring tool along with the Java verbose GC historical data.
Following the discovery of the above problems, the decision was taken to move to the next phase of the RCA and perform a JVM heap dump analysis of the affected Weblogic (JVM) instances.
JVM heap dump analysis

** A video explaining the following JVM Heap Dump analysis is now available here.
In order to generate a JVM heap dump, the supported team did use the HotSpot 1.5 jmap utility which generated a heap dump file (heap.bin) of about ~1.5 GB. The heap dump file was then analyzed using the Eclipse Memory Analyzer Tool. Now let’s review the heap dump analysis so we can understand the source of the OldGen memory leak.
MAT provides an initial Leak Suspects report which can be very useful to highlight your high memory contributors. For our problem case, MAT was able to identify a leak suspect contributing to almost 600 MB or 40% of the total OldGen space capacity.
At this point we found one instance of java.util.LinkedList using almost 600 MB of memory and loaded to one of our application parent class loader (@ 0x7e12b708). The next step was to understand the leaking objects along with the source of retention.
MAT allows you to inspect any class loader instance of your application, providing you with capabilities to inspect the loaded classes & instances. Simply search for the desired object by providing the address e.g. 0x7e12b708 and then inspect the loaded classes & instances by selecting List Objects > with outgoing references.
As you can see from the above snapshot, the analysis was quite revealing. What we found was one instance of org.apache.openjpa.enhance.PCRegistry at the source of the memory retention; more precisely the culprit was the _listeners field implemented as a LinkedList.
For your reference, the Apache OpenJPA PCRegistry is used internally to track the registered persistence-capable classes. Find below a snippet of the PCRegistry source code from Apache OpenJPA version 1.0.4 exposing the _listeners field.

/**
 * Tracks registered persistence-capable classes.
 *
 * @since 0.4.0
 * @author Abe White
 */
publicclass PCRegistry {
  // DO NOT ADD ADDITIONAL DEPENDENCIES TO THIS CLASS
  privatestaticfinal Localizer _loc = Localizer.forPackage
  (PCRegistry.class);
  // map of pc classes to meta structs; weak so the VM can GC classes
  privatestaticfinal Map _metas = new ConcurrentReferenceHashMap
  (ReferenceMap.WEAK, ReferenceMap.HARD);
  // register class listeners
  privatestaticfinal Collection _listeners = new LinkedList();
……………………………………………………………………………………
Now the question is why is the memory footprint of this internal data structure so big and potentially leaking over time? The next step was to deep dive into the _listeners LinkedLink instance in order to review the leaking objects.
We finally found that the leaking objects were actually the JDBC & SQL mapping definitions (metadata) used by our application in order to execute various queries against our Oracle database. A review of the JPA specifications, OpenJPA documentation and source did confirm that the root cause was associated with a wrong usage of the  javax.persistence.EntityManagerFactory such of lack of closure of a newly created EntityManagerFactory instance.
If you look closely at the above code snapshot, you will realize that the close() method is indeed responsible to cleanup any recently used metadata repository instance. It did also raise another concern, why are we creating such Factory instances over and over…
The next step of the investigation was to perform a code walkthrough of our application code, especially around the life cycle management of the JPA EntityManagerFactory and EntityManager objects.
Root cause and solution
A code walkthrough of the application code did reveal that the application was creating a new instance of EntityManagerFactory on each single request and not closing it properly.
 public class Application {
      
       @Resource
       private UserTransaction utx = null;
       
       // Initialized on each application request and not closed!
       @PersistenceUnit(unitName = "UnitName")
       private EntityManagerFactory emf = Persistence.createEntityManagerFactory("PersistenceUnit"); 

       public EntityManager getEntityManager() {
             return this.emf.createEntityManager();
       }
      
       public void businessMethod() {
            
             // Create a new EntityManager instance via from the newly created EntityManagerFactory instance
             // Do something...
             // Close the EntityManager instance
       }
}

This code defect and improver use of JPA EntityManagerFactory was causing a leak or accumulation of metadata repository instances within the OpenJPA _listeners data structure demonstrated from the earlier JVM heap dump analysis.
The solution of the problem was to centralize the management & life cycle of the thread safe javax.persistence.EntityManagerFactory via the Singleton pattern. The final solution was implemented as per below:
  • Create and maintain only one static instance of javax.persistence.EntityManagerFactory per application class loader and implemented via the Singleton Pattern.
  • Create and dispose new instances of EntityManager for each application request.
Please review this discussion from Stackoverflow as the solution we implemented is quite similar.
Following the implementation of the solution to our production environment, no more Java heap OldGen memory leak is observed.
Please feel free to provide your comments and share your experience on the same.


 

Memory (storage engine) garbage collection Java (programming language) application Data structure Dump (program) Apache OpenJPA

Published at DZone with permission of Pierre - Hugues Charbonneau, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Troubleshooting Memory Leaks With Heap Profilers
  • All You Need To Know About Garbage Collection in Java
  • Performance Engineering Management: A Quick Guide
  • Java Memory Management

Partner Resources

×

Comments

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • [email protected]

Let's be friends: