Our series of posts about solving the OutOfMemoryError in our hypothetical production system is coming to a conclusion. We have covered a number of different ways to attack OutOfMemoryErrors so far - profilers, JDK bundled tools and heap dump analyzers. Our today’s mercenaries will be Application Performance Management tools, or APMs.
APM are positioned as the Holy Grails on a quest for a solution of your production environment’s performance problems. You just setup APM tool of your choice, let it monitor your whole cluster from front-end load-balancer down to your Oracle or Neo4j database and then relax in that Aeron chair of yours. APM provides you with all the information your operations or developers need in order to achieve a desired level of customer satisfaction.
If you thought there was a grain of sarcasm in what I have just said, you are right. I was always suspicious about Jacks of all trades. But let us cast all the doubts aside and just try them out.
Our fist list of APMs to try consisted of 5 names: AppDynamics, CA APM (former Introscope),dynaTrace, HPjmeter and New Relic. HPjmeter fell away at once, as it is available for HP-UX platforms only. New Relic has no memory leak tracking capabilities yet, so there we lost a second contestant. For CA APM and dynaTrace we were unable to obtain a free evaluation version. This left us with AppDynamics alone. Kudos for the AppDynamics team who gladly provided us the opportunity to try the solution!
AppDynamic’s installation consists of AppDynamics Controller, a central web-based dashboard, collecting and processing data from Java Application Server Agents, which are JVM agents, just like Plumbr. These agents can be attached to different servers and applications. Attached agents send runtime information to Controller. For our sample application I’ve got the following view in AppDynamics Controller:
If we dive a little deeper, we can see memory related data per node:
Screenshot above is only a subset of the information provided on the screen. It is scrollable all the way down for much more graphs and trends and data. Which is all nice, but not exactly what we were looking for.
As our goal was to let AppDynamics help us finding a root cause of memory leak we have suffered for so long we switch to “Automatic Leak Detection” tab and activate it (it is switched off by default). Then we start “On Demand Capture Session” and let the application run under the load of our stress tests. Results are... a bit disappointing. I have tried this several times with different parameter values, that AppDynamics allows to configure. Result was the same:
All while my application was crashing with OOM. No luck here today.
It was very difficult for me to write this post. First of all I had only one tool to test, and to get even this one, was not an easy task. Second, I wasted several hours before I saw a first bit of information in AppDynamics dashboard (do not ask, I will not write about it). And third, I have no results to show my readers. But here we are and what can I conclude from this experience?
- APM tools require planning. If you have fire in the house, it is too late to go fetching them. Like in case of life insurance you should have thought about them way before the moment you really need them.
- APM tools give you tons of information about inner workings of your application performance-wise. But I was hoping for something answering my question ”Why do I have memory leak and what should I do?” more directly.
- All in all, APM can be great tool for monitoring and proactive planning of performance related maintenance of your application. I cannot recommend it for problem solving.
P.S. If you can point to any other tool for solving memory problems in Java applications that we have missed, or can provide us with evaluation licenses of aforementioned products, we will be glad to hear from you. Just contact us at firstname.lastname@example.org or @JavaPlumbr.