DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations
Building Scalable Real-Time Apps with AstraDB and Vaadin
Register Now

Trending

  • How to Submit a Post to DZone
  • Avoiding Pitfalls With Java Optional: Common Mistakes and How To Fix Them [Video]
  • Extending Java APIs: Add Missing Features Without the Hassle
  • Auditing Tools for Kubernetes

Trending

  • How to Submit a Post to DZone
  • Avoiding Pitfalls With Java Optional: Common Mistakes and How To Fix Them [Video]
  • Extending Java APIs: Add Missing Features Without the Hassle
  • Auditing Tools for Kubernetes
  1. DZone
  2. Coding
  3. Languages
  4. RMI Enforcing Full GC To Run Hourly

RMI Enforcing Full GC To Run Hourly

Nikita Salnikov-Tarnovski user avatar by
Nikita Salnikov-Tarnovski
·
Jan. 10, 14 · Interview
Like (0)
Save
Tweet
Share
6.51K Views

Join the DZone community and get the full member experience.

Join For Free

during all the troubleshooting exercises i have carried out in my career, i have picked up a feeling that the bugs i am after are evolving and becoming meaner and uglier over time. or maybe it is just my age kicking in. this particular heisenbug – lookalike this post is about has again kept me awake for more than i would actually have liked to.

as other particularly nasty bugs, the one i was now after surfaced with a symptom that “sometimes the system is slow”. great, i already felt the shivers going down my spine. things got a bit better when i discovered that the system had a simple monitoring solution set up in production, monitoring the system. essentially it was not much more than pingdom , measuring the response time once a second, but the latency graphs immediately caught my eye.

load peaking jvm

there were seemingly random spikes in response times plotted in the graphs – every once in a blue moon the responses started to take almost 10x more time to compose. and within a short while everything was back to normal again.

first suspects – regular cron jobs or expensive reporting processes were nowhere in sight. neither was the storage monitoring exposing any particularly ugly queries hitting the databases. but before calling it a night, i added some more logging options to the application configuration, rebooted the whole stuff and went to bed puzzled about the behaviour.

next morning i had something extra to look at, gc logs among the other things. within ten minutes i found myself staring at something like the following:

11.408: [full gc [psyounggen: 192k->0k(48640k)] [paroldgen: 16324k->14750k(114688k)] 16516k->14750k(163328k) [pspermgen: 31995k->31637k(64000k)], 0.1543430 secs] [times: user=0.58 sys=0.00, real=0.16 secs]
...
3613.362: [full gc [psyounggen: 256k->0k(47104k)] [paroldgen: 15142k->13497k(114688k)] 15398k->13497k(161792k) [pspermgen: 32429k->32369k(72192k)], 0.1129070 secs] [times: user=0.51 sys=0.00, real=0.11 secs]
...
7215.278: [full gc [psyounggen: 224k->0k(44544k)] [paroldgen: 13665k->13439k(114688k)] 13889k->13439k(159232k) [pspermgen: 32426k->32423k(70144k)], 0.0881450 secs] [times: user=0.44 sys=0.00, real=0.09 secs]
...

i could now correlate the seemingly random performance degradation to the full gc running each time around 3,600 seconds has passed. attaching visualvm to the jvm left me puzzled – there was no evidence of high memory usage which should be causing the full garbage collector to run. checking with the developers i made sure they were not running explicit gcs via cron jobs. so there i stood, confused by the regularity of the gc.

what does an engineer do when he is puzzled by a question? he googles . and there it was, staring right into my face. pages after pages with people who are concerned about the hourly full gc taking place without any apparent reason.

and the culprit – rmi. apparently when your application either exposes its services via rmi or consumes any services over rmi, you are bound to have an additional garbage collection cycle. as the rmi documentation says:

“when it is necessary to ensure that unreachable remote objects are unexported and garbage collected in a timely fashion, the value of this property represents the maximum interval (in milliseconds) that the java rmi runtime will allow between garbage collections of the local heap. the default value is 3600000 milliseconds (one hour).”

solution for the time being was increasing the sun.rmi.dgc.server.gcinterval length from the default 3,600 seconds. i wonder how things used to look prior to the change introduced in jdk 6, when rmi used to force a full gc once a minute. considering all the ejb frenzy back then, i guess none of the apps even stood a chance performance-wise. if you have memories from the ancient times, maybe you can shed some light upon how the apps were able to survive this madness.

and – if the content was interesting, consider subscribing to our twitter feed to be alerted on future posts.


garbage collection

Published at DZone with permission of Nikita Salnikov-Tarnovski, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Trending

  • How to Submit a Post to DZone
  • Avoiding Pitfalls With Java Optional: Common Mistakes and How To Fix Them [Video]
  • Extending Java APIs: Add Missing Features Without the Hassle
  • Auditing Tools for Kubernetes

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com

Let's be friends: