Try to Avoid -XX:+UseGCLogFileRotation

Developers take advantage of the JVM argument -XX:+UseGCLogFileRotation to rotate GC log files.

Example:

"-XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/home/GCEASY/gc.log -
XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=20M"

As shown above, the JVM will rotate the GC log file whenever its size reaches 20MB. It will generate up to five files, with extensions gc.log.0,  gc.log.1, gc.log.2, gc.log.3, and gc.log.4.

But this approach has a few challenges:

Losing Old GC Logs

Suppose you configured  -XX:NumberOfGCLogFiles=5, then over a period of time, five GC log files will be created:

• gc.log.0 ← oldest GC Log content

• gc.log.1

• gc.log.2

• gc.log.3

• gc.log.4 ← latest GC Log content

The most recent GC log contents will be written to gc.log.4 and old GC log contents will be present in gc.log.0.

When the application starts to generate more GC logs than the configured  -XX:NumberOfGCLogFiles, in this case, five, then old GC log contents in gc.log.0 will be deleted. New GC events will be written to  gc.log.0. It means that you will end up not having all the generated GC logs. You will lose the visibility of all events.

Mixed-Up GC Logs

Suppose an application has created five GC log files, including:

• gc.log.0

• gc.log.1

• gc.log.2

• gc.log.3

• gc.log.4

Then, let’s say you are restarting the application. Now, new GC logs will be written to gc.log.0 file and old GC log content will be present in gc.log.1, gc.log.2, gc.log.3, gc.log.4, etc.

• gc.log.0 ← GC log file content after restart

• gc.log.1 ← GC log file content before restart

• gc.log.2 ← GC log file content before restart

• gc.log.3 ← GC log file content before restart

• gc.log.4 ← GC log file content before restart

So, your new GC log contents get mixed up with old GC logs. Thus, to mitigate this problem, you might have to move all the old GC logs to a different folder before you restart the application.

Forwarding GC Logs to a Central Location

In this approach, the current active file to which GC logs are written is marked with the extension  .current. For example, if GC events are currently written to the file gc.log.3, it would be named as: gc.log.3.current.

If you want to forward GC logs from each server to a central location, then most DevOps engineers use  rsyslog. However, this file naming convention poses a significant challenge to use rsyslog, as described in this blog.

Tooling

Now, to analyze the GC log file using the GC tools such as (GCeasy, GCViewer, etc.), you will have to upload multiple GC log files instead of just one single GC Log file.

Recommended Solution

We can suffix the GC log file with the time stamp at which the JVM was restarted, then the GC Log file locations will become unique. Then, new GC logs will not override the old GC logs. It can be achieved by suffixing %t to the GC log file name, as shown below:

"-XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/home/GCEASY/gc-%t.log"

 %t suffixes timestamp to the GC log file in the format:  YYYY-MM-DD_HH-MM-SS. So, the generated GC log file name will start to look like: gc-2019-01-29_20-41-47.log.

This simple solution addresses all the shortcomings of -XX:+UseGCLogFileRotation.