Rotating Garbage Collection Logs

Garbage collection logs are essential artifacts to optimize application’s performance and troubleshoot complex memory problems. They can be generated in a particular file path by passing the -Xloggc JVM argument.

Example: -Xloggc:/home/GCEASY/gc.log

However, the challenge to this approach is that whenever the application is restarted, the old GC log file will be over-ridden by the new GC log file as the file path is same (i.e., /home/GCEASY/gc.log).

Thus, you wouldn’t be able to analyze the old GC logs that existed before restarting the application. If the application has crashed or had certain performance problems, then you especially need old GC logs for analysis.

Because of the heat of the production problem, most of the time, IT and DevOps teams forget to back up the old GC log file. This is a classic problem that happens again and again, that we all are familiar with. Most human errors can be mitigated through automation and this problem is no exception to it.

A simple strategy to mitigate this challenge is to write new GC log contents in a different file location. Here are two different strategies to do that are shared with you:

1. Suffix Timestamp to GC Log File

If you can suffix the GC log file with the time stamp at which the JVM was restarted, then GC log file locations will become unique. Then, new GC logs will not override the old GC logs. This can be achieved as shown below:

Unix

"-XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/home/GCEASY/gc-$(date +%Y_%m_%d-%H_%M).log"

Windows

# Generating time stamp & replacing '' with 0 and / with -
set file_suffix=%DATE:~-4%-%DATE:~4,2%-%DATE:~7,2%_%TIME:~0,2%-%TIME:~3,2%-%TIME:~6,2%
set file_suffix=%file_suffix: =0%
set file_suffix=%file_suffix:/=-%

set JAVA_OPTS=%JAVA_OPTS% -XX:+PrintGCDetails -XX:+PrintGCDateStamps
 -Xloggc:c:\workspace\newUploads\gc-%file_suffix%.log

This strategy has one minor drawback: growing file size.

If you don’t restart your JVMs, then the GC log file size can grow to a huge size because, in this strategy, new GC log files are created only when you restart the JVM. This is not a major concern in my opinion because one GC event only occupies a few bytes. Typically, GC log file size will not grow beyond a manageable point.

2. Use -XX:+UseGCLogFileRotation

Another approach is to use the JVM system properties:

"-XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=5, -XX:GCLogFileSize=2M"

When -XX:-UseGCLogFileRotation is passed, GC log rotation is enabled by the JVM itself.

-XX:NumberOfGClogFiles sets the number of files to use when rotating logs, must be greater than or equal to one. The rotated log files will use the following naming scheme: <filename>.0, <filename>.1, …, <filename>.n-1.

-XX:GCLogFileSize defines the size of the log file at which point the log will be rotated. It must be greater than or equal to 8K.

However, this strategy has few challenges:

Losing Old GC Logs

If you configure -XX:NumberOfGCLogFiles=5 then over a period of time, five GC log files will be created:

gc.log.0 — oldest GC Log content
gc.log.1
gc.log.2
gc.log.3
gc.log.4 — latest GC Log content

Most recent GC log contents will be written to gc.log.4 and old GC log content will be present in gc.log.0.

When the application starts to generate more GC logs than the configured -XX:NumberOfGCLogFiles (in this case five), then old GC log file contents in gc.log.0 will get deleted. Then you will end up not having all the generated GC logs. You will lose the full visibility of the events.

Mixed-Up GC Logs

Suppose application has created five GC log files, i.e.,

gc.log.0
gc.log.1
gc.log.2
gc.log.3
gc.log.4

Then, let’s say you are restarting the application. Now new GC logs will be written to gc.log.0 file and old GC log content will be present in gc.log.1, gc.log.2, gc.log.3, gc.log.4 i.e.

gc.log.0 — GC log file content after restart
gc.log.1 — GC log file content before restart
gc.log.2 — GC log file content before restart
gc.log.3 — GC log file content before restart
gc.log.4 — GC log file content before restart

So, your new GC log contents get mixed up with old GC logs. To mitigate this problem, you might have to move all the old GC logs to a different folder before you restart the application.

Tooling

To analyze the GC log file using the GC tools (such as gceasy.io, GCViewer, etc.), you will have to upload multiple GC log files instead of just one single GC Log file.

Conclusion

You can debate about which approach you want to take for rotating GC log files, but don’t debate about whether to rotate the GC log files or not. It will come very handy when the need arises — and you never know when it will arise.