5 Ways to Change Log Levels at Runtime

Logs are very important for any application for debugging. Logs are the first and, many times, the last tool available to get it to the root cause of any issue.

In this blog, I am not going to discuss what to log and how to log effectively. But I will walk through the critical aspect of logging which is log levels. In every environment whether it is production/QA/Dev each one has particular log levels configured. In the production environment, log levels are strictly managed to make sure log file size is minimum.

Every typical day is not the same, bugs are unavoidable in software development. Very often a situation occurs where the current logs which are getting logged at a particular level are insufficient. So, need to arise to bump up the log levels.

In a poorly designed application where log files are inbuild (either in the jar or image), it all together needs new build to merely change the log levels. In decently design application log configuration is externalized. But still, to merely change the log levels it needs an application restart.

In terms of scaling, the challenge is to change log levels in each instance. So to avoid all pitfalls following is needed:

1. Dynamically change the log level at runtime without application restart.
2. Propagation of log level changes across the application instances.

Logging frameworks have undergone lots of changes in the past 5 years and we are going to leverage that. As everyone is moving towards microservice architecture we are inevitably going to adhere that and will use Spring Boot, sl4j, and Logback as our application stack.

The following are the options to achieve that. Usage is not only restricted to microservice architecture it can be applied in other styles of architecture too provided we are using the sl4j and Logback.

Option 1: Using the Spring Boot Admin

This is the best and most recommended option if you are using Spring Boot as everything comes out of the box. In this option, your application needs to enable the spring actuator and get it registered with Spring Boot Admin.

The user will use the spring boot admin UI to change the log levels. 

Merely select the logger and change the log level in just click of a button. As highlighted in the figure it will change the log level in all running instances you no need to perform anything also.

If you are new to spring boot admin you can get details here. You can find the entire working code here.

Option 2: Scan the logback.xml

You can configure Logback to periodic scan and update the logging configuration via logback.xml

<?xml version="1.0" encoding="UTF-8"?>

<configuration scan="true" scanPeriod="15 seconds">

...

</configuration>

If you alter the log levels in the logback.xml file then changes will get effect max after 15 seconds. To propagate the log level changes across all container instance at once follow the below architecture: 

Over here you externalize the logback.xml file and make it available to the container via NFS mount point. Pass the log file path to the container via JVM argument: ” -Dlogging.config=/logpath/logback.xml”.

Option -3 Access logback.xml via URL

In this option you need to expose the logback.xml via URL with the help of nginx/apache/HAProxy etc. and the application will consume the logback.xml via JVM arg: “-Dlogging.config = http://host:9999/logback.xml” Woefully, the auto-scan doesn’t work if you expose the logback.xml via URL. So to achieve that we need some polling mechanism.

xxxxxxxxxx

public void load() throws JoranException, MalformedURLException, InterruptedException {

        String prev = null;

        URL url = new URL("http://localhost:9999/logconfig/logback.xml");

        while (true){

            if(!(getNewContentIfChanged(prev,url) == null)){

                LoggerContext context = (LoggerContext) LoggerFactory.getILoggerFactory();

                context.reset();

                JoranConfigurator configurator = new JoranConfigurator();

                configurator.setContext(context);

                configurator.doConfigure(url);

                System.out.println("Log reloaded  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>");

            }

            TimeUnit.SECONDS.sleep(15);

        }

    }

Log level changes propagation: 

Option 4: Creating API To Change the Log Levels

Another way is to change the log level programmatically by exposing the API.

xxxxxxxxxx

@RequestMapping("/changelevel")

   public String changeLogLevel(@RequestParam String loggerName, @RequestParam String level){

       LoggerContext loggerContext = (LoggerContext) LoggerFactory.getILoggerFactory();

       ch.qos.logback.classic.Logger logger = loggerName.equalsIgnoreCase("root")?

               loggerContext.getLogger(loggerName):loggerContext.exists(loggerName);

       if( logger !=null){

           logger.setLevel(Level.toLevel(level));

           return "Changed logger: "+loggerName+" to level : "+level;

       } else {

           return "Logger Not Found Make Sure that logger name is correct";

       }

   }

To propagate the log level changes across the instances we need to follow one of the methods below:

We need a custom utility that will expose the API to change the log level. The utility will fetch the instance/s IP address and execute the API call on each running APP instance.

Option 5: Pub / Sub

In this option, the application instance needs to subscribe to the topic where log level changes will get published.

As per me, this is termed over-engineering if you are doing for just log level changes. But this can be very effective in microservice architecture if you want to change the log levels of all applications in one shot.

Conclusion

There are other ways to expose and load the log configuration via FTP, socket, etc. but the approach is the same. Which approach to take is utterly dependent on the deployment topology, application architecture style, and need.

The above represent the approaches but securing the log configuration and auditing the changes are out of the scope of the blog. I am assuming you will take care of your respective environment.

Hope you enjoy the post. Feel free to click the like button.

Entire code you can get here.

Happy coding.