Structured Logging in Java With Elastic Stack

Logging is an important pillar of observability in microservices architecture. Structured logging can be leveraged to enable many business-critical functionalities.

Randhir Singh

Jul. 28, 21 · Tutorial

Likes (16)

Comment

Save

20.0K Views

Logging is an important pillar of observability in a microservices architecture. Logs are commonly used to find out issues in the software functionality. However, they can be leveraged to enable many business-critical functionalities, e.g., real-time analytics and many others.

In this article, we will use logging libraries in Java to produce structured logs and Elastic stack to collect and aggregate logs. We will then query Elasticsearch to derive insights from the indexed logs.

Scenario

Let us say we have deployed an API for customers to use. We would like to know some statistics on the API usage, e.g., in a given time, who invoked it, how many times, how many were successful, what were the failures, etc.

The Java developer who wrote the API may log the incoming request as follows:

     Java 
   
   log.info("API called by customer {}, product {} for {}", 
				getCustomerId(),
				getProductId()
				getAction());

This prints a log in an unstructured format.

     Java 
   
   10:03:35.331 [main] INFO none.rks.myproduct - API called by client client-1, product product-1 for purpose-1

Doing any kind of analytics on unstructured logs is difficult. If we want to search on client or product, a log aggregator will not allow us to do that as those fields will not be available in the index.

Structured logging in Java

To create logs in a structured format, we can start by using Java ECS logging in our Java application. The logs generated in this manner will be structured as JSON objects and will use Elastic Common Schema (ECS)-compliant field names.

In our application, we'll configure Log4j2 to use Java ECS logging.

Add the following dependency to pom.xml.

     Java 
   
 
 
           <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-log4j2</artifactId>
        </dependency>
        <dependency>
            <groupId>co.elastic.logging</groupId>
            <artifactId>log4j2-ecs-layout</artifactId>
            <version>${ecs-logging-java.version}</version>
        </dependency> 
  

As Spring Boot uses logback as the default logger, let us exclude it from the spring-boot-starter dependency.

     Java 
   
 
 
   	<dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-starter-logging</artifactId>
                </exclusion>
            </exclusions>
        </dependency> 
  

Next, add a log4j2 configuration for log4j2 logger log4j2.xml in src/main/resources.

     XML 
   
 
 
   <Configuration name="LogToConsole" target="SYSTEM_OUT">
    <Appenders>
        <Console name="LogToConsole" target="SYSTEM_OUT">
            <EcsLayout serviceName="my-app"/>
        </Console>
        <File name="FileAppender" fileName="logs/app.log.json">
            <EcsLayout serviceName="my-app"/>
        </File>
    </Appenders>
    <Loggers>
        <Root>
            <AppenderRef ref="LogToConsole" />
            <AppenderRef ref="FileAppender" />
        </Root>
        <Logger name="none.rks.Main" level="debug" additivity="false">
            <AppenderRef ref="LogToConsole" />
            <AppenderRef ref="FileAppender" />
        </Logger>
    </Loggers>
</Configuration> 
  

Let us use Log4j2 in our application to log statements.

     Java 
   
 
 
   package none.rks;

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.message.StringMapMessage;

public class Foo {
  private static final Logger log = LogManager.getLogger(Foo.class);
  
  public void bar() {
    		log.info(new StringMapMessage()
                     .with("message", "API called")
                     .with("customer.id", getCustomerId())
                     .with("product.id", getProductId())
                     .with("customer.action", getAction());
  }
} 
  

This will produce logs in JSON structured format with custom fields.

     Java 
   
 
 
   {
  "@timestamp":"2021-07-01T11:20:12.176Z", 
  "log.level": "INFO", 
  "customer.id": "client-1",
  "product.id": "25a76f91-41dd-49da-8020-3553c9100267",
  "customer.action": "CREATE",
  "message":"API called", 
  "process.thread.name":"reactor-http-nio-2",
  "log.logger":"none.rks.Foo"
 } 
  

These fields will enable us to query log aggregator based on these fields. Let us setup and configure log aggregator next.

Setup Elastic Stack

Elastic stack comes in both on-premise version and managed Cloud version. For illustration purposes, let us install the required components of the Elastic stack on our Windows laptop.

First, download and install Elasticsearch that will index our logs. Navigate to http://localhost:9200 to make sure Elasticsearch is running fine.

     Java 
   
 
 
   {
  "name" : "LAPTOP-4S09L4LN",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "iPjGMFBuQE-v-8UwwC6JFA",
  "version" : {
    "number" : "7.13.1",
    "build_flavor" : "default",
    "build_type" : "zip",
    "build_hash" : "9a7758028e4ea59bcab41c12004603c5a7dd84a9",
    "build_date" : "2021-05-28T17:40:59.346932922Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
} 
  

Next, install a compatible version of Kibana to visualize those logs. Navigate to http://localhost:5601 to make sure Kibana is installed correctly. By default, it connects to Elasticsearch running at http://localhost:9200.

Finally, install Filebeat to collect and ship the logs to Elasticsearch. We will configure Filebeat to collect logs from the desired location and send it to Elasticsearch running at http://localhost:9200.

Locate filebeat.yaml where Filebeat was installed. Here we have shown relevant properties only.

     Java 
   
 
 
   filebeat.inputs:
- type: log
  enabled: true
  paths:
    - c:\programdata\elasticsearch\logs\*
  json:
    message_key: message    
    keys_under_root: true
    overwrite_keys: true
    
setup.kibana:
  host: "localhost:5601"  
    
output.elasticsearch:
  hosts: ["localhost:9200"] 
  

Run Analytics

To get statistics on how many times the API was called, for example, go to DevTools in Kibana and run queries.

     Java 
   
 
 
   GET my-index/_search
{
  "query": {
    "match_phrase": {
      "customer.id": "*"
    }
  }
} 
  

As the custom field customer.id was indexed, we are able to run queries and gather analytics. The same queries can be run using the Kibana console.

Conclusion

Structured logging enables us to go beyond the common troubleshooting role played by application logs. It enables many business-critical operations. In this article, we covered how to create structured logs. We used Elastic search as an example of a log aggregator. We showed how to install and configure components of the Elastic stack for our purpose. Finally, we were able to run queries on custom fields that we added to our structured logs.

Java (programming language)

Opinions expressed by DZone contributors are their own.

Related

Trending