Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Refined Data Parsing - Log4j Patterns

DZone's Guide to

Refined Data Parsing - Log4j Patterns

Defining and editing log4j patterns in SysLog for more refined data parsing.

· Java Zone
Free Resource

What every Java engineer should know about microservices: Reactive Microservices Architecture.  Brought to you in partnership with Lightbend.

XpoLog’s updated version is already here and will not overlook any piece of raw data, no matter how small or insignificant it may seem. In this series of posts I am covering some of the ways you can benefit from XpoLog V6’s new features and enhancements. I will concentrate mainly on how to get the most valuable information from your log4j event logs.

Once your log4j logs have been transferred to and properly defined in XpoLog Center, you can troubleshoot your Java application by running analytic search on your log4j data, measure your application performance, create your own AppTags for better monitoring, and create dashboards, charts, slide-shows, and make use of other visualization gadgets for maximum analysis. For details, check out our manual.

This post will show you how to define and edit your events and log patterns before and after they reach XpoLog Center, when sending them through SysLog. By creating the most readable data you will allow for XpoLog to perform the highest detailed analysis of your logs. To follow more easily as I go along you can download the software for free.

Since logs are written in free format, XpoLog has an advanced built-in mechanism to detect the structure, or pattern, of the incoming log. As a user, you can edit and fine-tune these patterns to suit your needs.

Defining Patterns in SysLog Appenders

When sending events to XpoLog through SysLog, be sure to create a detailed conversion pattern while configuring your log4j SysLog appender. Here is an example:

    #Logger definition

    log4j.logger., SYSLOG

    #Appender data for syslog

    log4j.appender..apache.log4j.net.SyslogAppender

    log4j.appender.SYSLOG..0.0.1

    log4j.appender.SYSLOG..apache.log4j.PatternLayout

    log4j.appender.SYSLOG.layout.conversionPattern=[%t] %c - %m%n

    log4j.appender.SYSLOG.

(t = thread, c = class, m = message, and n = new line)

The SysLog appender will write this event logger to the SysLog. Remember to define a SysLog Listener account inside XpoLog Center. See my previous article for the instructions on how to do that.

The events that arrive at XpoLog Center are written internally. Here is what they might look like when created by the XpoLog SysLog listener:

XPLG:[1436716542132] [local1] [INFO] [test-1] []: [http-30303-Processor24] audit - [Master] [-] [LOGIN] [login/logout] [SECURITY] [http-30303-Processor24] [-] [-] [-] [-] release user admin

XPLG:[1436716542140] [local1] [INFO] [test-1] []: [http-30303-Processor24] audit - [Master] [Admin] [LOGIN] [login/logout] [SECURITY] [http-30303-Processor24] [EDA6FECA79A7BBB4480BAFC0FFB911F1] [administrators] [127.0.0.1] [127.0.0.1] login with username admin ok

The text at the very beginning is the extra data added by the XpoLog Syslog listener. The other parts of the text in the SysLog file correspond to the layout you created in the log4j SysLog appender.

Once the data arrives into XpoLog, a log is created with the default SysLog pattern:

XPLG:[{timestamp:Timestamp,MM/dd/yyyy HH:mm:ss.SSS}] [{text:Facility}] [{priority:Level,DEBUG;INFO;WARN;ERROR;FATAL}] [{text:Source Device}] {block,start,}{string:Message}

Edit the log and set the pattern to reflect the layout you defined in the log4j configuration.

To Edit the Pattern in a Log in XpoLog:

  1. In XpoLog Center, go to Administration  and find the log under Folders and Logs in the tree in the left margin. Right-click on the log and select EditImage title
  2. Click Next to get to the Log Pattern section. The pattern can be edited in the Pattern1 field of the Pattern Editor, or you can add a new pattern in addition to the existing one by clicking the New tab.
  3. Toggle between the Manual button (far right) and the Wizard button to see either version of the pattern.  You can add as many patterns as you want by clicking the New tab. XpoLog will save all these patterns as templates for forthcoming logs.

  4. Click Save.

In the screen capture below you can see how to define the log data pattern. It is displayed in the Pattern1 field. The pattern for this log is the following:

XPLG:[{timestamp:Timestamp,MM/dd/yyyy HH:mm:ss.SSS}] [{text:Facility}] [{priority:Level,DEBUG;INFO;WARN;ERROR;FATAL}] [{text:Source Device}] {block,start,}{text:Application Name}[{text:Process Id}]: {block,end,}{string:Message}

Image title

Most of the pattern, up to and including {block,end,} is part of the SysLog protocol and functions as an prefix to the message – it contains the SysLog timestamp, facility, priority and the source device.

As mentioned previously, you can edit the pattern inside XpoLog Center after the event logs have been sent. If your messages all follow the same structure, we recommend further editing the pattern to include this structure, to receive a more refined parsing. Here is a more refined pattern of the log shown above:

XPLG:[{timestamp:Timestamp,MM/dd/yyyy HH:mm:ss.SSS}] [{text:Facility}] [{priority:Level,DEBUG;INFO;WARN;ERROR;FATAL}] [{text:Source Device}] {block,start,}{text:Application Name}[{text:Process Id}]: {block,end,}[{text:ServerIp}] [{text:User,User}] [{choice:Action Type,LOGIN;VIEW;CHANGE}] [{text:Action description,Action description}] [{choice:Context,LOGS;FOLDERS;VERIFIERS;CONFIGURATION;SECURITY;REPROTS;TASKS;JOBS;NODES;SEARCH_ENGINE}] {string:Message}

The following screen capture shows the same log as above, after editing. You can see the original message has been split into the relevant columns.

Image title

Note that by creating the most readable data, you will receive the most detailed analysis of your logs from XpoLog.

In my next post, I will discuss how to tag the logs with AppTags, for easier monitoring, troubleshooting, and search. Stay tuned or go directly to our hands-on-guide.

Microservices for Java, explained. Revitalize your legacy systems (and your career) with Reactive Microservices Architecture, a free O'Reilly book. Brought to you in partnership with Lightbend.

Topics:
java

Opinions expressed by DZone contributors are their own.

THE DZONE NEWSLETTER

Dev Resources & Solutions Straight to Your Inbox

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

X

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}