Over a million developers have joined DZone.

Developer Games: RegExp and log4j Parsing

How to extract valuable data from log4j logs with virtual fields.

· Java Zone

Discover how powerful static code analysis and ergonomic design make development not only productive but also an enjoyable experience, brought to you in partnership with JetBrains

Extracting Valuable Data from log4j logs with Virtual Fields

In our recent upgrade to XpoLog V6 we enhanced the features of log4j analysis. In this series of posts I am covering some of the ways you can benefit from XpoLog V6’s new features and enhancements. I will concentrate mainly on how to get the most valuable information from your log4j event logs. We have also prepared a hands-on-guide-in-one if you prefer to read all the posts in one go.

Once your log4j logs have been transferred to and properly defined in XpoLog Center, you can troubleshoot your java application by running Analytic Search on your log4j data, measure your application performance, create your own AppTags for better monitoring, and create dashboards, charts, slide-shows, and make use of other visualization gadgets for maximum log analysis.

This post will show you how to define log4j logs in XpoLog, to create the most readable data and thus allow for XpoLog to perform highly detailed analysis of your logs. I will also show you an example of how you can virtually extract specific data from your message using RegularExpression to allow for XpoLog to perform a more refined parsing of your data.

If you want to follow my steps as you read along, you can download XpoLog V6 for free.

Defining Patterns in XpoLog Center

To let XpoLog access and pull data from your files, define the logger with a name, pattern and data pattern, and then define the log patterns in XpoLog Center.

For Example:

#Logger definition



#Appender data for mylog


log4j.appender.mylog.layout.ConversionPattern=[%d][%t] [%p] [%c] [%l] %m%n

(d= date, t = thread, p = priority, c = class, l =method, m = message, and n = new line)

Defining the Log Pattern in XpoLog Center:

  1. 1.      In XpoLog Center, add a new log. (See the instructions in my previous post.) Once you have filled in the details, click Next to get to the Log Pattern screen.

  2. 2.      In the Wizard of thePattern Editor, define the log pattern.

Click Manual in the Pattern Editor and edit the XpoLog data pattern to comply with the log4j layout:

a.      [%d] =[{date:Date,locale=en,yyyy-MM-dd HH:mm:ss,SSS}]

b.     [%t] = [{text:Thread}]

c.      [%p]= [{priority:Priority,DEBUG;INFO;WARNING;ERROR;FATAL}]

d.     [%c]= [{string:Class}]

e.     [%l]= [{string:Method}({text:Source}:{number:LineNumber})]

f.       %m = {string:Message}

g.      %n =  new line

The XpoLog pattern in our example will be:

[{date:Date,locale=en,yyyy-MM-ddHH:mm:ss,SSS}] [{text:Thread}] [{priority:Priority,DEBUG;INFO;WARNING;ERROR;FATAL}][{string:Class}] [{string:Method}({text:Source}:{number:LineNumber})]{string:Message}

  1. 3.      Click Save.

You can also edit the pattern after you have added the log, of which I will speak more of in my next post.

Virtually Extract Specific Data from yourMessage

XpoLog can also extract data from within the message if you use Regular Expression prior to the data transfer.

This is what the message might look like in the LogPattern section of the Add Log screen without using RegularExpression:

In the Pattern Editor, all you see is {string:Message}.

If you use Regular Expression to extract any word that appears after the word “Manager”, the Log Pattern section of the AddLog screen would look as follows:

In the Pattern Editor, you will now see:

{regexp:HTMLManager state,refName=Message,HTMLManager:(\w+)}{string:Message}

In the Log records analysis result section below, XpoLog has added the column HTMLManager state for the data you wished to extract.

In the Manager Interface of XpoLog Center, where you view your logs, you will also see this extra column, HTMLManager state, for the extracted data:

By extracting the HTMLManager state into a new virtual field we can now measure and monitor the HTMLManager state performance and activity.

In the next post, I will show how to define and edit the log4j patterns when sending log events and log messages to XpoLog throughSysLog. Stay tuned, or check out our “spoiler” for the full tutorial.

Learn more about Kotlin, a new programming language designed to solve problems that software developers face every day brought to you in partnership with JetBrains.


The best of DZone straight to your inbox.

Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}