Enable Custom Logging For MuleSoft Application Using Log4J2 Splunk HTTP and HTTP Appender
Logging is an important part of your environment for visualizing the data, troubleshooting, debugging and identifying the production issues quickly.
CORE
·
Join the DZone community and get the full member experience.
Join For FreeIntroduction
Logging is a very important part of your environment for visualizing the data, troubleshooting, debugging and identifying the production issues quickly.
Logging must be robust, consistent and reliable and most importantly that we can have some centralized and easy to use tools that can be used for logging purposes. There are various logging tools like Splunk, ELK, etc.
MuleSoft provides its logging framework for storing applications and system logs. The main reason for using a custom logging framework is that MuleSoft in Cloudhub stores data for 30 days or up to 100 MB.
Once the application is deleted, we will lose all the logs. So we required logging tools like Splunk for storing data and can persist for a longer time.
Splunk logging tool can be used to enable log MuleSoft On-Premise Runtime logs, On-Premise MuleSoft Application and CloudHub Application logs.
In the last article, we have seen how to set up Splunk and enable logging for MuleSoft Application. Click here for more details.
Enable Logging For CloudHub Application
Before enabling the logging for Cloudhub application, you need Disable CloudHub logs. By default, this option is not available and you need to raise a ticket with MuleSoft for providing this option.
Once you disabled CloudHub logs, MuleSoft is not responsible for below things
- MuleSoft is not responsible for lost logging data due to the misconfiguration of your log4j appender.
- MuleSoft is also not responsible for misconfigurations that result in performance degradation, running out of disk space, or other side effects.
- When you disable the default CloudHub application logs, then only the system logs are available. For application worker logs, please check your own application’s logging system. Downloading logs is not an option in this scenario.
- Only Asynchronous log appenders can be used, Synchronous appenders should not be used.
- Use asynchronous loggers and not synchronous ones is to avoid threading issues. Synchronous loggers can lock threads waiting for responses.
You need to create log4j2.xml at location src/main/resources.
There are two ways you can send logs to Splunk
- HTTP Appender
- Splunk HTTP
We will see how we can send logs from CloudHub application to Splunk using Splunk HTTP.
You will be requiring the token and connection details related to Splunk. So you can go through the article and understand how to fetch those details.
You need to add a SplunkHttp appender in log4j.xml. Provide URL to connect Splunk, source, pattern layout, etc.
xxxxxxxxxx
<SplunkHttp name="SPLUNK-CLOUD" source="mule-cloudhub-app"
url="http://localhost:8088/"
token="b3a34eabd-b5f4-471431-97d2-e5b234af5bae9e" index="main"
disableCertificateValidation="true">
<PatternLayout pattern="[%d{MM-dd HH:mm:ss}] %-5p %c{1} [%t]: %m%n" />
</SplunkHttp>
You need to make sure the logger must be asynchronous and it is recommended by MuleSoft also.
While using SplunkHttp, you need to add the following dependency in pom.xml of your application.
xxxxxxxxxx
<dependency>
<groupId>com.splunk.logging</groupId>
<artifactId>splunk-library-javalogging</artifactId>
<version>1.7.3</version>
</dependency>
You may require to add below two additional dependencies.
xxxxxxxxxx
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.10.0</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.10.0</version>
</dependency>
You will be also required to add a repository into your pom.xml of your application.
xxxxxxxxxx
<repository>
<id>splunk-artifactory</id>
<name>Splunk Releases</name>
<url>http://splunk.jfrog.io/splunk/ext-releases-local</url>
</repository>
You can add more loggers like Log4J2CloudhubLogAppender into your log4j2.xml to enable logging on the CloudHub log console of your application.
xxxxxxxxxx
<Log4J2CloudhubLogAppender name="CLOUDHUB"
addressProvider="com.mulesoft.ch.logging.DefaultAggregatorAddressProvider"
applicationContext="com.mulesoft.ch.logging.DefaultApplicationContext"
appendRetryIntervalMs="${sys:logging.appendRetryInterval}"
appendMaxAttempts="${sys:logging.appendMaxAttempts}"
batchSendIntervalMs="${sys:logging.batchSendInterval}"
batchMaxRecords="${sys:logging.batchMaxRecords}" memBufferMaxSize="${sys:logging.memBufferMaxSize}"
journalMaxWriteBatchSize="${sys:logging.journalMaxBatchSize}"
journalMaxFileSize="${sys:logging.journalMaxFileSize}"
clientMaxPacketSize="${sys:logging.clientMaxPacketSize}"
clientConnectTimeoutMs="${sys:logging.clientConnectTimeout}"
clientSocketTimeoutMs="${sys:logging.clientSocketTimeout}"
serverAddressPollIntervalMs="${sys:logging.serverAddressPollInterval}"
serverHeartbeatSendIntervalMs="${sys:logging.serverHeartbeatSendIntervalMs}"
statisticsPrintIntervalMs="${sys:logging.statisticsPrintIntervalMs}">
<PatternLayout pattern="[%d{MM-dd HH:mm:ss}] %-5p %c{1} [%t]: %m%n" />
</Log4J2CloudhubLogAppender>
Below is full log4j2.xml which can be used for your application for enabling custom logging on CloudHub and Splunk.
x
<Configuration status="INFO" name="cloudhub"
packages="com.mulesoft.ch.logging.appender,com.splunk.logging,org.apache.logging.log4j">
<Appenders>
<Log4J2CloudhubLogAppender name="CLOUDHUB"
addressProvider="com.mulesoft.ch.logging.DefaultAggregatorAddressProvider"
applicationContext="com.mulesoft.ch.logging.DefaultApplicationContext"
appendRetryIntervalMs="${sys:logging.appendRetryInterval}"
appendMaxAttempts="${sys:logging.appendMaxAttempts}"
batchSendIntervalMs="${sys:logging.batchSendInterval}"
batchMaxRecords="${sys:logging.batchMaxRecords}" memBufferMaxSize="${sys:logging.memBufferMaxSize}"
journalMaxWriteBatchSize="${sys:logging.journalMaxBatchSize}"
journalMaxFileSize="${sys:logging.journalMaxFileSize}"
clientMaxPacketSize="${sys:logging.clientMaxPacketSize}"
clientConnectTimeoutMs="${sys:logging.clientConnectTimeout}"
clientSocketTimeoutMs="${sys:logging.clientSocketTimeout}"
serverAddressPollIntervalMs="${sys:logging.serverAddressPollInterval}"
serverHeartbeatSendIntervalMs="${sys:logging.serverHeartbeatSendIntervalMs}"
statisticsPrintIntervalMs="${sys:logging.statisticsPrintIntervalMs}">
<PatternLayout pattern="[%d{MM-dd HH:mm:ss}] %-5p %c{1} [%t]: %m%n" ></PatternLayout>
</Log4J2CloudhubLogAppender>
<SplunkHttp name="SPLUNK-CLOUD" source="mule-cloudhub-app"
url="http://localhost:8088/"
token="b3a34eabd-b5f4-471431-97d2-e5b234af5bae9e" index="main"
disableCertificateValidation="true">
<PatternLayout pattern="[%d{MM-dd HH:mm:ss}] %-5p %c{1} [%t]: %m%n" ></PatternLayout>
</SplunkHttp>
</Appenders>
<Loggers>
<AsyncLogger
name="org.mule.runtime.core.internal.processor.LoggerMessageProcessor"
level="INFO" ></AsyncLogger>
<AsyncLogger name="com.mulesoft.agent" level="INFO" ></AsyncLogger>
<AsyncRoot level="INFO">
<AppenderRef ref="SPLUNK-CLOUD" ></AppenderRef>
<AppenderRef ref="CLOUDHUB" ></AppenderRef>
</AsyncRoot>
<AsyncLogger name="com.gigaspaces" level="ERROR" ></AsyncLogger>
<AsyncLogger name="com.j_spaces" level="ERROR" ></AsyncLogger>
<AsyncLogger name="com.sun.jini" level="ERROR" ></AsyncLogger>
<AsyncLogger name="net.jini" level="ERROR" ></AsyncLogger>
<AsyncLogger name="org.apache" level="WARN" ></AsyncLogger>
<AsyncLogger name="org.apache.cxf" level="WARN" ></AsyncLogger>
<AsyncLogger name="org.springframework.beans.factory"
level="WARN" ></AsyncLogger>
<AsyncLogger name="org.mule" level="INFO" ></AsyncLogger>
<AsyncLogger name="com.mulesoft" level="INFO" ></AsyncLogger>
<AsyncLogger name="org.jetel" level="WARN" ></AsyncLogger>
<AsyncLogger name="Tracking" level="WARN" ></AsyncLogger>
<AsyncLogger name="org.mule" level="INFO" ></AsyncLogger>
<AsyncLogger name="com.mulesoft" level="INFO" ></AsyncLogger>
<AsyncLogger name="org.mule.extensions.jms" level="INFO" ></AsyncLogger>
<AsyncLogger name="org.mule.service.http.impl.service.HttpMessageLogger"
level="INFO" ></AsyncLogger>
<AsyncLogger name="org.mule.extension.salesforce" level="INFO" ></AsyncLogger>
<AsyncLogger name="org.mule.extension.ftp" level="INFO" ></AsyncLogger>
<AsyncLogger name="org.mule.extension.sftp" level="INFO" ></AsyncLogger>
<AsyncLogger name="com.mulesoft.extension.ftps" level="INFO" ></AsyncLogger>
<AsyncLogger name="org.mule.modules.sap" level="INFO" ></AsyncLogger>
<AsyncLogger name="com.mulesoft.extension.mq" level="INFO" ></AsyncLogger>
<AsyncLogger name="com.mulesoft.mq" level="INFO" ></AsyncLogger>
<AsyncLogger name="org.mule.extension.db" level="INFO" ></AsyncLogger>
<AsyncLogger name="httpclient.wire" level="DEBUG" ></AsyncLogger>
<AsyncLogger name="org.mule.transport.email" level="DEBUG" ></AsyncLogger>
</Loggers>
</Configuration>
JSON Logger
JSON Logger is a component for logging the information, warning, errors. It logs the data in the form of JSON. It is recommended to use JSON Logger for your application and it logs the data in a well-formed way in any tools like Splunk, ELK, etc.
You can install JSON Logger in Anypoint Studio from Exchange.
Configure the JSON Logger as shown below.
You can set the application name, application version and environment in the properties file.
<mule xmlns:ee="http://www.mulesoft.org/schema/mule/ee/core"
xmlns:json-logger="http://www.mulesoft.org/schema/mule/json-logger"
xmlns:api-gateway="http://www.mulesoft.org/schema/mule/api-gateway"
xmlns:http="http://www.mulesoft.org/schema/mule/http" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/api-gateway http://www.mulesoft.org/schema/mule/api-gateway/current/mule-api-gateway.xsd
http://www.mulesoft.org/schema/mule/json-logger http://www.mulesoft.org/schema/mule/json-logger/current/mule-json-logger.xsd
http://www.mulesoft.org/schema/mule/ee/core http://www.mulesoft.org/schema/mule/ee/core/current/mule-ee.xsd">
<http:request-config name="HTTP_Request_configuration" doc:name="HTTP Request configuration" doc:id="4dea45bf-d0c6-4ee8-baa0-0b8a9e508c88" basePath="/api/timezone" >
<http:request-connection host="worldtimeapi.org" />
</http:request-config>
<configuration-properties doc:name="Configuration properties" doc:id="9dacbbe5-0ec7-4ce4-b094-ff89131a07cc" file="cloud.properties" />
<global-property doc:name="Global Property" doc:id="7a60a2c8-efcf-4538-81c8-64248fe0dec4" name="mule.env" value="dev" />
<http:listener-config name="HTTP_Listener_config" doc:name="HTTP Listener config" doc:id="694e8fba-882a-4f04-805b-50c66ecec1a3" >
<http:listener-connection host="0.0.0.0" port="${http.private.port}" />
</http:listener-config>
<json-logger:config name="JSON_Logger_Config" doc:name="JSON Logger Config" doc:id="59e12b0a-d64d-4300-af40-23b29f11c2dc"/>
<flow name="test-application-cloudFlow" doc:id="7db040d5-7309-4688-a4c5-fea0f43c6103">
<http:listener doc:name="Listener" doc:id="da11b76c-b4d5-4140-a539-5841c4690b8c" config-ref="HTTP_Listener_config" path="/v1.0/timezone/{region}/{city}">
<http:error-response statusCode="#[vars.httpStatusCode]" >
<http:body ><![CDATA[#[payload]]]></http:body>
</http:error-response>
</http:listener>
<http:request method="GET" doc:name="Request" doc:id="730aec0d-fb82-4430-8cf3-c14264a7e9ae" config-ref="HTTP_Request_configuration" path="/{region}/{city}">
<http:uri-params ><![CDATA[#[output application/java
---
{
city : attributes.uriParams.city,
region : attributes.uriParams.region
}]]]></http:uri-params>
</http:request>
<json-logger:logger doc:name="Logger" doc:id="f2257ace-a8b0-4f5f-83d5-4ac8a710983f" config-ref="JSON_Logger_Config" message="#['Information']"/>
<error-handler >
<on-error-propagate enableNotifications="true" logException="true" doc:name="On Error Propagate" doc:id="1d5f427f-ceaa-4af1-8e2d-2be0aaf45b46" type="HTTP:FORBIDDEN">
<ee:transform doc:name="Transform Message" doc:id="b978dae0-7538-4910-82ee-3c12236b6603" >
<ee:message >
<ee:set-payload ><![CDATA[%dw 2.0
output application/json
---
{
error:error.description
}]]></ee:set-payload>
</ee:message>
<ee:variables >
<ee:set-variable variableName="httpStatusCode" ><![CDATA[403]]></ee:set-variable>
</ee:variables>
</ee:transform>
<json-logger:logger doc:name="Logger" doc:id="3c5e26da-4183-4318-9f6a-a07b86b717ff" config-ref="JSON_Logger_Config" message="Data From Timezone" priority="ERROR"/>
</on-error-propagate>
</error-handler>
</flow>
</mule>
Now, you can verify the logs in Splunk and it will be in JSON format.
Enable Splunk Logging For On-Premise MuleSoft Runtime
For enabling the logging for On-Premise MuleSoft Runtime, you can use HTTP Appender.
You can add HTTP Appender in log4j2.xml located at $MULE_HOME/conf/
x
<Http name="Splunk" url="http://localhost:8088/services/collector/raw">
<Property name="Authorization" value="Splunk eert8e102-5f55-40ad-a047-b55a82245419"></Property>
<PatternLayout pattern="%-5p %d [%t] [event: %X{correlationId}] %c: %m%n"></PatternLayout>
</Http>
This will allows sending logs to Splunk.
Full log4j2.xml
x
<Configuration>
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%-5p %d [%t] [event: %X{correlationId}] %c: %m%n" />
</Console>
<Console name="ConsoleLogUtil" target="SYSTEM_OUT">
<PatternLayout pattern="%m%n" />
</Console>
<Http name="Splunk" url="http://localhost:8088/services/collector/raw">
<Property name="Authorization" value="Splunk eert8e102-5f55-40ad-a047-b55a82245419"></Property>
<PatternLayout pattern="%-5p %d [%t] [event: %X{correlationId}] %c: %m%n"></PatternLayout>
</Http>
</Appenders>
<Loggers>
<AsyncLogger
name="org.mule.runtime.core.internal.processor.LoggerMessageProcessor"
level="INFO" />
<AsyncLogger name="com.mulesoft.agent" level="INFO" />
<AsyncLogger name="org.mule.runtime.core.internal.logging"
additivity="false" level="INFO">
<AppenderRef ref="ConsoleLogUtil" />
</AsyncLogger>
<AsyncRoot level="INFO">
<AppenderRef ref="Console" />
<AppenderRef ref="Splunk" />
</AsyncRoot>
<AsyncLogger name="com.mulesoft.mule.runtime.gw" level="INFO" />
<AsyncLogger name="com.mulesoft.extension.policies" level="INFO" />
<AsyncLogger name="com.mulesoft.extension.gateway" level="INFO" />
<AsyncLogger name="com.mulesoft.analytics" level="INFO" />
<AsyncLogger name="org.mule" level="INFO" />
<AsyncLogger name="com.mulesoft" level="INFO" />
</Loggers>
</Configuration>
This is how you can enable Splunk logging using SplunkHTTP and HTTP Appender for MuleSoft applications.
Opinions expressed by DZone contributors are their own.
Comments