Over a million developers have joined DZone.

Monitoring Consistently Failing Flows From MMC

· Integration Zone

Is iPaaS solving the right problems? Not knowing the fundamental difference between iPaaS and dPaaS could cost you down the road. Brought to you in partnership with Liaison Technologies.

The Mule Management Console (MMC) is a great tool for monitoring Mule instances, dispatching alerts if something is not as it should be, and many other cool features. However, at the time of this article's publication, it has no support for sending out alerts in an intelligent way when a flow is consistently failing.

Let me explain in more detail. Consider a simple Web Service proxy in Mule:

<?xml version="1.0" encoding="UTF-8"?>

<mule xmlns:http="http://www.mulesoft.org/schema/mule/http" xmlns:cxf="http://www.mulesoft.org/schema/mule/cxf" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
  xmlns:spring="http://www.springframework.org/schema/beans" version="EE-3.3.1"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd 
http://www.mulesoft.org/schema/mule/cxf http://www.mulesoft.org/schema/mule/cxf/current/mule-cxf.xsd 
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd 
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd ">
	
    <flow name="CxfProxyFlow" doc:name="CxfProxyFlow">
        <http:inbound-endpoint exchange-pattern="request-response" host="localhost" port="8081" doc:name="HTTP" path="proxy"/>
        <cxf:proxy-service doc:name="SOAP" payload="body"/>
        <cxf:proxy-client doc:name="SOAP" enableMuleSoapHeaders="true" payload="body"/>
        <http:outbound-endpoint exchange-pattern="request-response" host="localhost" port="8081" doc:name="HTTP" path="service"/>
    </flow>

</mule>

It’s true that monitoring the target service is completely out of scope for MMC. However your proxy flow will consistently fail whenever a message is received due to a connection refused exception when the target service is unreachable. If your Mule instance is hiding the backend services, then it’s your support team that have to answer questions as to why your service is down.

So what can we do about this? The obvious option is to get alerted that your flow is consistently failing so someone can act on it. This can be done using the log regex feature. The problem is that you will receive an alert for every exception, and this can potentially generate lots of alerts. In addition, if you have a different alert, it will be buried in the long list of alerts generated by the log regex.

Unfortunately there is no ‘out of the box’ MMC feature to solve this, so we have to cater for it ourselves. Luckily Mule provides some APIs that give us good statistical information. Using these APIs, we have written a small class that calculates the ratio of the error messages coming in a flow compared to the messages that were processed successfully. The complete Java code of our class is shown in the following gist.

package com.ricston.monitor;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.mule.construct.Flow;

public class ConsistentlyFailingMonitor {
  
	private long lastRecordedEventsReceived = 0;
	private long lastRecordedFailedEvents = 0;
	private float threshold = 0.5f;
	private float lastRecordedRatio;
	private boolean constantlyFailing = false;
	private String constantlyFailingString = Boolean.toString(constantlyFailing);
	
	private Flow flow;
	protected Log logger = LogFactory.getLog(getClass());
	private int count;
	
	private final String logLineLastRecorded = "lastRecordedEventsReceived: %s -- lastRecordedFailedEvents: %s";
	private final String logLineNewEvents = "newEventsReceived: %s -- newFailedEvents: %s";
	private final String logLineDiffEvents = "diffEventsReceived: %s -- diffFailedEvents: %s";
	private final String logLineRatioFailed = "Ratio: %s -- Failing: %s";

	public synchronized void monitor() {
		count++;
		
		// quick fix for spring task scheduler scheduling our tasks twice
		if (count % 2 == 0) {
			
			//retrieve values of events received and events failed
			long newEventsReceived = flow.getStatistics().getTotalEventsReceived();
			long newFailedEvents = flow.getStatistics().getExecutionErrors();
			
			//calculate difference between now and last recorded statistics
			long diffEventsReceived = newEventsReceived - lastRecordedEventsReceived;
			long diffFailedEvents = newFailedEvents - lastRecordedFailedEvents;
			
			//some logging
			logger.debug(String.format(logLineLastRecorded, lastRecordedEventsReceived, lastRecordedFailedEvents));
			logger.debug(String.format(logLineNewEvents, newEventsReceived, newFailedEvents));
			logger.debug(String.format(logLineDiffEvents, diffEventsReceived, diffFailedEvents));
			
			//store last recorded values
			lastRecordedEventsReceived = newEventsReceived;
			lastRecordedFailedEvents = newFailedEvents;
			
			//if we did not receive any good events since last time recorded, lets check if we received error events 
			if (diffEventsReceived == 0) {
				
				//if we received only error events, set failed to true
				if (diffFailedEvents > 0) {
					lastRecordedRatio = 1;
					updateConstantlyFailing(true);
				} 
				//if we received no events at all, then set failed to false
				else {
					lastRecordedRatio = 0;
					updateConstantlyFailing(false);
				}
			} 
			//if we received good events, lets calculate the ratio of bad events against good events
			//and update the failed status accordingly (checking with the threshold)
			else {
				lastRecordedRatio = (float) diffFailedEvents / diffEventsReceived;
				updateConstantlyFailing(lastRecordedRatio > threshold);
			}
			
			//some more logging
			logger.debug(String.format(logLineRatioFailed, lastRecordedRatio, constantlyFailingString));
		}
	}

	protected void updateConstantlyFailing(boolean value) {
		constantlyFailing = value;
		constantlyFailingString = Boolean.toString(constantlyFailing);
	}

	//all getters and setters
}

The idea is to have the monitor() method execute every couple of seconds. On each execution, we receive updated statistics of execution errors versus completed executions. From these values, we subtract the same values recorded in the previous run so that we get the values for just this period. Once this calculation is complete, we divide the errors with the total events to get a ratio of the two values. If the ratio exceeds a certain threshold (by default set to 0.5), then the String class variable constantlyFailingString is set to true.

Having the monitoring code ready, we need to instruct Mule to execute this code every few seconds; 10 seconds for example . We also need to tell ConsistentlyFailingMonitor which Flow to monitor. Spring comes in very handy to do just that.

<?xml version="1.0" encoding="UTF-8"?>

<mule xmlns:http="http://www.mulesoft.org/schema/mule/http" xmlns:cxf="http://www.mulesoft.org/schema/mule/cxf" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
  xmlns:spring="http://www.springframework.org/schema/beans" xmlns:task="http://www.springframework.org/schema/task" version="EE-3.3.1"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd 
http://www.mulesoft.org/schema/mule/cxf http://www.mulesoft.org/schema/mule/cxf/current/mule-cxf.xsd 
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd 
http://www.springframework.org/schema/task http://www.springframework.org/schema/task/spring-task-3.1.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd  ">

	<spring:beans>
		 <spring:bean id="CxfProxyFlowMonitor" class="com.ricston.monitor.ConsistentlyFailingMonitor"> 
		 	<spring:property name="flow" ref="CxfProxyFlow" /> 
		 </spring:bean> 

		<task:scheduler id="CxfProxyFlowMonitorScheduler" pool-size="1" />
		
		<task:scheduled-tasks scheduler="CxfProxyFlowMonitorScheduler">
			<task:scheduled ref="CxfProxyFlowMonitor" method="monitor" fixed-delay="10000" />
		</task:scheduled-tasks>
	</spring:beans>

</mule>

To review what we have done so far:

First we defined our monitoring bean, and wired the CxfProxyFlow to be the monitored flow. Second we defined a Spring task scheduler and a scheduled task so that our monitoring code is executed every 10 seconds.

So at this stage, we have a class monitoring our flow for a stream of consistently failing messages. This is all well and good, but we need a way to raise an alert if our flow is failing. A couple of weeks ago, I showed you how to create an alert based on a JMX attribute. We can use the same technique, so all we really need to do is expose the monitoring bean as an MBean, configure the alert, and we’re ready. To expose our bean as MBean we can use Spring again:

<?xml version="1.0" encoding="UTF-8"?>

<mule xmlns:http="http://www.mulesoft.org/schema/mule/http"  xmlns:cxf="http://www.mulesoft.org/schema/mule/cxf" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
  xmlns:spring="http://www.springframework.org/schema/beans" version="EE-3.3.1"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd 
http://www.mulesoft.org/schema/mule/cxf http://www.mulesoft.org/schema/mule/cxf/current/mule-cxf.xsd 
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd  ">

	<spring:beans>

		 <spring:bean id="CxfProxyFlowMonitorExporter" class="org.springframework.jmx.export.MBeanExporter">
			<spring:property name="beans">
				<spring:map>
					<spring:entry
						key="Mule.${app.name}:category=custom,name=CxfProxyFlowMonitor" value-ref="CxfProxyFlowMonitor" />
				</spring:map>
			</spring:property>
		</spring:bean> 

	</spring:beans>
	
</mule>

To review what we have done so far:

First we defined our monitoring bean, and wired the CxfProxyFlow to be the monitored flow. Second we defined a Spring task scheduler and a scheduled task so that our monitoring code is executed every 10 seconds.

So at this stage, we have a class monitoring our flow for a stream of consistently failing messages. This is all well and good, but we need a way to raise an alert if our flow is failing. A couple of weeks ago, I showed you how to create an alert based on a JMX attribute. We can use the same technique, so all we really need to do is expose the monitoring bean as an MBean, configure the alert, and we’re ready. To expose our bean as MBean we can use Spring again:

<?xml version="1.0" encoding="UTF-8"?>

<mule xmlns:http="http://www.mulesoft.org/schema/mule/http"  xmlns:cxf="http://www.mulesoft.org/schema/mule/cxf" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
  xmlns:spring="http://www.springframework.org/schema/beans" version="EE-3.3.1"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd 
http://www.mulesoft.org/schema/mule/cxf http://www.mulesoft.org/schema/mule/cxf/current/mule-cxf.xsd 
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd  ">

	<spring:beans>

		 <spring:bean id="CxfProxyFlowMonitorExporter" class="org.springframework.jmx.export.MBeanExporter">
			<spring:property name="beans">
				<spring:map>
					<spring:entry
						key="Mule.${app.name}:category=custom,name=CxfProxyFlowMonitor" value-ref="CxfProxyFlowMonitor" />
				</spring:map>
			</spring:property>
		</spring:bean> 

	</spring:beans>
	
</mule>

Then the only remaining task is to configure the JMX alert in MMC.

What we have achieved here is a system that helps to diagnose consistently failing flows. It will not throw an alert for each exception and potentially flood your system with alerts. Instead it will generate one single alert when a threshold of error messages is reached.

Enjoy!









Discover the unprecedented possibilities and challenges, created by today’s fast paced data climate and why your current integration solution is not enough, brought to you in partnership with Liaison Technologies.

Topics:

Published at DZone with permission of Alan Cassar, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

The best of DZone straight to your inbox.

SEE AN EXAMPLE
Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.
Subscribe

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}