Starting with JDK 7 update 40 (7u40), the Java Mission Control is bundled with the HotSpot JVM. The Java Mission Control can be used for simple monitoring of a Mule instance, for instance during development or for those content with what the community edition of Mule has to offer. In fact, Java Mission Control can be used to monitor any Java application that exposes a set of JMX MBeans. Since I work with system integration on the Mule platform, I have chosen to use these familiar grounds in my example.

In this post I will show how to set up some simple monitoring of a Mule instance using Java Mission Control that will:

• Display a value on a dial.
  The dial will show the current value and the highest value that has been encountered (a so-called watermark).
  

• Plot a chart for a value.
  

• Send an alarm when a value reaches a certain level for a certain time duration.

This post may also serve as a quick introduction on the JMX monitoring capabilities of the Java Mission Control.

Preparations

Since we are going to monitor, we need something to monitor. To make it easy for myself, I will use the Hello World Mule example project for Mule CE, which will be available if you have installed the MuleStudio plugin installed in your Eclipse or Spring Tool Suite development environment and added the Mule community edition runtime.

• In Eclipse or STS, create a new project using the Mule Example Project wizard.
• Select the Hello World template and the Mule CE runtime.
  
  I have used version 3.4.0 of the Mule CE runtime, but any version will do.
• Ensure that the JRE System Libraries of the project uses Java 7 with an update equal to, or greater than, update 40.
  As can be seen in the picture below, I have used Java 7 version 1.7.0u51 when developing this example.
  
• Modify the Hello World flow in the hello-world example project to look like this:

<?xml version="1.0" encoding="UTF-8"?>
<mule
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://www.mulesoft.org/schema/mule/core"
xmlns:http="http://www.mulesoft.org/schema/mule/http"
xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:spring="http://www.springframework.org/schema/beans"
xmlns:core="http://www.mulesoft.org/schema/mule/core"
xmlns:test="http://www.mulesoft.org/schema/mule/test"
xsi:schemaLocation="
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/test http://www.mulesoft.org/schema/mule/test/current/mule-test.xsd ">
<flow
name="HelloWorldFlow1">
<http:inbound-endpoint
exchange-pattern="request-response"
host="localhost"
port="8081"
doc:name="HTTP"/>
<set-payload
value="#['Hello World']" />
<test:component throwException="true"/>
</flow>
</mule>

• The above modification will result in an exception being thrown each time a message enters the flow.
• Launch the Hello World Mule example program from within Eclipse.

Java Mission Control

In the Windows operating system, I found the Java Mission Control program here:

C:\Program Files\Java\jdk1.7.0_55\bin\jmc

In OS X, I found Java Mission Control at:

/Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home/lib/missioncontrol/Java Mission Control.app

The above paths depend on the version of the Java JDK installed and will thus vary accordingly.

• Start Java Mission Control.
• In the JVM Browser on the left, fold down the node which names starts with ”org.mule.tooling” and double-click the MBean Server item under that node.
  A tab should open on the right, showing something like in the picture below:
  

We are now monitoring a Mule application with Java Mission Control and in the above picture we can see:

• Java Heap Memory usage.
  Leftmost dial in the Dashboard.
• JVM CPU Usage.
  Center dial in the Dashboard.
• Live Set + Fragmentation
  Rightmost dial in the Dashboard.
• Computer and JVM CPU Usage.
  Shown in the topmost chart, below the Dashboard.
• Used Physical Memory and Used Java Heap, both in percent.
  Both these values are shown in the second chart, in the bottom of the window.

 Adding a Dial

Dials in the Dashboard can be added and removed as desired, as to compose your own dashboard enabling you to quickly get an overview of the status of a Mule instance.
The entire Dashboard can also be removed if you do not want to show any dials.

We are now going to replace one of the original dials with one that presents the average processing time by the flow in the Hello World Mule application.

• Click the rightmost dial in the Dashboard, the one that has the title Live Set+Fragmentation.
• Click the small minus-sign in the title-bar of the Dashboard.
  This is the second icon from the left in the group of icons in the top-right corner of the Dashboard.
  Clicking the minus-sign should remove the dial and you should be left with only two dials; the Used Java Heap Memory dial and the JVM CPU Usage dial.
• Click the plus-sign immediately to the left of the minus-sign you clicked to remove the dial.
  A dialog should appear which allow you to specify the properties of the new dial that is to be created.
• Click the Select button to the right of the Attribute Name field.
• Another dial should appear, allowing you to select an MBean attribute which value to display on the dial.
• Drill down to the Mule.hello-world -> Flow -> HelloWorldFlow1 -> AverageProcessingTime attribute, like shown in the picture below:
  
• Click the OK button.
• In the Dial Properties dialog, also click the OK button.
• The new dial entitled AverageProcessingTime should now appear in the Dashboard.
• In a browser, issue a number of requests to the URL: http://localhost:8081/
• Observe the Now and Max values of the AverageProcessingTime dial.
  The first request probably yields a high value for both these numbers, while subsequent requests only affects the Now value, which becomes lower.

Sometimes one will want to reset certain statistics, as we want to do now in preparation for subsequent steps of this example:

• Click the MBeans icon to the left in the tab of the JVM being monitored.
• In the MBean Tree, navigate to the flow which statistics you want to reset and select the Operations tab.
  In the example program there is but one single flow, the HelloWorldFlow1.
  
• Select the clearStatistics operation and click the Execute button in the bottom of the window.

If we now go back and examine the AverageProcessingTime attribute of the flow, it should be zero. Note that you may need to go to another node and then select the flow node again in the MBean Tree, in order for the values to be updated.

Adding a Chart

As with dials, charts can be added and removed as desired. A chart occupy significantly more screen-space than a dial. The good news is that charts can be folded up and unfolded, enabling you to quickly choose which chart(s) to view. In addition one chart may contain graphs of several values.
This picture shows the two default Processor and Memory charts in the folded state:

• To create a new chart, click the green plus-sign in the upper-right corner of the tab of the program being monitored.
  In the picture below, the plus-sign in question has been marked with a red circle.
  The new chart will be empty without any values being plotted.

•  To add a new graph to the chart, click the Add… button in the lower-right corner of the chart.
• Select an MBean attribute in the same way that an MBean attribute was selected for the dial added earlier.
  In this example I chose the Mule.hello-world -> MuleContext -> FreeMemory attribute, as shown in this picture:
  
• Click the OK button to select the attribute.
  The plotting of the graph starts immediately and scrolls left, as time passes:
  

Notifications

Not only is Java Mission Control able to present information on dials and charts, but it can also send out notifications (and take other actions as well) when the value of an MBean attribute surpass a certain value.
Actions that Java Mission Control can take can be extended by plug-ins. One such example is the Twitter Action plug-in.

I want to receive email notifications when the number of execution errors in the flow of my Mule application becomes higher than five during Saturdays and Sundays only.

• Click the MBeans icon.
  Located on the left in the tab of the JVM that is monitored, marked with red in the picture below.
  
• Click the Triggers tab in the bottom of the window.
  Marked with red in the above picture.
  In the Triggers tab, you will see a number of predefined trigger rules on the left and, if a trigger rule has been selected, the right side will display details of the selected rule.
• Click the Add button to the right of the list of trigger rules.
• In the Add New Rule dialog that appears, select the MBean attribute that you want the new rule to trigger on.
  In this example we will monitor the number of execution errors in our example Mule application. You find that attribute at:
  Mule.hello-world -> Flow -> HelloWorldFlow1 -> ExecutionErrors
  
• Click the Next button.
• In the dialog that appears, the conditions that need to be met in order for the rule to trigger are to be set:
  Max Trigger Value is the value that need to be reached. Set this value to 5.
  Sustained is the time in seconds during which the value need to be reached.
  This value can be used to avoid false positives from temporary peaks. In this example, the value will never decrease so this value does not matter.
  Limit Period is the minimum time in seconds that need to pass before another trigger event is generated. This value can be used to limit the number of notifications sent. Again, the value monitored in this example never decreases, so this value is left as-is.
  Trigger When Condition is Met means that an action will be taken, for instance sending a notification, when the value changes to being above the trigger value.
  Trigger When Recovering From Condition means that an action will be taken when the value changes from being above the trigger value to being below the trigger value.
  
• Click the Next button.
• In the next dialog you specify the action that is to be taken when the rule is triggered.
  I will use the Send E-Mail action for this example:
  
• If you do not have an outgoing mail server and required username and password handy, use the Application Alert action.
• Click the Next button.
• In the next dialog you can select the time during which the rule will be active.
  I want my rule to be active during weekends, when I am not at the office.
  
  You may also chose to skip this step altogether.
• Click the Next button.
  In the last dialog a name is given the rule and group of rules to which the rule belongs. I call my rule ”Mule Error Rule” and the group ”Mule Rules”.
• Click the Finish button.
  We have created a new rule, but the rule is not active.
• Navigate to the Mule Error Rule in the group Mule Rules and tick the check box next to it.
  Notice how information about the rule is showed in the right side of the pane, including information about the current value of the monitored MBean attribute.
  
• We are now ready to start issuing requests to the hello-world Mule service.
• In a web browser, issue requests to http://localhost:8081
  Check for notifications from Java Mission Control after each request.

Not surprisingly, there will be a notification after the sixth request. I received a mail that looked like this:

A notification event has been triggered!

Notification creation time was: Sun Apr 27 15:53:16 CEST 2014
The notification source is: org.mule.tooling.server.application.ApplicationDeployer hello-world (1867)
The notification rule is: Mule Errors Rule
Type description:
attribute://Mule.hello-world:name="HelloWorldFlow1",type=Flow/ExecutionErrors
Rule trigger condition: value > 5.0 for 0 seconds.
The condition was met for 1 seconds.

If you chose to become notified by application alerts, this is what such an alert will look like in Java Mission Control:

So we have been notified, which is all find and dandy, but lets say I went in to the office on the weekend and fixed the problem. Before going home again, I would like to reset the error counter so that I become notified if something happens again.

• Select the MBean Browser tab in the bottom of the pane.
• In the MBean Tree, navigate to Mule.hello-world -> Flow -> HelloWorldFlow1
  In the Attributes tab on the right you will then see the different attributes of the flow that can be monitored and their current values.
• Select the Operations tab.
• Select the clearStatistics operation and click the Execute button.
  This zeroes all the values of all the MBean attributes of the flow.
  Note that it is up to the application being monitored, Mule in this case, to supply an operation such as clearStatistics. Java Mission Control just give us the ability to execute operations exposed using JMX.

Since I configured my trigger to trigger when recovering from the condition, I get a notification as I clear the value:

Your notification has recovered!

Notification creation time was: Sun Apr 27 15:54:33 CEST 2014
The notification source is: org.mule.tooling.server.application.ApplicationDeployer hello-world (1867)
The notification rule is: Mule Errors Rule
Type description:
attribute://Mule.hello-world:name="HelloWorldFlow1",type=Flow/ExecutionErrors
Rule trigger condition: value > 5.0 for 0 seconds.
The condition was met for 2 seconds.

Notifications of recoveries are also very useful for instance in the case where there are several persons receiving the notifications. If one person fixes the condition, the others will receive a notification that the condition has recovered which means they do not need to go to the office on the weekend.