Using the MuleESB Registry and Property Files for Value Translations

Value Translations in Integration

Value translations (or transcodifications, or cross-references) are very often necessary in the data mappings within integrations and API and service implementations.

There are two broad use cases for value translations.

1. We need to relate the values of a field in the source and target data models. For example, we would not expose technical backend codes (i.e., a SAP partner function code such as "WE") in a high-level API, but would instead use "ShipTo".  Therefore we need a "ShipTo" --> "WE" value translation in our data mapping.

2. We need a simple form of content enrichment in our data mapping that does not require a secondary source system, for example, associating a set of descriptions with a set of codes such as units of measure ("KG" --> "kilogram", "MT" --> "meter", etc.). 

Clearly, hardcoding these translations in the integration logic in a big conditional or "switch" statement is neither flexible nor reusable. We need to:

1. Be able to easily configure the translations (independent of code).

2. Build reusable lookup logic that uses the configured translations.

The article shows how to implement a very simple reusable value translation logic using Mule ESB (I'm using version Mule 3.8 for my examples).

The value translation scheme is based on Java property files and on exploiting the Mule ESB Registry as a simple in-memory cache. 

I need to emphasize that I am taking a very simple approach here that cannot be suited to every use case. A best-practice general solution would involve a database with a cached query mechanism (caching of the lookup would be done via the Mule Cache Scope or via a Java component using some type of third-party cache such as Ehcache).

Requirements of the Solution

As stated above, we would like to configure our translations for the Mule application in a simple property file, such as the one below:

#
# translations.properties
#
#  
# UoM descriptions
UOM.EA=each
UOM.KG=kilogram
UOM.LT=liter
UOM.MT=meter
UOM.KM=kilometer
#
# Incoterms descriptions
Incoterm.FH=Free House
Incoterm.FOB=Free On Board
Incoterm.DDP=Delivery Duty Paid
Incoterm.FCA=Free Carrier
Incoterm.EXW=Ex Works

As can be easily guessed, the property names represent the lookup keys (i.e., the values to be translated), and the corresponding property values represent the lookup values (i.e., the translated values). Suitable prefixes (such as  UOM  and  Incoterm  can be used for functional categorization of the properties, thus avoiding the use of multiple property files by category). Within each functional group, composite keys may be used, as wel. For example:

UOM.LT.en=liter
UOM.LT.fr=litre
UOM.LT.it=litro

Thanks to the "prefix" approach, a Mule application could thus use a single property file named, for example,  translation.properties and located under the  src/main/resources  folder, which is on the application classpath:

The lookup logic should automatically load this property file as a java.util.Properties object in memory when the application starts up. Furthermore, a generic synchronous Mule flow (propertyLookup) should be available for reuse across multiple Mule applications to provide them with the value translation capability.

The propertyLookup flow must be referenceable in DataWeave transformations within these Mule apps via the DateWeave lookup function:

lookup("propertyLookup", {key: <value to be translated>})

Implementation of the Solution

The implementation of the value translation functionality is a very simple Mule propertylookup application:

There are just two flows in this Mule application:

1. The  loadTranslations flow (to be executed only one at application startup), which looks in the classpath for a property file named translations.properties, loads its contents into a Properties object and registers this object into the Mule registry.

2. The  propertyLookup  flow (configured with synchronous Processing strategy), which exposes the translation and lookup functionality.

The loadTranslations Flow

This flow needs to be executed every time the application is started or restarted. Since in the current version of Mule (3.8) there is still no "Execute on startup" attribute for a flow, the most correct (albeit not the simplest) way around this is a Mule CONTEXT Notification listener (as shown in David Dossot's answer to this StackOverflow question).

However, to simplify the solution and keep the focus on the property lookup I have used the workaround of the Quartz connector with Repeat Count = 0 and quartz:event-generator-job:

The property load logic is found in this Groovy script:

The code above is pretty straightforward, but depends on one key fact: after we package this propertylookup application in a JAR file, and we place this JAR as external library into the classpath of the "client" Mule application, the invocation...

Thread.currentThread().getContextClassLoader().getResourceAsStream("translations.properties") 

...will execute against the class loader of the client application. Therefore, it will be able to find the translations.properties file in the classpath of the same client application. 

Clearly, since translations are application-specific, each client application on a Mule server will have specific contents in its  translations.properties file and these contents will not clash across applications since each application has its own ClassLoader instance. 

The  propertylookup  application does NOT contain any translations.properties file.

The final Logger step just logs that the properties were indeed loaded into the registry.

The propertyLookup Flow

We just have a single trivial Groovy script here:

We just retrieve the Property object and then use it. One can see that for the lookup to work the caller of the propertyLookup flow will have to pass as input payload an object with a property named key.

Important note: propertyLookup must be defined as a flow and not as a sub-flow, as a sub-flow is not a lifecycle-enabled object in Mule and attempting its use in the "client flow" causes the following error:

Could not apply lifecycle into prototype object propertyLookup (org.mule.api.MuleRuntimeException).

Use

Once the lookup application described above, which contains Mule configuration file propertylookup.xml has been packaged into a JAR file (propertylookup.jar) this JAR can be added to the classpath of any "client" Mule application as referenced library either:

• manually (in Anypoint Studio: right-click on client Mule project > Build Path > Add External Archives...), or

• as a Maven dependency (best practice).

In the client Mule application, in order to use be able to access the propertyLookup flow (i.e., resolve it via the DataWeave lookup() function), we additionally need to import the Mule configuration file into the underlying Spring configuration:

    <spring:beans>
        <spring:import resource="classpath:propertylookup.xml"/>
    </spring:beans>

This configuration can be also generated via the Global Elements --> Create --> Beans --> Import dialog.

In the client application, once the said setup (propertylookup.jar in the classpath and global configuration above) is completed and we have a valid translations.properties file under our  src/main/resources folder, then we are all set!

In the client application, I can write a DataWeave transform like this...

%dw 1.0
%output application/json
---
{ 
value: lookup("propertyLookup", {key: flowVars.key})
}

...and (as long as the property file contains the needed properties), the translation will just work!

In the example above the lookup key (i.e., the value to be translated) comes from a flow variable also named key, but of course, it can come from anywhere within the DataWeave transform input.

My Mule application to test the lookup, which contains the transform above, is trivial:

<?xml version="1.0" encoding="UTF-8"?>

<mule xmlns:dw="http://www.mulesoft.org/schema/mule/ee/dw" xmlns:metadata="http://www.mulesoft.org/schema/mule/metadata" xmlns:http="http://www.mulesoft.org/schema/mule/http" xmlns:ee="http://www.mulesoft.org/schema/mule/ee/core" xmlns:scripting="http://www.mulesoft.org/schema/mule/scripting" xmlns:tracking="http://www.mulesoft.org/schema/mule/ee/tracking" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:spring="http://www.springframework.org/schema/beans" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/ee/tracking http://www.mulesoft.org/schema/mule/ee/tracking/current/mule-tracking-ee.xsd
http://www.mulesoft.org/schema/mule/scripting http://www.mulesoft.org/schema/mule/scripting/current/mule-scripting.xsd
http://www.mulesoft.org/schema/mule/ee/dw http://www.mulesoft.org/schema/mule/ee/dw/current/dw.xsd
http://www.mulesoft.org/schema/mule/ee/core http://www.mulesoft.org/schema/mule/ee/core/current/mule-ee.xsd">

    <http:listener-config name="HTTP_Listener_Configuration" host="0.0.0.0" port="8081" doc:name="HTTP Listener Configuration"/>
    <spring:beans>
        <spring:import resource="classpath:propertylookup.xml"/>
    </spring:beans>
<flow name="testLookupFlow">
        <http:listener config-ref="HTTP_Listener_Configuration" path="/testlookup" allowedMethods="GET" doc:name="HTTP"/>
        <set-variable variableName="key" value="#[message.inboundProperties.'http.query.params'.key]" doc:name="Variable"/>
        <dw:transform-message metadata:id="5540c302-bd8f-44ca-85c3-7bb68599d62b" doc:name="Transform Message">
            <dw:input-payload/>
            <dw:set-payload><![CDATA[%dw 1.0
%output application/json
---
{ 
value: lookup("propertyLookup", {key: flowVars.key})
}]]></dw:set-payload>
        </dw:transform-message>
    </flow>

</mule>

Conclusion

Although the Mule registry is not recommended for use a general purpose data cache, in this case, its use is to be considered rather safe, since we are registering a single java.util.Properties object at application startup and then just retrieving this same object over and over at each lookup.

The biggest limitation of the technique presented consists in the fact that an update in the property file can only be captured at runtime after the client application is redeployed or restarted.

It is technically possible to trigger the loadTranslations flow on demand, too (via a composite source containing both the quartz endpoint and an HTTP endpoint), but this approach is inconvenient and may not be thread-safe.

Registering a Properties object containing a very large number of key-value pairs is also not recommended. However, in cases where we are dealing with a not-too-large number of translation entries that are also static in time, the solution presented is well worth a try!