The Power of Transformation, Part I: An Introduction to DataWeave

When you work with the integration of different systems, sooner or later, you will have to transform data. In today’s world, very few systems talk in the same data language. For example, you may have an HTTP server that understands JSON, but it needs to talk to a server that only understands SOAP XML. Even if it understands JSON, the chances of having same JSON schema or format is very slim, so in these cases, you need to do transform the data.

Mule provides a very powerful tool to do these kinds of transformations: DataWeave.

DataWeave (DW)

DW is a new data transformation engine of Mule. DW is the replacement to DataMapper. You may have used DataMapper before; it provided a graphical drag and drop approach to data transformation. Please keep in mind that from Mule 3.7, Mule runtime hasn't had DataMapper and if you want to use it, you need to separately provide the dependencies. From Mule v4, there will be no longer support for DataMapper! So, I highly encourage everyone to either move to DW or convert your DM.

Canonical Data Format

Before we embark on the journey to understand DW, we need to understand that DW is designed in canonical data format standards. In simple terms, canonical data format is a way in which we represent any kind of data or object in a standard format. Any expression we write in DW will be in canonical format.

There are only three data types that can result from a DataWeave expression:

1. Simple types (like strings and numbers).
2. Arrays.
3. Objects.

Each of these can be expressed literally as below:

DataSense and DataWeave

When we work with DW, we get to use the powerful abilities of DataSense. With DataSense, you get suggestions and views of data with which you are working. You should also be mindful of the mime types of the Message data that you have to transform. When this is not set explicitly at design time and you use any of your connectors, this will default to application/Java.

For HTTP requests and responses, Mule will look at the content type header and set it accordingly. You should set the mime type explicitly when you are using the set variable and set payload processors. Failing to set the mime type explicitly for XML content will result in it being interpreted as a mere Java string by the engine.

Variable Reference Expression

Understanding DW is all about understanding how DW engine normalizes your input data and how it is represented by the engine internally whenever references are made to variables, session variables, inbound properties, outbound properties, etc. All these expressions convert the incoming data into the canonical DW format that we discussed above. So, thinking in terms of DW simple types, arrays, and objects will help to write better transformation.

Below is the sample of XML conversion to DW literals. If we change the output type to application/DW, we can see that the conversion is done by the engine of the input XML. Here is how all the elements of XML are normalized as a key:value pair in the object:

In the case of complex element types, the normalized value is itself an object and repeating values are normalized as repeating key:value pairs.

Anypoint Studio

Anypoint Studio supports DW development with a DW editor. In DW editor, you will find three panes.

On the left, DataSense displays the structure of the incoming message along with any example of data you have provided.

On the right pane, the expected outgoing structure is displayed as per DataSense. It will also display the design time result of the transformation that constantly refreshes itself as the expression of transformation is written.

The middle pane is where you write your transformation logic. It is divided into two sections. The header is where the declaration of mime type of transformation output is done, along with some reusable functions and global variables as also namespaces for XML use cases. The mandatory part of the header is the declaration of the DW version (i.e., %dw 1.0 and the output mime type).

Below the dotted line is where you write the transformation expressions. Only one expression is needed for this. There is a number of expression types, but the majority of the transformations can be done with semi-literal expressions of DW objects.

The same DW transformer can produce multiple outputs from the same incoming message. This can be achieved by clicking the plus circle at the bottom right of the transformation pane. With this, you will get another transformation pane in which you can select the output from the drop-down at the top of the transformation pane. The important thing to note is that you can use different flow vars or specify different targets for your transformation.

Output Rendering

The DW engine separates the actual transformation process within the canonical format from the final rendering of the same in the output mime type you defined in the header.

Irrespective of how complex your expression is, if it’s an object with deep-nested structure and expressions and operators which combine expressions, all of these must be executed first and return their value before the outer expression returns its value. It is this final value that is rendered in the output mime type.

Be aware of the choice of mime type, as there will be some limitations associated with them. Arrays will not be rendered in XML. Choose to repeat keys instead. Likewise, repeating keys cannot be rendered in JSON. Generate arrays instead.

To render an object to XML in line with the rule that XML documents may only contain one root element, the object may only contain one key:value pair. Its value can, as we have seen, itself be an object of any complexity.

Conclusion

This was an introduction to DW. I will be creating further videos and blogs related to DataWeave to go more in details of various ways to use DW in your transformation.

Let me know if you have found this blog helpful and informative.