Introduction to DataWeave

Weaving is a method of textile production in which two distinct sets of yarns or threads are interlaced at right angles to form a fabric or cloth. DataWeave is the tool by MuleSoft using which we can weave one or more different types of data together to create another data format.

When you are trying to integrate disparate systems, it is possible that your source system understands data in one format, say JSON and target system understands data in another format, say XML. In this scenario, it is necessary to use Data Transformation for converting data from JSON-to-XML.

DataWeave is a MuleSoft tool for transforming data. The Transform Message component carries out the transformation of Mule message that follows a transform script. You can write your own transform script in DataWeave code or you can use UI to build the script.

It attempts to bring together features of XSLT (mapping), SQL (joinBy, splitBy, orderBy, groupBy, distinctBy operators), Streaming, Functional Programming (use of functions in DataWeave code) to make it a power-packed data transformer.

DataWeave supports DataSense i.e., metadata from connectors, schemas, and sample documents is availbale to more easily design transformations. It also provides content assist while you are coding and auto generates lines of code from actions performed on UI.

e.g. If we drag and drop element from input section onto another element in output section, then corresponding code is automatically generated.

If you add the Transform Message component in your Mule flow, its Properties view looks like this:

The properties view is divided into different components as shown in the following screenshot:

The Graphical UI

The Graphical UI exposes the known input and output structures. You can easily click and drag one field onto another to map these structures. You can do following through the UI:

1. Drag an element from the input structure over to another on the output structure. This casts a line that joins them and also generates necessary lines of DataWeave code that describes this mapping.

2. Drag a high-level object that contains inner fields inside it onto another in the output.

3. Double-click on an output field to add it into the DataWeave code with a static value. This adds an Fx icon next to it as well as a line to the DataWeave code that assigns a default null value to the field. You can change this value in the code as per your need.

4. Select an element to have its corresponding line in the DataWeave code highlighted. This helps when dealing with large transforms.

5. If an input field is mapped to two or more output fields, you can right-click it and then select which of the multiple outputs you want to highlight in the DataWeave code.

6. Filter the views displayed in the input and output structures by typing a name in the search boxes at the top. Only those fields that match your search are then displayed. This is particularly useful when you are dealing with large data structures with many nested elements.

DataWeave Text Editor

This is the section where you write the actual DataWeave code that carries out the transform.

DataWeave Header

The header of a DataWeave script defines directives that indicate the input, the output, and constant values or functions that can be referenced anywhere in the code. These directives can be manually set on the code or automatically defined based on the metadata of your flow or what you map in the UI. The structure of the header is a sequence of lines, each with its own directives. The header section is separated from body by a separator ---.

Through the use of the following directives, key aspects of the transformation are defined:

1. DataWeave version, i.e., %dw 1.0

2. Output type, i.e., %output application/java

3. Input type, i.e., %input payload application/xml

4. Namespaces to import into your transform, i.e., %namespace ns http://example.org/my-service/

5. Constants that can be referenced throughout the body, i.e., %var discount=0.05

6. Functions that can be called throughout the body, i.e., %varnewUrl(name) url ++ "?title=’" ++ name ++ "'" 

You can also use %function for defining functions.

All directives are declared in the header section of your DataWeave document and act on the entire scope of it.

DataWeave Body

The body contains expressions that generate the output structure. In dataweave body, we define an expression that generates ouptut consisting of three data types:

1. Simple values: String, boolean, number, date, regex

2. Arrays: Represented as comma separate values

3. Objects: Represented as collection of key-value pairs

In the body, it is possible to write expressions which are composed of other expressions. These expressions can be nested inside Arrays or Objects or we can use operators for creating them.

Let us take an example of an API which returns information about books in JSON and our backend only accepts XML input. The XML contains total no. of books. Books are sorted by the year in which they were published. Price for a particular book is displayed only if it is greater than 30.00 dollars. In this case, we can use DataWeave for easy transformation.

Our input JSON is in the following format:

[{
"title" : "Only Time Will Tell",
"author" : "Jeffrey Archer",
"year" : "2011",
"price" : "30.00"
}, {
"title" : "The Hostage",
"author" : "James Patterson",
"year" : "2016",
"price" : "49.99"
}, {
"title" : "Harry Potter",
"author" : "J K. Rowling",
"year" : "2005",
"price" : "29.99"
}, {
"title" : "Twilight",
"author" : "Stephenie Meyer",
"year" : "2007",
"price" : "39.95"
}
]

Output XML has following format:

<bks:bookstore xmlns:bks="http://example.com/bookstore">
<bks:totalBooks>4</bks:totalBooks>
<bk:book xmlns:bk="http://example.com/books" title="http://example.com/books/book?title='Harry Potter'">
<year>2005</year>
<author>J K. Rowling</author>
</bk:book>
<bk:book xmlns:bk="http://example.com/books" title="http://example.com/books/book?title='Twilight'">
<year>2007</year>
<price>39.95</price>
<author>Stephenie Meyer</author>
</bk:book>
<bk:book xmlns:bk="http://example.com/books" title="http://example.com/books/book?title='Only Time Will Tell'">
<year>2011</year>
<author>Jeffrey Archer</author>
</bk:book>
<bk:book xmlns:bk="http://example.com/books" title="http://example.com/books/book?title='The Hostage'">
<year>2016</year>
<price>49.99</price>
<author>James Patterson</author>
</bk:book>
</bks:bookstore>

Then, our DataWeave Header will look like below:

%dw 1.0
%output application/xml
%input payload application/json
%namespace bks http://example.com/bookstore
%namespace bk http://example.com/books
%var url="http://example.com/books/book"
%function newUrl(name) url ++ "?title='" ++ name ++ "'"

DataWeave Body section will look like:

bks # bookstore: {
bks # totalBooks: sizeOf payload,
(payload orderBy $.year map {
bk # book @ (title: newUrl($.title)): {
year: $.year,
(price: $.price)when $.price > 30.00,
author: $.author
}
})
}

Note: DataWeave is a new feature of the Mule 3.7 runtime that replaces the DataMapper.