{{announcement.body}}
{{announcement.title}}

Filtering Data Using DataWeave in Mule 4 Batch

DZone 's Guide to

Filtering Data Using DataWeave in Mule 4 Batch

Let's look at filtering data using DataWeave in Mule 4 batch.

· Database Zone ·
Free Resource

DataWeave is a MuleSoft expression language for accessing, querying, and transforming data that flows through the Mule application. It is tightly integrated with the Mule runtime engine, which reads the script and expression in our application. It is a powerful template engine that transforms data to and from any kind of format like CSV, JSON, XML, MAP, POJO's, etc.

Dataweave now() function is used to return the current date and time when used inside our application. 

Mule 4 Batch

Mule can process messages in batches. It splits the large messages into individual records that are processed asynchronously within batch jobs.

Phases of Batch Job

Each batch job contains three different phases in Mule 4:

1. Load and Dispatch: This is an implicit phase. It works behind the scenes. In this phase, Mule turns the serialized message payload into a collection of records for processing in batch steps.

2. Process: This is a mandatory phase of batch. It can have one or more batch steps to asynchronously process the records.

3. On Complete: This is an optional phase of batch. It provides a summary of the records processed and helps the developer to get an insight of which record was successful and which one failed so that you can address the issue properly.

                  failedRecords, loadedRecords, processedRecords, successfulRecords, totalRecords

Sample CSV

The sample CSV file is something like this. Here, I am only showing just 3-4 records, but in actuality, I had 25 around records. You too should test this application with that many records. You can have the values of your choice.

id, name, startDate, endDate, department, designation

  1. Kuldeep Singh Rana, 2019-1-1, 2079-1-1, software, developer

  2. Atul Singh Chauhan, 1959-1-1, 2018-1-1, software, architect

  3. Anil Singh Negi, 2019-1-1, 2079-1-1, finance, associate

  4. Siddharth Singh Chauhan, 1960-1-1, 2018-1-1, software, architect

Mule Flow Structure:

Image title

On New or Updated File Listener: In the Listener, configure the working directory (that contains the file) and make sure to move the input file to another directory after the file has been processed.

<file:listener doc:name="On New or Updated File" doc:id="d1018daf-3334-4088-a3ea-2127b0a80ce1" config-ref="File_Config" recursive="false" moveToDirectory="C:\Users\kuldeep\Desktop\Mule 3 slides\output">
<scheduling-strategy >
<fixed-frequency />
</scheduling-strategy>
<file:matcher filenamePattern="*.csv" />
</file:listener>

Transform Message (Dataweave 2.0): The Transform Message component transforms the CSV file into JSON payload, which then will be used by the batch job to insert the data into the database.

<ee:transform doc:name="Transform Message" doc:id="6a384cca-c6df-40bc-b05f-178035d4275a">
<ee:message>
<ee:set-payload><![CDATA[%dw 2.0
output application/json
---
payload map {

id: $.id,
firstName: ($.name splitBy(' '))[0],
middleName: ($.name splitBy(' '))[1],
lastName: ($.name splitBy(' '))[2],
joiningDate: $.startDate,
leavingDate: $.endDate,
designation: $.designation,
department: $.department,
date: now() as String {format: "yyyy-MM-dd"}
 }]]></ee:set-payload>
</ee:message>
</ee:transform>

Batch Job: Splits the transformed JSON payload into individual records to be processed inside the batch job.

<batch:job jobName="employeeBatch_Job" doc:id="a006546e-0dcb-4862-baa1-e5c2aac4e667" >
<batch:process-records >
<batch:step name="Batch_Step1" doc:id="18865ef4-2c73-440d-844e-6cb7215dad63">
<set-variable value="#[payload.date]" doc:name="current date" doc:id="ab3ef797-f555-4da7-aeb5-f4b2a5a25d28" variableName="currDate" />
<set-variable value="#[payload.leavingDate]" doc:name="leaving date" doc:id="d5d8ec68-7806-4197-852d-02a8afd32013" variableName="ldate" />
<choice doc:name="Choice" doc:id="48de0622-4a55-4c48-bbd8-bc3019f603bb" >
<when expression='#[vars.ldate &lt; vars.currDate]'>
<db:insert doc:name="In active emp" doc:id="68bef45d-92cb-4feb-a90c-4de8c8666ac4" config-ref="Database_Config" >
<db:sql >INSERT INTO INACTIVEEMP(EMPLOYEEE_ID,First_Name,Middle_Name,Last_Name,Joining_Date,Leaving_Date,Designation,Department)
VALUES (:id,:firstName,:middleName,:lastName,:joiningDate,:leavingDate,:designation,:department)</db:sql>
<db:input-parameters ><![CDATA[#[payload]]]></db:input-parameters>
</db:insert>
</when>
<otherwise>
<db:insert doc:name="active emp" doc:id="9834ea1b-ef9d-41f6-8990-9e1ea08ab634" config-ref="Database_Config" >
<db:sql >INSERT INTO ACTIVEEMP(EMPLOYEEE_ID,First_Name,Middle_Name,Last_Name,Joining_Date,Leaving_Date,Designation,Department)
VALUES (:id,:firstName,:middleName,:lastName,:joiningDate,:leavingDate,:designation,:department)</db:sql>
<db:input-parameters ><![CDATA[#[payload]]]></db:input-parameters>
</db:insert>
</otherwise>
</choice>
<logger level="INFO" doc:name="Logger" doc:id="c77c3174-be38-4953-b8cb-1e3902b414ea" message="#[payload]" />
</batch:step>
</batch:process-records>
<batch:on-complete >
<logger level="INFO" doc:name="Logger" doc:id="125e0792-7d3b-41d6-ba5c-a69d7f5561fe" message="#[payload]"/>
</batch:on-complete>
</batch:job>

Database Insert: Create two tables in the database. One table will store active employees and the second table will store the inactive employees. Look at the CSV file to know the number of columns required in each table.

The database insert query lets us insert data into our database by configuring the connector with required libraries and connections that include host, port, username, and password.

Image title

Complete Code

The complete code for the application can be found at my GitHub account here.

Once the application is running successfully, you will get a successful response from the 'On complete' phase of batch.

Image title

Conclusion

This post shows how to use the DataWeave function to separate active employees and inactive employees of an organization into two different tables in the database.

The application matches the employee's last date of working with the current date (system date) and executes the process.

Thank you!

Topics:
integration ,mule 4 ,mule 3 ,dataweave ,dataweave 2 ,tutorial

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}