DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Join us tomorrow at 1 PM EST: "3-Step Approach to Comprehensive Runtime Application Security"
Save your seat
  1. DZone
  2. Software Design and Architecture
  3. Integration
  4. Concurrently Processing Split Payloads and Collecting Result in MuleSoft

Concurrently Processing Split Payloads and Collecting Result in MuleSoft

Learn how to use the MuleSoft Splitter and Aggregator components to achieve Split-Join functionality with parallel processing.

Rakesh Kumar Jha user avatar by
Rakesh Kumar Jha
·
Nov. 14, 17 · Tutorial
Like (7)
Save
Tweet
Share
15.45K Views

Join the DZone community and get the full member experience.

Join For Free

As a MuleSoft Certified Architect, Designer, and Developer, I recently worked on API implementations for one of our clients using MuleSoft’s CloudHub. One feature that we use in some APIs implementations is to split an input payload into multiple fragments, process each payload unit in parallel, and then merge the responses of each payload unit to send a final consolidated response.

In this article, I will demonstrate splitting a Mule payload into multiple fragments, processing each fragment in parallel, and finally, collecting the output of each fragment into a final response to be sent back to the consumer using MuleSoft.

As we know, Mule provides the Request-Reply scope to embed a "pocket" of asynchronous processing within a Mule flow. Since we are talking about parallel processing and sending a final response, we will use the Request-Reply scope in Mule. To split and join the payload, we will use the MuleSoft Splitter and Aggregator component, respectively.

Let’s get started and see how we can use these components to achieve Split-Join functionality with parallel processing in MuleSoft.

1. Setting Request and Reply Scope

The very first step we will take is to set up a request and reply scope of Mule and use VM transport to separate the Mule message processing thread.

Image title

As we can see in the above screen, the Input message coming to the Mule flow will be sent to an outbound VM queue, which submits an asynchronous request to another flow. The response VM will receive an asynchronous response from another flow.

2. Splitting Mule Payload

We will use the Mule Splitter component to split our Mule payload since I am working with an XML payload; I have used an XPATH expression to split my Mule payload.

Image title

As you can see in the above screen, I have used one variable component to store the original payload before splitting. My requirement is to call an external web service with each split fragment of the input payload. I have used Mule DataWeave to transform each payload fragment according to the external web service format to make a successful call. This transformation uses values from the original payload, which I have used from flowVar defined before the split component. The final payload is again being posted to a VM queue which will be processed on a separate thread to achieve parallel processing by running on different threads.

3. Calling an External Web Service With the Split Payload

Now let’s make an outbound call to the external web service for each split payload and get the response in a VM queue.

Image title

As you can see in the above screen, I have wrapped the external web service call inside the message enricher but mapped the payloads. This has been done to keep MULE_CORRELATION_GROUP_SIZE, MULE_CORRELATION_SEQUENCE, and MULE_CORRELATION_ID inbound properties, which the splitter generates for each payload unit. We have to keep these properties in the flow to use the collection aggregator to collect the final output.

4. Aggregating Final Responses

Now, we have responses from the external web service call for each payload unit in the VM which need to be aggregated.

Image title

As shown above, we have used the collection aggregator component to collect the responses for each payload unit in one payload and do the transformation to prepare the format we want the final response for the consumer. The final transformed payload has been published to the VM queue to which our main flow request-reply scope response VM is listening.

In conclusion, we received a Mule payload from the consumer with a repetitive block. We split each repetitive block using an XPATH expression and did a transformation to call an external web service. The response for each payload unit has been aggregated using the collection aggregator and sent back to the consumer in its defined format. The best part is that we have processed all of these steps in separate threads, and they worked in parallel.

Let’s share our knowledge to expand our MuleSoft community.

Thank you!

Payload (computing) Processing MuleSoft Web Service

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • What Is Policy-as-Code? An Introduction to Open Policy Agent
  • The Enterprise, the Database, the Problem, and the Solution
  • Explaining: MVP vs. PoC vs. Prototype
  • NEXT.JS 13: Be Dynamic Without Limits

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: