{{announcement.body}}
{{announcement.title}}

MuleSoft Polling-As-A-Service

DZone 's Guide to

MuleSoft Polling-As-A-Service

Let's take a quick look at how MuleSoft can be used to implement Polling Consumer pattern as a service. Also, explore how it can minimize the risk of the problem found below.

· Integration Zone ·
Free Resource

Introduction

MuleSoft is a leading Integration-Platform-as-a-Service (IPaaS) that provides an integration platform to help businesses connect data, applications, and devices across on-premises and cloud-operated environments. Anypoint Platform includes various tools to develop, manage, and test integration solutions and APIs and supports most of the widely used integration patterns.

One such frequently implemented Integration pattern is Polling Consumer. MuleSoft provides a Poll component that helps to implement this pattern. Using this component, any Mule application can poll for a message, process it and then poll again for another message from the source system. Note that there are many advanced capabilities that this component provides and will not be discussed here.

Problem

There are many projects that use the Poll component and its capabilities to implement the Polling Consumer pattern. However, there is a challenge that is also usually noted when many applications, deployed on on-premise MuleSoft runtime, uses this component. Over time, we will notice that the poll runs throughout the clock (across applications) and the runtime server gets busy which can cause stability issues. This usually happens due to the lack of evenly distributed polling cycles across applications. Let’s say an application A needs to poll every 3 mins and application B polls every 7 minutes. It is mostly possible to make both the application poll every 5 mins. In my experience, for any Data Integration scenarios, the polling requirements of interfaces can be aligned in an evenly distributed fashion.

It should also be noted that working with Poll component is always costly, as in many cases poll cycles will be triggered without any processing due to no data being fetched from the source system. This will waste CPU cycles as the interface will be instantiated yet not doing any work. The first step is to make (or even mandate) the distribution of polling cycles in an even fashion.

However, mandating the distribution of the polling cycles is also not practically easy when working on larger projects that have many interfaces being developed in parallel. This write-up explains a pattern that can be adopted to provide Polling as service in the platform and have this capability implemented to generate Immutable events which applications can subscribe to and perform the logic after consuming the event.

Solution

The below diagram explains the pattern at a high level. There are three major actors to the solution:

  • Poll Event Producer
  • Event Topic
  • Event Consumers

Image title

Poll Events Producer

This is a Mule application that will have the Poll interfaces that generate events on an evenly distributed time. This application is going to help control the polling cycles centrally yet allowing other applications to implement their logic independently (by decoupling these interfaces using a broker, which will be discussed later). Having this managed centrally will help the operations team to control the poll triggers. Let’s say that a blackout window needs to be planned as part of a maintenance schedule. All that the operations team will have to do is to pause the appropriate poll in the platform. It will then naturally pause the application that would have been triggered during that time window. If done otherwise, it would have been necessary to identify the applications individually and pause the poll (imagine there are hundreds of interfaces running in the platform).

Event Topic

It is important to understand that the events generated by the Poll Events Producer should be immutable. It means that the events should be sourced to the consuming application strictly through a reliable interface as replaying the poll schedule is not safe and shouldn't be allowed. It is also important to deliver the events in a First-In-First-Out (FIFO) fashion as the consuming application would rely on sequencing.

I have proposed the Active MQ — Artemis server to implement the broker that will store and forward the events. Below points were considered which makes this server a good fit:

  • The Artemis server supports JMS specification which can be easily integrated with Mule
  • FIFO implementation is supported out of the box and doesn’t need any specific logic to be placed in the application
  • It implements persistence with highly performing Journal with reduced disk head movements
  • It supports larger in-flight messages by using the feature called Paging
  • It also supports protocols like STOMP, MQTT, AMQP, which can help utilize this central polling feature to be integrated with non-Mule applications
  • The server also supports pausing and resuming consumers individually. This also helps in scheduling downtime for an interface without disturbing other interfaces that are part of the same schedule cycle as the producer is not paused in this case

The Poll events should be published to Topics which can have many consumers.

Event Consumers

These are Mule (or even non-Mule) applications that will consume the events from the topic and trigger the logic. The consumers can also subscribe to multiple topics depending on the requirement. Mule application can connect to the broker using the JMS. If the consumer needs a replay of the event in case of failures, the application can be implemented to support JMS transaction and roll-back the event back to the broker in case of any exception. Since the events are sourced in a FIFO manner, the application can safely replay the event binding it with a transaction.

Event Data Model

The below data model represents an event that can be created by the Event Producer.

{
  "Event": {
    "Header": {
      "ID": "1fb4fbb1-66fa-4664-9045-93f76e52e6d1",
      "Name": "POLL-Event",
      "Timestamp": 1535531971,
      "Version": "v1"
    },
    "Body": {
      "LastPollDate": "2018-08-29T01:00:00Z",
      "CurrentPollDate": "2018-08-29T02:00:00Z"
    }
  }
}

Note: It is important to version the event so that we can support breaking data model changes and the events can be consumed based on this version by using JMS Selectors (if the version is used in message properties) or filters in the consuming applications.

Conclusion

This solution can be adopted to centrally manage and control the schedules of any polling interfaces deployed on the platform. If we notice an unusual server load at any time, it is easy to identify the consumers active on that time frame. Scheduling maintenance activities and downtime is much simpler than if the Poll capability is implemented on every interface.

Topics:
mulesoft esb ,polling ,integration ,ipaas ,data integration

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}