Event-Driven Process Orchestration: A Practitioner’s Viewpoint

Customers today seek agile, nimble, flexible, and composable services. Services that are unhindered and unencumbered. Services that are easy to access and even easier to experience. Services that are quick and precise. These factors affect the collective CSAT and NPS of a modern-day enterprise. Enterprises acknowledge this, and hence, around 85% of medium to large-sized enterprises are already using the microservices architecture. The distributed architecture of microservices applications renders the components of the applications independent, decentralized, failure resistant, maintained, and upgraded in isolation, therefore fueling self-sufficiency, scalability, system reliability, and simplified service offerings. 

However, while microservices architecture readies the application for agile servicing, true customer experience arises not solely from the decoupled application components but the way in which every step in a customer success workflow triggers a logical subsequent step automatically to ensure customer delight. This is because as the business process extends and more components get added, “cohesion chaos” can become a reality. The absence of proper orchestration of process steps in a logical flow, keeping the customer end goal in mind, can quickly render the supposed benefits of the microservices landscape futile.

Therefore, the microservices applications can be clustered, and the sequence of steps in each process flow can be orchestrated via an event streaming platform like Kafka while being managed and governed by a BPM or Integration engine, say a RHPAM or a Camunda or even MuleSoft that promises seamless co-existence of API led architecture and events-based architecture. Such an architecture will encapsulate various microservices in an event stream, with each service listening intently to the action taken by a user through the topic published into the event stream and the basis of that action, triggering a corresponding service as per the logical process flow defined. Therefore, each service is self-responsible and acts or reacts basis their trigger point in the true spirit of event-based orchestration. 

In my conversations with enterprises cutting across various geographies and domains, customers usually test the waters of this model of servicing through an event streaming platform like Kafka or centrally orchestrate the service through a BPMN engine like RHPAM. However, both these options have their own pros and cons. The hybrid model, which considers both a BPMN engine for centralized process orchestration while coordinating with the worker services via an event stream, is gaining very good traction and the rise of Enterprise Integration behemoths such as MuleSoft, which claim to support event-driven architecture alongside the more familiar API led integration is making the solution options very interesting for customers. 

Let’s evaluate these scenarios one by one by taking the use case of Bob, who wants to book a Ridola from Amsterdam to Den Haag, and see which services need to interact with each other to make the experience pleasant for Bob and how some of these tools make the experience seamless.

The Mechanism at Play in Bob’s Ordering of a Ridola

In ordering a Ridola, assuming he is signing in for the very first time, Bob would open the application and will undergo a journey through the following microservices- Customer Profile Service - Location Service, Cab and Driver Management, Trip Management, and Payments.

The value chain, in layman's terms, flows with Bob opening the app, registering himself by providing his profile details, and then choosing the locations to and from which he needs to journey, upon which Ridola searches and recommends the available cabs and drivers in his vicinity with the associated tariff. Upon Bob’s decision of the cab, the trip management service manages the trip by guiding the driver and getting Bob his chosen cab, initiating the trip from Amsterdam towards Den Haag, post the completion of which, Bob is requested for the payment and upon payment a bill is sent to his email address.

Options Available for Ridola To Provide a Well-Orchestrated Service to Bob

BPMN-Driven Centralized Orchestration

In this approach, Ridola would ingrain the business workflow logic in the centralized BPM Engine (say RHPAM, Camunda, etc.) These BPM technologies follow the Business Process Modeling Notation (BPMN), which is a standard for modeling business processes. The integration between Passenger UI and the application would be through REST. The moment Bob logs into the application, the centralized engine (the brain of the business workflow) triggers a command to the worker services and awaits their response. Such commands are issued by the Java delegate or something like AWS Lambda. The overall “cab ordering service” is built as a Springboot microservice.

This means the first command from the centralized engine, upon Bob’s log-in, will be issued to the Customer Profile Service, which shall pop up and request Bob to sign in. Upon the completion of that step, the centralized engine shall command the location service to kick in that enquires Bob on his current location and his origin and destination stations. Thereafter, the cab and driver management service gets triggered centrally, and so on and so forth.

BPMN-based orchestration architecture

The moot point to note in this approach is the central orchestration engine triggering the actions on the worker services. The worker services do not pass the command to each other. This is all centrally managed using BPMN standards, thus enabling easy maintenance, support, and upgrades. The BPM engine can also become the single repository providing transaction updates, service state, etc., and, therefore, a source to drive observability. 

However, on the flip side, such one-on-one integration between the centralized orchestration engine and each of the worker services can render the landscape “tightly coupled” with “point-to-point integration,” precisely what an enterprise wants to avoid by embracing a microservices architecture.  Therefore, while this approach is fair when the number of transactions is low, for a large enterprise like Ridola with a massive number of transactions, such an approach can very quickly heat the orchestration engine and spoil the experience that Ridola wants to provide the end customer.

Event-Driven Orchestration

Vis-à-vis the centralized approach explored in the above section, many customers seem to be choosing an event streaming platform-led orchestration. This could entail the usage of technologies such as Apache Kafka, IBM Data Streams, Amazon Kinesis, Confluent, etc. This is a decentralized approach where the business logic is imbued across all the microservices that Bob shall encounter while getting his cab service from Ridola. Each of these services – be it customer profile service, location service, cab & driver management service, trip management, or payment service, is integrated with a central event stream (say a Kafka or a Confluent) and listens to the required topic that pertains to the service. This topic that is published to the event stream is a result of the action taken by Bob (by signing into the app) or a result of the action taken by the preceding service (say, customer profile service). This topic is also a trigger or cue for the next service (say location) to get triggered by asking Bob about his location and his from and to stations. Likewise, each service becomes aware of its turn and responsibility through the topic published on the event stream, and the cab ordering process gets streamlined service after service in a true event-driven manner.

Event Streaming-based orchestration

While this approach brings in the “loose coupling” that the first approach lacked, the maintenance and upkeep of services will become tedious when the overall business process undergoes a change, thereby affecting the sub-services within that value chain. Likewise, there is no centralized observability of performance available, and each service needs to be referred to for logs and traces. This means in case of any error or troubleshooting, we would need to check each service one by one, which would take time.

However, if a process value chain is fairly established and the services comprising the process are quite established, such an approach can work. The business owner needs to evaluate the scenarios and take a call. 

Hybrid Approach: BPMN-Led Event Orchestration

Many customers understand the potent combination of the earlier two approaches and choose to undertake proof of value and, thereafter, full-blown implementation of such a hybrid solution. In this, while the centralized BPM engine (RHPAM or Camunda) houses the business logic, the communication with worker services downstream doesn’t take place in a point-to-point manner. This communication is established via the event broker. Therefore, loose coupling is ensured between the services.

BPMN-led Event Orchestration

As seen above, the moment Bob logs into the application, the centralized engine triggers a command to the worker services via the event streams and awaits their response. Such commands are issued by the Java delegate or something like AWS Lambda. 

Through this approach, the enterprise stands to gain centralized governance and observability benefits while not making the ecosystem tightly coupled and difficult to maintain. This is a very good model for large enterprises and is seeing wide adoption.

MuleSoft + Event-Driven Orchestration

Enterprise Integration is a given in any large enterprise, and most enterprises today, including Ridola, leverage API-led integration. However, even in an API-led architectural setup (which is synchronous in nature), there are scenarios where asynchronous communication becomes very important from a business standpoint. This is where event-driven architecture becomes an able foil for the API-led architecture, and they both can complement and co-exist beautifully to ensure that a customer like Bob is not hampered owing to internal architectural limitations.

Some such scenarios where such a marriage of synchronous (API-led) and asynchronous (event-led) are seen as plausible are:

• Asynchronous Backend Updates: MuleSoft follows a three-layered architecture with Experience APIs servicing the customers across multiple channels on the top, Process APIs, which are the pipelines that process the actual task at hand and pass on the outcome to the Experience APIs, and the System APIs at the bottom which are the repository of enterprise data which is tapped on to by the Process APIs to make the solution contextual and tailored.

Sometimes, there may come an avalanche of customer requests and the Process APIs may get overwhelmed by the repeated need to fetch data from the System APIs. Such to and fro can add to the latency in servicing the needs of the Experience APIs.

Event Stream layer between System and Process APIs helps faster processing

It is in such a scenario (as shown above) that an event broker can act as a storehouse of the most requested information (say customer information) and will act as a “one-stop shop” source for this information for the Process APIs and can asynchronously update appropriate systems with the needed information, thus preventing any unnecessary, repeated to and fro call to the CRM system for every Process API request.

MuleSoft possesses connectors to various systems, which can help in capturing the data change and publishing it as simple events to the event broker. The upstream applications can then act as event consumers to subscribe to these events and update the end systems.   

• Delayed processing owing to system overload and acknowledgment of customer requests: Sometimes, when the system layer fails after reaching its peak capacity, the to and fro between Experience APIs, Process APIs, and System APIs would continue repeatedly in a futile manner, further overloading the system. It could also happen that the system is down for maintenance over the weekend, but the requests are being received during that window.

In such a scenario, MuleSoft-driven applications can generate simple events for every request at the Experience or Process layers, and those events can be stored in an event broker until the end system is ready to process the request. The requesting system will acknowledge the customer request, and those requests that can be addressed using the available Process or System APIs will still get processed in a timely manner. for those others for which sufficient information is not available, a notification stating possible delay can be sent to avoid any customer dissonance.

These are emerging as key themes with modern-day customers who want to use the best of API-led and Event-led architectures to ensure seamless customer service, coupled with the avoidable burden on the systems.

Conclusion

Enterprises are wading through the “experience economy” and the only way to win market share is by winning the confidence of customers. This is where BPMN-led Event Orchestration strategically struck a balance between API-led and event-led architectures that can ensure system resilience, process pragmatism, and delightful customer experience, all in a continuum. This is the right time for enterprises to explore use cases contextual to their respective domain and then evaluate how can a combination of the above approaches help them in their business pursuit. All these approaches have their pros and cons and depend on several factors, such as the process value chain maturity of an enterprise, the frequency, and intensity of changes, the scale of these changes, the number of transactions, the breadth of the customer base, so on and so forth.

Making the right decision and choosing the right option for the right use case can be a challenging process that may require careful due diligence and therefore, several enterprises worldwide are partnering with the leading System Integrators in the ecosystem. Thus, if you are thinking about embarking on Event orchestration, you have many ready partners to guide and walk you through the journey. Get started now !!