Integration Patterns in Microservices World

Integration, Events, and Microservice Patterns

Overview

It is intended for architects and developers to use this information to guide their integration solutions. Once an integration style has been selected, they should review the associated integration style section to understand the typical patterns relevant to that integration style, abide by the constraints and review the implementation considerations.

Understanding “Integration” – Applications, Integration, and Microservices

There is an important distinction around the responsibilities of an integration system and applications. Integration facilitates the transmission of information between systems and may be realized through the use of technologies such as messaging systems, API gateways, and their associated protocols (MQ, AMQP, HTTP, FTP). Over the years, many technology vendors have implemented additional capabilities within their integration products that have enabled developers to add business logic embedded within the integration solution. This has resulted in business logic being spread outside of the application layer and integration an integration system that is often managed and maintained by a single team resulting in a resource bottleneck.

In following a microservice-based architecture that supports a decentralized development model, applications are implemented as independently developed and operated modules that encapsulate business logic. Whilst microservices may connect with other microservices or other systems of record they do so via a communication mechanism that does not hold business logic or manage business state.

Business logic is the programmatic encoding of real-world business rules that govern the access and lifecycle of business domain data. Simply, if an action requires the processing of business rules against business information then it constitutes business logic.

Business logic includes:

• Semantic Transformation – requires an interpretation of business entities and their attributes following business rules
• Process, Orchestration, or Workflow – requires the execution of the steps of a business process, with business context, rules and management of state.
• Data Enrichment – requires business context, state and/or rules.

Business logic should only be implemented within an application, not within an integration layer. The integration layer facilitates the exchange of information between applications and includes the following technical components: API gateways, ESBs, File Transfer Systems and Messaging Systems. Syntactic transformation, the straight through mapping of attributes with the same business meaning, may be implemented within the integration layer.

Synchronous vs Asynchronous Integration

Synchronous communication involves the consumer submitting a request to a service and waiting for the service to respond before the consumer can continue executing its process. The technical implementation of this form of communication has evolved over time with sophisticated frameworks and computing capabilities so that an application does not necessarily need to block in whole. From a user experience perspective, a User Interface (UI) may continue to be responsive to interaction whilst a synchronous request has been submitted.

Whilst a synchronous process may utilise different integration styles a common synchronous communication protocol is HTTP, used with the API style of integration. HTTP by nature is synchronous in that a request is submitted (either requesting information or submitting information) and the submitting application waits for the response (either a simple OK or some data in the payload is returned). Modern frameworks ensure the UI or process can continue to operate in parallel to perform other tasks however some thread of execution is blocked to await the response. Typical request/response time frame is ideally measured in milliseconds to a few seconds with timeout set to 30-60 seconds.

Asynchronous communication involves the consumer submitting a message to a service but not needing to wait for the service to respond. A consequence of the message in the form of a new message may be returned to the consumer at some point in the future or not at all. The consumer often does not know the destination service and simply submits the message to a messaging system which then delivers the message to one or more destination services. Asynchronous communication may be realised through different integration styles but is typically implemented using asynchronous protocols such as AMQP, MQ, MQTT, Web Sockets or proprietary binary protocols over TCP sockets such as with Kafka.

Web browsers largely use the HTTP protocol and web applications will include JavaScript that execute within the browser to make API calls over HTTP. Modern JavaScript frameworks can ensure the UI is not blocked when making these synchronous calls. For cases where an asynchronous interaction is required, polling is used by the JavaScript code to determine the presence of response data from the service however asynchronous integration may also be used such as web sockets and MQTT.

Integration Styles and Patterns

An overview of the identified styles and patterns adopted within the Digital Platform are shown in Figure 1 using the definition of style and pattern as:

Style — High level, abstract, approaches to integration that represent a number of specific patterns

Patterns — Technology agnostic, generic designs that address common business problems/scenarios

The following integration styles will be supported by the Digital Platform going forward:

• API
• Messaging
• File Transfer

For each of these styles there are various patterns and constraints that will be defined. There are situations where some or all participants in an integration do not support directly connecting with a messaging system. These participants may utilise an intermediary service to perform a translation between an API integration to a messaging interface.

Figure 1 Integration Styles and Patterns

Selecting Integration Style

The following table provides a set of scenarios under each integration style to help select a suitable style. Identify a matching scenario to select the matching integration style.

General Integration Principles

Related Enterprise Architecture Principles

Integration Patterns – API

API may be invoked to initiate behaviour and/or exchange information. A consumer may call a target system API directly or through an intermediary.

This style of integration is synchronous in nature and is used where a response is required within a short time frame, typically as a result from a user action or a step in an executing process. API calls may also be used to trigger processes, expose interfaces externally in a secure and restricted manner or act as an intermediary translation style for the messaging integration style.

Patterns

Constraints

Integration Patterns – Messaging

A connection between applications via a common messaging system is used to exchange data or invoke behaviour using messages of a pre-defined format. The messaging style maximises decoupling between systems not only from the interface perspective but also from a time-based perspective. The messaging system includes queueing to allow for a consumer reading a message from the messaging system when it is ready, allowing for the handling of high-volume spikes.

Patterns

Constraints

Integration Patterns – File

Information is exported from a source application to a file system as a file in a format such as CSV and picked up by the target system for processing. The file can be directly imported by the consuming file system or transported to a destination file system for the consuming system to retrieve. This style of integration is used where a bulk amount of information may be transferred as part of a batch process often executing at pre-defined times. File Transfer must not be used where API or messaging styles of integration are available and suitable for participant applications.

Patterns

Constraints

Application Architecture Styles and Patterns

Microservices

In a microservices application architecture style an application is composed of many discrete network-connected components termed microservices. Each component is a small application itself that implements a part of the overall application. They are independently developed, packaged, deployed, scaled and managed. This is in contrast to monolithic applications that are developed as a single application that is packaged and deployed as a single unit.

Application Boundary

In order to provide some level of organisational alignment in terms of ownership and accountability, a collection of microservices may be logically grouped to form an application boundary. The application boundary may align to a domain or subdomain, defined by a bounded context through Domain Driven Design. Within this grouping a single or small selection of microservices may be exposed as service interfaces for consumption outside the domain/subdomain. The service interfaces may be API or messaging based. Some service interfaces may be exposed to external parties via an external facing API gateway or even exposed internally via an internal API gateway as managed APIs. Within an application boundary, microservice to microservice communication may be established through a service mesh under a cloud deployment environment where the service mesh provides capabilities such as service discovery and load balancing.

Ideally microservices should be independent, with their own persisted data where appropriate and not relying on calling downstream services in order to fulfil their responsibilities. However, there are cases where some microservices do integrate with other packaged applications or microservices some of which may be provided by external parties.

Co-existing with packaged applications

Packaged applications are typically implemented as monolithic applications. Even though their internal structure may be divided as modules that communicate through some form of inter-process communication or internal messaging system. They are more or less deployed as a single deployment unit and require a vendor to incorporate changes through a re-deployment of the entire package as a whole or through patches. Microservices and packaged applications may co-exist simple through decisions around the allocation of domain responsibilities and associated service composition.

A packaged solution may be responsible for an entire domain of which it provides one or more service interfaces as part of its bounded context. In these cases, an adaptor is often used as a façade to avoid directly coupling to the packaged application’s specific interface. This provides a level of protection from having to change consumer integrations if the packaged application is eventually changed. A packaged application may also be “hollowed out” where certain modules are disabled and bypassed, or it lacks some capability that may be provided by one or more custom built microservices. The microservices complement the packaged application and collectively they form the logical application represented by the dotted bounded box in the diagram below. The packaged application could expose an interface as of which the microservice consumes. The microservice’s interface would be exposed a business API at the application boundary.

The diagram below illustrates a generalised example of an application architecture that comprises applications integration through the invocation of APIs and/or event based.

                               Figure 2 -  Packaged Application & Microservices Architecture Example

Microservices Patterns

The patterns defined in the table below provide a refence for commonly used arrangements of microservices with their associated consumers/providers. Whilst a persistent storage is included with each pattern and is typical of an ideal microservice it does not mandate the need for one as some microservices may not need to hold state.

There are a number of microservice implementation patterns however the focus of this section is around the integration between the microservices and consumers/provider. The purpose is to clarify the options for how they are arranged and highlight areas where data transformation is required and where it should be implemented.

Other microservice implementation patterns include:

• Service Discovery
• External Configuration
• Database per Service
• Distributed Tracing
• Health Check
• Circuit Breaker
• Blue-Green Deployment

Microservice Pattern – API consumer with persistence

• Consumer initiates communication through an API call directly with the microservice.
• Microservice interface exposes defined common information model, that is, it is not specific to any packaged solution information model. A custom built microservice API information model may be driven by the needs of the consumer.
• The consumer is typically custom developed as it does not require transformation to map to its own information model.
• Consumer may submit data or request data. In some cases, submitting data will return a result with data.
• This is an API style of integration following a request/response or request/acknowledge pattern.
• The microservice preforms a function following business rules.
• The microservice state is persisted in a storage component such as a database (e.g. SQL or NoSQL).
• The microservice is independent. Everything it requires to operate is contained within its runtime and linked persistent store.

Event processing with persistence

• Microservice subscribers to events received via a messaging system (e.g. IBM MQ, Rabbit MQ, Kafka)
• Microservice performs a function following business rules.
• As a result of executing its function, the microservice may publish a message via a messaging system.
• The microservice state is persisted in a storage component such as a database (e.g. SQL or NoSQL).
• The microservice is independent. Everything it requires to operate is contained within its runtime and linked persistent store.
• No data transformation is required.

Downstream Provider

• Microservice is dependent on a downstream provider.
• Microservice calls provider API directly and performs any necessary mapping to its own information model internally.
• Provider is a strategic component and is unlikely to be replaced, thus not requiring a facade or abstraction layer.

Downstream Provider Adaptor

• Microservice is dependent on a downstream provider.
• Provider may be replaced in the future. In order to protect impact of replacing provider, an abstraction component is implemented as a provider adaptor. This allows for the majority of code within the microservice to remain unchanged.
• The adaptor provides transformation from provider specific interface and information model to the generalised common information model supported within the microservice.

Consumer Adaptor

• Consumer requires transformation of Microservice API data model to its own information model with semantic and/or syntactic transformation and some level of business logic/rules.
• Where it is not viable to implement transformation within consumer application run-time a consumer adaptor may be developed. The adaptor would be developed by the consumer application team and be deployed alongside the application either within the runtime or outside.
• If it is not viable for the consumer application team to implement the consumer adaptor it can be developed by another team. The consumer adaptor is an independent component that logically remains part of the consumer application boundary but would be deployed independently of the consumer application.

Event-Driven Architecture and Event Based Patterns

In an event-driven architecture, applications interact by producing and consuming events. An event is an immutable notification of the change of state of a business entity or user action within an application. One or more applications may subscribe to events of interest. Events are typically communicated via a messaging system. A number of application architecture patterns may be built upon events including:

• Events – basic form of communicating events between applications via a messaging system and typically does not persist events in the long term.
• Event Streams – a continuous flow of events that may be used to derive valuable business insights utilising event processing analysing, combining and enriching event data from which a subsequent stream may be derived, or an action may be performed.
• Clickstreams – events representing user interactions with a user interface including clicks, mouse-movements and gestures that may be analysed to gain a deeper understanding of user behaviour. Analysis may be offline or in real-time that results in actions.
• CQRS – Command Query Responsibility Segregation establishes different models for reading and writing information, gaining the opportunity to independently optimise each path. A combination of API style integration (for reads) and Messaging style (for writes, in the form of commands) are typically used with this approach.
• Event Sourcing – each change of state of a business entity is stored as an immutable event from which an aggregate may be formed by reading back the events in a sequence to derive the final representation.

Events

An event may be transferred between applications using an asynchronous, messaging style of integration following a publish-subscribe pattern. The level of event detail may vary within the message payload. The publishing application is not aware of the consuming applications and the consuming applications are not aware of the publisher. This provides the basis of decoupling between the applications. The messaging system does not provide a long-term persistence of the messages but may provide an optional persistence to facilitate delivery assurance in case of a breakdown in communication. Once a message is consumed by the subscribers the message is typically removed from the messaging system. It is the responsibility of the subscribers to persist and/or perform the necessary action upon receiving the event.

The diagram below illustrates an application publishing an event (or emitting an event) to a topic through a messaging system. One or more applications may subscribe to the topic and receive the event.

Figure 3 -  Application Architecture - Events Pattern

Command Query Responsibility Segregation (CQRS)

In separating the read and write models CQRS requires a mechanism to provide updates between them. This can be achieved through the use of an asynchronous messaging style of integration following a point-to-point pattern or publish/subscribe pattern. A command may be submitted via a synchronous API style o integration following a request/response pattern to update the write-optimised data store. Alternatively, an event may also be submitted to update an event store. When the data or event store are updated an event is emitted comprising the change details from which a read optimised data store will receive. The read-optimised store will persist the change in a materialised view appropriate for supporting queries. An application may perform a query against the read optimised store using an API style of integration following a request/response pattern.

Figure 4— Application Architecture - CQRS Pattern

Event Sourcing

An event sourced system persists a sequence of events rather than repeatedly overwriting a single state. The state is kept in-memory and is updated by events generated as a consequence of processing commands. A command may be submitted via a synchronous API style of integration following the request/response pattern where it is checked if it can be applied. If it can an event is created from which the state is changed. The event is persisted in an event journal and published to a topic using an asynchronous message style of integration following a publish/subscribe pattern from which other subscribers may also receive the event. The in-memory state machine may be recreated through replaying of the event history. Event Sourcing is typically complemented by the CQRS pattern to provide materialised views from the update events.

Figure 5 — Application Architecture - Event Sourcing Pattern

Event Driven Architecture – Synchronous and Asynchronous Combination Patterns

UI Interaction – API end to end

• UI Application makes API Backend for Frontend service that in turn makes API call to Microservice.

• User is waiting for response (e.g. submitted scheduled payment – you want to know it was successfully scheduled)

• Microservice either retrieves data from its own data store or calls a downstream provider via API call.

• Suitable for low volume scenarios.

• Direct coupling between components.

UI Interaction – API with message queue buffer

• UI Application makes API Backend for Frontend service that in turn makes API call to Microservice.
• User is waiting for response (e.g. submitted scheduled payment – you want to know it was successfully scheduled)
• Suitable for high volume scenarios where buffering is required.
• Microservice responsible for correlating response messages to request messages.
• Adapter required to pick messages off queue and make API calls to downstream provider.
• Messaging/Adapter/Queue components could be implemented using Integration Bus.

UI Interaction – API Polling

• UI Application makes API Backend for Frontend service that in turn makes API call to Microservice.
• UI Application receives immediate response from API with status “In progress”.
• UI Application does not block, user may continue to utilize other app features.
• Microservice processes request, calls downstream provider via API-to-messaging system adapter system.
• UI Application makes background API requests polling for status to determine if result of previous operation is completed.
• Microservice updates internally stored state to reflect completed operation.

UI Interaction – Async Update

• UI Application makes API Backend for Frontend service that in turn makes API call to Microservice.
• UI Application receives immediate response from API with status “In progress”.
• UI Application does not block, user may continue to utilize other app features.
• UI Application registers with Async Gateway using async technology such as Web Sockets, MQTT, socket.io.
• Microservice processes request, calls downstream provider via API-to-messaging system adapter system.
• UI Application receives response message via subscribed topic.
• Microservice also updates internally stored state to reflect completed operation.
• UI Application in future may retrieve stored state via API call.

UI Interaction – Assumed Success

• UI Application makes API Backend for Frontend service that in turn makes API call to Microservice.
• UI Application receives immediate response from API with status “Change Accepted”.
• UI Application does not block, user may continue to utilize other app features assuming change has been received.
• Microservice processes request, calls downstream provider via API-to-messaging system adapter system.
• Microservice only interested in failure messages otherwise it is assumed action performed successfully.
• Microservice updates state and initiates user communication of failure.
• Adapter publishes message to failure queue that microservice subscribes to.

Event Driven Architecture – Transactions

Transactions executed across microservices as part of a long running process can be implemented using the Saga pattern. The Saga pattern relies on each microservice performing a step in the process, updating its own data from which triggers the subsequent step to be completed by another microservice. There are two types of implementations for the Saga pattern: Choreography and Orchestration.

Transaction Pattern – Choreography

• There is no central coordinator.
• Each service operates independently.
• The sequence is inherent in the predefined events and system responsibilities.
• Changes within one microservice may be marked as temporary until a confirmation event is received later to ensure consistency across states.
• Changing the process requires changes to potentially multiple services.
• Roll back requires separate event.
• Used where number of steps in process is low < 5.
• Risk of cyclic dependency.

Transaction Pattern – Orchestration

• Centralized controller (orchestrator). Could be implemented as:
• custom microservice
• BPM (headless)
• Custom microservice with embedded process engine.
• Packaged application
• The sequence is defined within the orchestrator.
• Orchestrator maintains running state.
• Changing the process requires changes only to orchestrator.
• Roll back managed by Orchestrator.
• Used where number of steps in process is high > 5.