{{announcement.body}}
{{announcement.title}}

Architecture Decision for Choosing Right Digital Integration Patterns – API vs. Messaging vs. Event

DZone 's Guide to

Architecture Decision for Choosing Right Digital Integration Patterns – API vs. Messaging vs. Event

Determining whether a direct API, messaging, or an event-driven architecture is appropriate for your application depends on your requirements.

· Integration Zone ·
Free Resource

IT environments and application portfolios are growing rapidly, and they are fundamentally hybrid in nature. Due to the growing need to connect heterogeneous applications on hybrid cloud, in various locations and by different service providers, there is a need for using robust and fit-for-purpose integration techniques. Direct APIs (e.g. Rest, gRPC), Messaging, and Events are the most common integration choices for digital applications. While making these important architectural choices, often organizations and practitioners take an extreme stand towards one or the other of these integration patterns. However, it is not the question of whether to use Direct API, Messaging, or Event, but it is more a question of when to use each of these integration Patterns.

Problem Statement

The three integration patterns (Direct API, Messaging, and Events) are important for modern applications. However, it is often a challenge for designers and developers to determine the integration option best suited for different scenarios and the implications of the choice of integration. In the section below, we provide a guiding framework to help make this decision better based on the context of the integration and organizational factors

A Closer Look at API, Messaging, and Event-Based Integrations

Direct API 

Direct Application Programming Interface (API), allows two heterogeneous applications to talk to each other. For example, each time we use an app on our mobile devices, the app is likely making several API calls to various digital services.

Direct APIs can be designed to be Blocking (Synchronous) or Non-Blocking (Asynchronous). Of these, Non-Blocking APIs are preferred to ensure resources are not blocked when the consumer is waiting for a response from provider. Non-blocking APIs also help create independently scalable integration model between API Consumers and API Provider

Figure 1- Non-Blocking API Call

Figure 1- Non-Blocking API Call


Strengths

  • The operational complexity is lower for Direct APIs compared to the other two options
  • For channel applications as well as Business partner application interactions, several integration standards have emerged over the last decade for different industries. These standards can be leveraged while using API based Integration

Weaknesses 

  • While using APIs, it is important to consider scenarios when the API provider
    1.  is not available to serve exactly at that point in time when a request arrives (e.g. there are only limited instance of provider service, and all  are busy at that time)
    2. Provides extremely slow performance or is unable to scale up beyond a limit such scenarios need to be handled by API consumers, to deliver an acceptable response to the user.
  • Resiliency patterns should be adopted at both the consumer and the provider. Patterns such as Circuit Breakers should be used on the consumer side, and patterns like pacing, throttling should be evaluated to address overloading on the provider side.

One can argue that Blocking APIs (synchronous APIs) are not highlighted here. However, any Direct API can be designed to be blocking or non-blocking calls, but that needs to be considered during design time with a strong preference to non-blocking APIs

Messaging

A Message is fundamentally an asynchronous mode of communication between two applications — it is an indirect invocation, such that two applications do not directly connect to each other. Thus, the Messaging technique decouples the consumer and provider, and removes the need of provider being available at exact same point in time as the consumer. It also addresses the scalability limitations of the provider. For example, your legacy system can take only 100 requests per second and it cannot dynamically scale, message-based integration will, at a minimum, ensure it does not crash even when it gets 200 requests (significantly higher) per second.

Figure 2 – Messaging based integration

Figure 2 – Messaging based integration


Strengths

  • It decouples consumer and provider, removes the need of provider being available to act at exact same point in time, when the consumer is online
  • Can be used to throttle or pace the requests – Consider scenario when consumer service is not able to respond as fast as request are produced.
  • Provides a graceful mechanism to address provider availability and scalability limitations.
  • Most message bus provides mechanisms for assured delivery

Caution

  • Operational complexity is relatively higher than API as Messaging infrastructure needs to be managed and monitored. Most Messaging backbones are designed to handle very high volumes and keep latency to microseconds. However, if not configured correctly depending on the expected load, this can add significant latency.

Event

Event depicts the occurrence of business activity. Any number of interested and authorized parties (services) can subscribe to the event and take the appropriate action in response to the event. Events can occur as a continuous stream when business events continue to happen, like currency exchange rate change as an event. By taking advantage of this continuous stream, applications can react immediately and also reason about the future based on what happened in the past.

Figure 3 - Event-driven integration

Figure 3 - Event-driven integration


For example, most financial organizations have been leveraging events based integration, in the domain of high-value transactions so they can be used by multiple systems for analysis of fraud, anti money laundering, and compliance analysis.

Strengths

  • A high degree of loose coupling between producer and consumer
  • Strong enabler for parallel processing and distributed transactions – You should also explore usage of SAGA patterns for Distributed transaction management
  • Easy-to-throttle or scale-out, keeping high performance, as well as managing optimum cost
  • One-to-many interactions when multiple consumers are needed to process information from one provider
  • Consumers of an event can continue to grow without any change on producer side or to event
  • If properly designed Event analytics can be used to track and manage important Business KPIs

Weaknesses

  • For distributed transactions that are managed with event-driven architectures, transaction integrity needs to be built, monitored and managed explicitly. This requires applying the concept of a global transaction identifier and correlation ID, idempotency, compensation, and addressing all possible error scenarios, for end-to-end flow of events in each transaction.
  • Operational complexity to ensure Event Infrastructure is optimized for high performance is relatively high. This requires a comprehensive understanding of expected transactions, volumes and optimization parameters for event infrastructure, including careful planning and monitoring of the number of partitions for each topic, number of nodes, number of clusters, etc.

In addition to these integration modes, we must acknowledge that whole Industry is trying to move away from monolith applications into modular applications (if not microservices architecture) and distributed transaction management. With emergence of distributed transactions, need for event driven architecture and patterns for managing distributed transactions is gaining a lot of attention.

Event-driven solutions can also exploit serverless computing for zero operations, for details refer to our publication on "Emerging Patterns for Serverless Computing."

API vs. Messaging vs. Events – Comprehensive Decision Guide

Criteria

Direct API

1:1 Messaging 

Event based Mesaging 

Participating Applications

One to One

One to One

One to Many

Consuming Organization - Internal or External

Preferred for interactive or partner applications and to expose public facing interfaces.
Generally, industry-standard alignment of APIs is better, which is important for external integrations.
 Even for simple internal integrations API may be preferred — considering other factors in this table

Good for internal applications/services that will benefit from decoupled interactions

It is good for internal communications that will benefit from decoupled interactions, such as services defined on event-driven architectures.
 It is also good for certain external streams processing, if there is a need of high-frequency interface for continuous feed of data — such as IoT, Webclicks or key business events

Push versus Pull scenario (Just in time or ahead of time)

Efficient for “Pull” of the information when needed (on-demand)

Efficient for Push, publishing information as soon as it is available; usually to one consumer

Efficient for Push, publishing information as soon as it is available; usually too many consumers

Availability

Requires API provider to be available exactly at the time request is made. 

Does not require the provider to be available exactly at the time request is made. Even in real-time transactions momentary downtime can be tolerated without a big impact. 

Does not require the provider to be available exactly at the time request is made. Even in real-time transactions momentary downtime can be tolerated without big impact. 

Resiliency

Tight coupling requires the provider to be available exactly at the time the consumer invokes API. Need to employ resiliency patterns like circuit breaker, retries, timeouts to handle provider issues gracefully

Does not require provider to be available at exact same time - Momentary availability issues will not have severe impact (although overall response time will be impacted for an online request).
 Messaging can also be used to provide Load Levelling for APIs - Use a queue that acts as a buffer between a task and a service that it invokes in order to smooth intermittent heavy loads.

Does not require provider to be available at exact same time - Momentary availability issues will not have severe impact (although overall response time will be impacted for an online request)

Health Endpoint Monitoring - Strong platform support available that can be extended for functional checks

Health Endpoint Monitoring - Strong platform support available that can be extended for functional checks

Health Endpoint Monitoring - Strong platform support available that can be extended for functional checks

Performance

For a strictly sequential flow, API will provide relatively better performance (but the overhead of messaging or event should be in microseconds if configured rightly)

Although for sequential flow, it will add overhead generally in the microseconds range.     

Although for sequential flow, it will add overhead generally in the microseconds range. But for parallel processing it can improve performance - one published event can be picked up by many services instantly to perform different actions (compared to API where consumer will need to invoke each service and wait for acknowledgement)

Sequential versus Parallel processing (e.g. within a transaction processing)

Using Non-Blocking APIs parallel processing can be achieved 

Enables Parallel processing

Enables Parallel processing

Transaction Scenarios - ACID

API is best Suited to support ACID transaction scenarios and provide continuous consistency

It does not support immediate consistency transaction scenarios

It does not support immediate consistency transaction scenario

Transaction Scenarios - Distributed Transactions

Simple Transactions not involving many interactions between different applications
 API is not well-positioned to cater to requirements for complex distributed transactions, such as SAGA pattern

Messaging provides eventual consistency. Messaging is suited for Simple and Medium complexity transactions

This is well suited for eventual Consistency. This makes events best suited for Distributed transaction management and implementation of distributed transactions such as SAGA Pattern

Operational Complexity

Relatively easy. Dynamic scaling that can be configured and optimized for performance as the volume grows. 

Medium - Scalability need to be defined upfront considering expected loads for next couple of years.

Relatively higher. Scalability need to be defined upfront considering expected loads for next couple of years. Managing and optimizing event infrastructure for high performance requires deep understanding of Platform. 

Governance of Interface lifecycle

API Lifecycle needs to be managed; many products provide strong support for end to end API Lifecycle management.  

Message interface needs to be evolved on same lines however relatively limited support is available for its lifecycle management in most tools

Message interface needs to be evolved on same lines however relatively limited support is available for its lifecycle management in available tools

Evolution of Interface definition

Interface definition can evolve over time. Additionally, Graph QL for API provides great flexibility for consumers to ask exactly for what it needs an nothing more. This concept can also be extended to achieve version-less API

 Schema Evolution is feasible, for example with Avro, providing ability to evolve schema over time. However, every consumer gets complete Message details. 

 Schema Evolution is feasible, for example with Avro, providing ability to evolve schema over time. However, every consumer gets complete event details 

Onboarding of new Consumers

Consumers onboarding need to be governed, and it will create additional load for API Provider that need to be Managed (hopefully it is configured for on demand scaling)

Consumers onboarding need to be governed, and it will create additional load for Interface Provider that need to be Managed (hopefully it is configured for on demand scaling)

Consumers onboarding need to be governed. However, it will not create any additional load for Event Provider.

Key Takeaways

  • A Direct API is suitable for channel applications, external partners, compliance to Industry Integration standards (e.g. Open Banking) for partner integrations. It is also appropriate for simple transactions keeping operational complexity to a minimum. As a caution, to address availability issues, resiliency patterns such as circuit breakers and Bulkhead should be employed
  • Messaging is suitable for integrating with applications that need to be decoupled, to ensure guaranteed delivery (even when the other application is not available, a message will be delivered when it is available)
  • Loosely coupled nature of event-based design, with the ability to have events stored for on-demand consumptions by consumers, offers high resiliency as compared to tightly-coupled integration patterns.
  • Events play a major role in managing distributed transactions with high resiliency, performance, and availability. This is a major advantage in microservices-based advanced architectures, particularly modernizing legacy monoliths.
  • Although APIs would be high-performing compared to an exact scenario with event-based integration, for distributed transactions where end-to-end transaction is broken into smaller transactions, Event Orchestration can ensure parallel processing addressing performance requirement (consider this as one complex operation versus parallel processing of many simpler operations). For event-driven architectures and distributed transactions make sure you understand major design patterns.

Common Mistakes

  • Do not assume that APIs are always blocking calls for consumers, they can be designed to be non-blocking
  • Parallel processing for the consumer can only be achieved with Messaging/Events. With non-blocking APIs this can be achieved with APIs as well
  • One integration mode is superior to others. Choose the right integration mode based on your requirements and criteria listed above. Many organizations define standards in extremes and by default bias towards one specific pattern without a decision tree.
  • The interface definition is only required for API-based integration. Interface definition requires proper thinking irrespective of any integration mode, for continuous evolution, such that most changes are backward compatible, minimizing impact to consumers.
  • Not exploiting some of the proven design patterns to address common issues and trying to reinvent the wheel. For example, patterns for distributed transaction patterns like SAGA, CQRS, etc.
  • Ensuring sequence/ordering of events requires proper understanding of Event engines, do not assume events will always be sorted by default. Using the sequence guarantee from an Event Bus has implications on performance that must be evaluated.
Topics:
api patterns ,event driven architecture ,integration architecture

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}