Multi-Cloud Integration

SQS (Simple Queue Service) 

SQS is a simple query service or distributed queuing system. It’s a queue that stores the messages that should be processed later. SQS is a pull-based service that allows the decoupling of cloud components and microservices.

Pros:

• Includes dead-letter queue: Automatically separates unprocessed messages and pushes them into the dead-letter queue
• The choice between a standard queue and FIFO
• Automatic duplication removal: SQS will automatically detect and remove duplicated messages.

Cons:

• Doesn’t support one-to-many message broadcasting

SWF (Simple Workflow Service)

SWF is a task coordination and workflow service. It allows the building of parallels and sequential workflows. It allows for building a reliable distributed system where the tasks are logical units or functions of your application. SWF is suitable for processing human-based operations like order workflow or procedural requests.

Pros: 

• Reliability: Workflows and state tracking are reliable as they run at high-availability data centers
• Flexibility and logical separation: SWF allows architects and users to modify workflow elements easily. Also, SWF separates the control flow of your background job

Cons:

• A lot of bugs in the AWS Management Console and a lack of general support 
• The setup process is difficult and takes a lot of time
• Lack of limit control tracking: It can lead to throttling issues

AWS Step Functions

AWS step functions allow us to create a serverless workflow or state machines; for example, to automate call center tasks. Step Functions is an orchestration function that orchestrates other Lambda functions.

Pros: 

• Decreasing application complexity by decoupling workflow logic from business logic

Cons:

• Step Functions uses a specific state machine to define workflows, helpful only for the Step Functions service. This may lead to vendor lock-in.

Google Cloud (GC)

The GC has an integration ecosystem, an Integration-Platform-as-a-Service (iPaaS). It provides a set of tools and services to manage and connect applications. The Google Cloud iPaaS contains the following services:

Integration Designer

Integration designer is a no-code UI tool that provides the ability to build workflows that consist of triggers and tasks (see below). Tasks and triggers are connected with an edge and fork and join. This service looks pretty similar to SWF.

Triggers are an essential workflow element. Triggers are usually an entry point for the workflow and represent an event that initiates the task. For example, API trigger or pub/sub trigger.

Tasks

Tasks are units of work (or functions) that comes after triggers. One example is Data Mapping, where you can integrate one data model into another.  Some predefined tasks include the Call Integration task, For Each Parallel task, For Each Loop task, Timer task, and many others. 

Alongside many advantages, Google Cloud has some drawbacks:

• Lack of data centers compared to Azure and AWS 
• It is a relatively new framework. Some features may be in preview (or in beta) and contain some bugs.

Azure

Azure offers an Azure integration service set. It contains a variety of services to set up strong integration between applications. Below are set of the most powerful integrations services Azure offers.

API Management

API Management is an API gateway service that integrates several APIs and connects Azure Functions, Logic Apps, and Service Bus. 

Pros:

• API Management provides many features like authentication, authorization, cache responses, and fine-grain quotas and limits.
• Flexible customization through the API policies

Cons:

• The developer portal contains many usability issues and bugs.
• Scaling costs can be too high. API management scales automatically; however, there is no good transparency on scaling costs.

Logic Apps

Logic Apps is a serverless, low-code, workflow integration tool (like GC Integration designer and AWS SWF).

Pros:

• The service can connect SQL Server, Event Grid, Oracle, ServiceNow, and Office 365. We can select more than 200 connectors.
• Ability to host maps and schemas locally

Cons:

• Difficult to manage and maintain complex integration workflows
• Issues with user interface, especially with complex workflows

Service Bus

Service Bus is a service that implements the event bus pattern. It provides an event bus based on the queue and publish/subscribe integration principle when one component produces a message, and others consume it.

Pros:

• The service also supports translations, dead letters, multi-consumer topic-based logic, and other features.
• Powerful message routing and filtering capabilities

Cons:

• Slow performance of the HTTPs connection 
• Option to monitor messages, queues, and topics is quite limited 
• No integrated UI tool to debug messages, queries, and filters

Event Grid

With the Event Grid, you can join cloud resources that produce events (publishers) and resources that handle the events (subscribers). The event grid looks similar to AWS SNS. It is also based on the push approach. Event Grid listens to the events from one component and can trigger other when it reserves messages. For example, one application can send a message to the Azure queue and an email notification. 

Pros:

• Enterprise integration: Supports almost all Azure event sources; e.g., Functions, Event/IoT Hubs, Logic Apps, Service Bus, SignalR, and many more
• Contains options to monitor and debug messages
• Includes automatic dead-letter queue
• Lower cost: A pay-per-event price model results in very low cost.

Cons:

• Transparency on how to store and enable dead-letter can be improved.

Azure Arc

Azure Arc is a service that allows us to connect services from different cloud providers and implement multi-cloud integrations. This service will as a main integration service banking multi-cloud application. 

Pros:

• Supports services like SQL Server and Kubernetes
• Supports Windows and Linux operation systems

Cons:

• Sometimes, the agent connection can be unstable and hard to debug.

Conclusion

In the article, we’ve gone through the top components to build cloud integration. However, integration solutions don’t have to be enterprise-level: they can meddle or even be small. For example, we have application A which requires the exchange of data between application B. We can create integration middleware with Azure Durable Functions or AWS Step Functions. With these services, we can create workflows for data mapping, notification, and even some plain analytics.