Bypass Roadblocks With Cloud Integration
A new article from DZone's 2015 Guide to Enterprise Integration, available now for free!
Join the DZone community and get the full member experience.Join For Free
In today’s digital business, teams want to connect everything: weave multiple apps, APIs, data, and devices into a complete user experience that automates tasks and provides unprecedented access to data and insights. Old-school integration approaches don’t scale to address hyper-connected, multi-system integration scenarios, and reductionist IT approaches to simplify integration—by forcing adoption into a single, comprehensive ERP suite, central enterprise service bus (ESB), or massive data warehouse—fail to fulfill evolving business requirements. When teams rely on a central bus, warehouse, or business platform, they experience an unwieldy, expensive, and monolithic environment, which inhibits business agility and IT delivery.
In contrast, a cloud integration approach rapidly creates connections between users, their devices, their apps, and valuable business APIs. By decentralizing integration actions, providing snap-in integration templates, delivering ready-made app and API bridges, and offering graphical integration workbenches, cloud integration services reduce reliance on SOA specialists and democratize integration development. Teams using an integration Platform as a Service (iPaaS) can easily create sophisticated integration workflows linking several apps, APIs, and systems in a fraction of the traditional work effort and time frame. Integration at the edge and in the cloud (instead of through a central integration server) provides an opportunity to increase social media, mobile, and Internet of Things connections by orders of magnitude without incurring expensive cost, high-risk single points of failure, or operations management conflicts.
Because many connections start and end outside an on-premise environment, cloud integration services are becoming a popular and prevalent integration option. Cloud integration services don’t force data movement on-premise, and are the preferred alternative when moving data or integrating processes across multiple cloud regions. They are the best option when connecting:
Mobile applications or ‘in-the-cloud’ applications to cloud APIs
Software as a Service to ‘in-the-cloud’ applications or cloud APIs
IoT devices to cloud analytics
In-the-cloud applications to cloud APIs
Cloud-to-Cloud Integration Fast Lane
Cloud-to-cloud integration solutions create a fast lane towards project delivery and business value. Cloud integration solutions overcome on-premise integration barriers. For example, on-premise integration solutions require evaluating, procuring, and deploying complicated and expensive infrastructure (e.g., Enterprise Service Bus, gateways, API Management Platforms). Many teams don’t properly size on-premise solutions or know how to monitor and manage the environment, which results in low reliability, poor availability, and reduced scalability. Additionally, on-premise solutions force integration traffic through specific geographical locations (the on-premise data center) and require extra work to lock down network security policies and zones.
In contrast, cloud-to-cloud integration solutions provide instant on-boarding, self-service configurations, high Quality of Service (e.g., reliability, availability, scalability), and allow teams to cost effectively grow their business while defocusing attention on lower level integration infrastructures. Instead of acquiring hardware, building virtual images, and implementing operations management processes, integration teams simply select the appropriate service level agreement (SLA) and start building connections. Cloud-to-cloud integration platforms are fully-managed by the solution provider. The provider offloads tedious work; applying software updates, tuning performance, managing capacity, and monitoring security.
Every architecture and technology choice is a tradeoff. Cloud-to-cloud integration solutions must address regulatory compliance, prescriptive technology environments, and data gravity. Regulatory compliance mandates may require keeping the integration processing within specific sovereign geographical boundaries (e.g. the EU) or execute within certified environments (e.g. PCI). Prescriptive technology services may not allow general-purpose code execution (e.g. Java, C#) or allow connections over legacy protocols (e.g. JMS, MQ). Data gravity—the concept that moving large amounts of data to augment an integration message can be prohibitively expensive—may require cloud-to-cloud integration projects to rely on integration platforms that can flexibly deploy integration agents close to the data and enable smart routing.
Cloud Integration Best Practices
Today’s projects span a distributed persistence and process architecture. Cloud integration best practices connect distributed data stores, apps, microservices, and APIs using a loosely-coupled approach. The approach ideally will rely on integration triggers, webhooks, publish-subscribe channels, and event-based sourcing.
IFTTT pioneered straight-forward integration triggers. When a message comes in, it processes the message and conditionally triggers a distributed action. For example, when I update a document in Google Docs, IFTTT triggers notification alerts to document reviewers.
Webhooks are the equivalent of object callbacks. By registering a webhook URL with a data source or integration hub, the integration solution can notify participants when key integration events occur. For example, when a report job has completed, or when the order status changes, the cloud integration service will push the data in real-time or trigger an action.
Publish-subscribe channels compliment webhooks by registering information queues that endpoints can access to pull data, messages, or events. A cloud-based publish-subscribe endpoint offers an always-available channel that can be accessed by all Internet connected apps, APIs, and devices.
Event-based sourcing keeps distributed data stores synchronized. By sending change events across participants (instead of complete data records), each participant can completely rebuild the data store by re-running the events, rewinding the data to a prior temporal state, or replaying events to build a prior temporal view.
Cloud-to-Cloud Provider Evaluation Matrix
A new breed of cloud integration startups are delivering innovative cloud services that reduce integration project work effort and shorten delivery time frames. When selecting a cloud integration service, evaluate whether the vendor excels in the following areas:
Intuitive Developer Experience
Pre-Built Integration Blocks
Intuitive Developer Experience
Democratize integration development, enlist citizen integrators, and reduce reliance on integration specialists by choosing solutions offering an intuitive developer experience. The solution should provide a drag-and-drop user experience, enable project self service, and visually map message formats and flows. Evaluate whether the development workbench intuitively promotes building integration workflows with webhooks, triggers, mediation actions, and pre-built connectors.
Pre-Built Integration Blocks
Building sophisticated integration flows that tie together multiple apps and APIs is always challenging. Reduce time and effort by selecting an environment offering an extensive library of pre-built connectivity to APIs, apps, and business intelligence tools. The environment should also offer higher-level templates, recipes, and flows. The integration library or cloud service API marketplace should offer hundreds of pre-built integration blocks.
Standard data integration-operations to import, export, cleanse, replicate, and synchronize should be readily available. More sophisticated systems offer pre-built mappings between common apps, APIs, and a canonical model. Ontologies and semantics are used to flexibly connect source messages with target message formats.
Moving integration flows through on-premise environments provides high visibility into message movement, integration flow status, and overall performance. Cloud-based integration solutions must provide robust error management and comprehensive performance monitoring, and must correlate interactions with end-to-end transactions.
Teams will want to schedule integration actions, define appropriate Quality of Service (QoS) policies, and report on service level agreement compliance.
Early generation cloud-to-cloud service offerings forced all traffic through a central cloud location. Today’s hybrid deployment models respect data gravity and will flexibly deploy and co-locate integration execution agents close to where the data or application resides. While monitoring and transaction correlation information is posted into the cloud, data is accessed, filtered, reduced, and transformed close to the source, reducing network bandwidth and latency.
New-school integration solutions are available in the cloud, offer an intuitive user experience, promote cloud integration patterns (e.g. webhooks, triggers, API-driven integration), and provide rich connections to applications and cloud APIs. A short list of innovative integration platform alternatives include:
If you are embarking on a new greenfield integration project or building the next-generation integration platform to serve multiple projects, take time to re-think legacy integration approaches and evaluate new cloud-to-cloud integration service offerings.
A cloud integration approach rapidly creates connections between users, their devices, their apps, and valuable business APIs
A cloud integration approach ideally will rely on integration triggers, webhooks, publish-subscribe channels, and event-based sourcing.
When selecting a cloud integration service, evaluate whether the vendor excels in the following areas:
Intuitive Developer Experience
Pre-built Integration Blocks
Chris helps teams select and implement cloud integration solutions that address legacy challenges and create hyper-connectivity across apps, devices, data, and APIs. He works with teams to connect everything; weave multiple apps, APIs, data, and devices into a complete user experience that automates tasks and provides unprecedented access to data and insights.
Opinions expressed by DZone contributors are their own.