Integration Architecture: How We Got Here

[This article appeared in the 2014 Guide to Enterprise Integration, and was written by Asankha Perera and Mitch Pronschinske. Asankha is the founder and CTO of AdroitLogic—the creators of the free and open source Enterprise Service Bus UltraESB. He is a member of the Apache Software Foundation and the VP of the Apache HttpComponents project.]

Name some of the most interesting software systems you know and I’ll bet that they’ve had to deal with integration.

Particularly in enterprises (even the small ones), multiple applications, systems, and components need to work together and give employees, partners, and customers the performance and feel of a single application. However, many architects design integrations the way they would design a single application, unaware that the nature of integration significantly alters the way applications must be developed. Let’s briefly look at when enterprise integration first became a topic of intense focus in the modern IT industry.

EAI: The Origins

Back when enterprises were built on large mainframes and minicomputer-based application systems, Enterprise Application Integration (EAI) evolved as a mechanism to allow the exchange of information between systems without having each application create information silos within organizations. Methods for integration included file transfers, shared databases, and application functionality access via remote procedures. These methods are still used today wherever they are appropriate. A good example is the Extract Transform and Load (ETL) method, which is a file transfer integration strategy often favored for large, batch data analysis.

Many systems from the mainframe era were batch-oriented and generally based on files and databases, which lead to EAI technologies evolving around the ETL method. Although ETL mechanisms allowed the integration of applications implemented in different technologies or residing across different systems, it did not provide a solution for real-time integration needs. Shared databases and remote procedure invocations also didn’t perform well for real-time integration because of one persistent obstacle: the unpredictable interruptions of computer networks.

MOM: A Paradigm Shift in EI

In the 1990s, Point-to-Point (P2P) integration using Messaging-Oriented Middleware (MOM) emerged as one of the most versatile technologies to support both real-time and non real-time needs. Messaging systems use channels (queues) that other applications can read and recognize. Each application using the queue has to have a sender/producer program or a receiver/consumer program. These endpoints contain some form of message storage so that, even if the network connection is down, the program will know that a message hasn’t gone through and will queue it. This type of communication is more efficient because it can be asynchronous, meaning that the consumer can handle messages received in any order, and it can wait to handle messages until the appropriate time. The communication is also non-blocking, so if one message is taking a long time, the channel and the consumer can still handle other messages while it’s waiting. Message queues can even support systems with one producer and many consumers, such as the publish-subscribe paradigm. These messaging paradigms defined the message bus topology, which became widely used with the advent of Java Messaging Service (JMS) APIs.

Another popular topology for message queues is the broker model (hub and spoke topology), which uses a central hub that all communications pass through for inspection, management, translation, and routing. This allows for better governance of systems, but the downside is that forcing each message to pass through the hub introduces latency, and can be a possible single point of failure.

Message queues are widely used today. Hardware and software design enhancements have significantly improved the performance of these systems over the years. The market has also made significant advancements toward open protocols for messaging, such as the Advanced Message Queuing Protocol (AMQP), which started the trend in 2004. These developments have brought down the cost of MOM and pushed distributed systems into wider use.

The Enterprise Service Bus

Inside enterprises, where the central governance of a broker topology is traditionally more appealing, the Enterprise Service Bus (ESB) emerged as a technology that took advantage of the strengths of both broker and message bus topologies. It’s not a perfect analogy, but you could say that message queues are like the popular code editors (e.g. SublimeText), while ESBs are like IDEs (e.g. Eclipse). Queues can do a lot and still be lightweight, but ESBs try to cover more scenarios and provide more tools out of box.

ESB solutions also emerged at a time when web services, especially the SOAP protocol, were becoming widespread. ESBs quickly moved to offer built-in support for web services along with other protocols and message formats. Although some of the early ESB solutions were implemented on top of older MOM implementations, or required Java EE containers for the run-time, the space quickly evolved to support higher performance and lightweight deployment patterns. Today, there are a few vendors with more than one ESB offering, depending on the needs of customers.

A fully-formed ESB product or open source distribution isn’t the only option either. Minimal frameworks that harness all of the well-known enterprise integration patterns (e.g. Apache Camel, Spring Integration) can be paired with a message queue and other open source frameworks to provide smart routing, transformation, orchestration, and other features of ESBs.

Service-Oriented Architecture

Loose-coupling and modular applications were nothing new in the 2000s, but a stronger emphasis on these strategies started when messaging integration paradigms made it easier to build modular pieces of software that worked together to form complete systems and applications. This is when the idea of Service-Oriented Architecture (SOA) came into mainstream use. The goal of SOA is to make software architecture more flexible to change. It is an architecture style that uses discrete software services, each having one clearly defined business task, with well-defined, loosely-coupled interfaces that are orchestrated to work as a complete system by sharing functionality.

Developers generally try to build loosely-coupled software components in their applications, so the basic concept of SOA is intuitive to many developers. In the late 2000’s, however, the perception of SOA became tied to the ESB architecture, which many argue is not the best way to build SOA. Though I’ve given the basic definition of SOA above, it is trickier to define in practice [1]. ESBs and SOA received some backlash from the development community because enterprise-scale ESBs often seem to have too many unnecessary features or too strict a tie to a vendor’s product suite. Developers using ESBs also tend to use them as a place to hide complexity, instead of dealing with it more effectively.

Microservices

Major web companies like Netflix and SoundCloud have recently put the spotlight back on SOA after sharing their methods for integration and scalability, which center around an architecture composed of microservices. Some developers are saying that microservices are what SOA is supposed to be, but regardless of whether microservices fall under that broad definition, their definition is much clearer.

The community that has formed around microservices favors an integration approach that differs from the classic ESB strategy. Instead of a smart ESB mediator in the middle of the services-based architecture, microservices will connect to each other by having smart endpointson each service and having dumb pipes—which are fast messaging channels with very little logic present in the transfer stage [2]. Microservices, like their name implies, are also strict about the size of each service. Each microservice tends to make up one business feature of the system. A microservice is a unit of software that can be independently removed, plugged in, and upgraded. Unlike software libraries, they are out-of-process components that communicate via web services, remote procedure calls, or other similar methods. This strict componentization makes it easy to have cross-functional developers or teams dedicated to each service, and that’s exactly what Netflix does.

API Architectures

Integrating systems with websites and applications made by other people was another challenge in the history of integration that is handled today through web APIs. The architectural patterns of web APIs have made it easier for developers to connect unfamiliar software to their own projects, whether they’re from another department in a large enterprise, or from an entirely different company. There are certainly strong opinions about the mechanisms being used in these APIs (SOAP vs REST, XML vs JSON, or HATEOAS vs pragmatic REST), but each option has its pros and cons. An organization needs to find the right fit for their needs. The current focus for many toolmakers is building solutions that can manage an API-dense architecture and marketplace.

iPaaS

As cloud services proliferated several years ago, integration with those services and on-premise systems was another major challenge. While there are many possible solutions to this obstacle—many involving a new way of thinking given the distributed, elastic nature of cloud infrastructure—an obvious solution was to build integration utilities into cloud development platforms. The iPaaS (integration Platform-as-a-Service) has emerged as a viable option for providing integration utilities similar to those found in ESBs, but with various cloud integration scenarios in mind.

The EI Outlook

The core messaging paradigm of modern integration is here to stay as we enter a new era of connected devices through the Internet of Things. ESBs are still a great solution for many tech firms given the variety of options, from lightweight to large legacy-integration scale. SOA may fade into obscurity as a term, but its patterns and lessons will live on in the fabric of modern development, especially in emerging microservice architectures. Will microservices become the new standard of scalable architecture? Loose-coupling is already a standard, but microservices will take some time to become mainstream, and if they do, there will always be enterprises that misuse its patterns and give it a bad name. It’s important that we learn not to repeat the mistake of trying to hide complexity in an integration black box, rather than dealing with it head on.

References:

[1]: http://martinfowler.com/bliki/ServiceOrientedAmbiguity.html

[2]: http://martinfowler.com/articles/microservices.html

This article appeared in the 2014 Guide to Enterprise Integration, which you can download for free.

DOWNLOAD YOUR FREE COPY TODAY