The Future of Integration With Microservices

This article is featured in the DZone Guide to Enterprise Integration, 2015 Edition. Get your free copy for more insightful articles, industry statistics, and more.

The world of IT as we know it today is changing drastically. Just a couple of years ago, developers spent months or even years developing infrastructures and working on the integration of various applications. Huge projects with multiple participants were required to implement desired features. With the advent of DevOps, various Platform-as-a-Service (PaaS) environments, containers, and microservices, many complex requirements can be met within a much shorter time. The Internet of Things (IoT) is also expected to change established applications and infrastructures. As a result of these converging trends, the way in which system integration will work is set to undergo a fundamental change in the coming years.

System integration has come a long way, from point-to-point connections between individual systems towards the first integration solutions that helped in standardizing those connections. And in the advent of a much more business-centered design—and the broader shift into more service-oriented organizations—the Enterprise Service Bus (ESB) evolved from a pattern to a variety of products. They all promised to deliver reusability and exchangeability by standing up as a centralized and managed infrastructure component.

As a direct result, applications had to be sliced—and even partly rebuilt—to support exchangeable services. Service-oriented architectures (SOAs) were the new paradigm behind this. Unfortunately, the interface technology of choice tended to be Web services (WS). Web services transport data between systems by encoding the data into XML and transmitting it via the Simple Object Access Protocol (SOAP), and they introduced a significant amount of additional code and descriptors into most projects. With the extra code and configuration came extra complexity on various levels. Government of service versions and cross-service security, as well as documentation and requirement engineering, had to be tweaked to focus on services instead of features. All of this had to be managed.

Outer Architecture Gains More Importance

With the next technology evolution—Microservices—the need to manage even more potentially-polyglot and distributed services became overwhelming.

Looking back, we can see that integration complexity moved from the inner architecture of applications towards the outer architecture of the complete system. The outer architecture refers to the platform capabilities you need to help all those simple little microservices work together.

And it isn’t enough just bringing the technology aspects together. The outer architecture also has to support development teams and the Software Development Lifecycle (SDLC) to deliver on the promises of flexible and scalable development and deployment.

Looking at the architecture diagram above, it becomes very obvious that the most important parts of your application are now outside of your services. Instead of solely focusing on integration flow and logic, the correct approach to run your services becomes even more important than it ever was.

Picking a Baseline - xPaaS

Instead of selecting a single ESB or integration product you now need to find a solid combination of all of the needed parts. While the acronym iPaaS (Integration platform as a service) started to emerge a couple of years ago, you will end up with even more than one xPaaS (everything as a service) service. The best starting point for the outer architecture is an aPaaS (application platform as a service) platform or framework, which has the potential to integrate seamlessly with the relevant parts of the underlying PaaS. There are many options available. Classical enterprises might still stick with a Java EE based aPaaS, while others will quickly move on to something more lightweight. But the platform language is no longer the critical aspect here: more important is how the aPaaS hooks into the centralized cross-cutting and operational capabilities.  

Operational Capabilities

While aPaaS and iPaaS both refer to complete product stacks mostly, they have one thing in common: the base PaaS offering on which they rely on. A modern cloud platform knows a lot more about the applications it runs than the ones years ago. With the help of deployment templates and descriptors, the individual services can easily be deployed, scaled, and exposed as needed. And even the service orchestration is encapsulated. All of this works because other emerging technologies like containers and container orchestration (e.g. Kubernetes) is built into the core of today’s PaaS. What very quickly started to be a commodity under the hood is enhanced by some vendors with operational consoles—and enhanced by a very few vendors with additional developer tooling. Developer support, especially, mostly ends with what DevOps teams and continuous delivery best practices demand.

Developer Support

But there is a lot more effort needed to develop highly-distributed systems. While complex IDE plug-ins from ESB times were able to tweak the centralized service repository, loosely-coupled microservices do need more runtime information to build an application. Instead of centrally wiring services together, we now want to lookup service endpoints at runtime. Registration, versioning, and additional meta-information like SLAs also need to be resolvable by every service from the centralized registry. Dispatching requests to one of the available instances will be done by the underlying PaaS. A developer will have to provide most of the relevant information at time of development, and the services will auto-register themselves while the platform uses their individual information to distribute the workload most efficiently. Tooling to browse service documentation or lookup existing services will also be a very critical feature. When a microservice-based application is finally in production, it is time to have the ability to follow the distributed request-flow throughout the system. Being able to track down errors and assist with debugging is a complex task, which the outer architecture of our integration solutions also needs to support.

The Changing Face of Data Integration

A last—but no less important—part of new integration architectures will be the next level of data-integration approaches. With every microservice being responsible for its own database, the classical ETL (Extract, Transform, Load), data-warehousing, and data federation systems will quickly reach an unmaintainable state. New approaches to master data management will be required to handle these kind of distributed and segregated data volumes. Among the possible solutions on the horizon are Big Data or stream data solutions, which take data-changing events and allow operations on a copy of the data. But virtualized data-models will also help for reporting or warehousing requirements. More complex solutions might also allow the exposure of data services, which themselves can be consumed by business microservices.

Ready for Prime Time?

Vendors have already started to “microservice-wash” their tools and platforms to get your attention. And the segment architecture approach is still too new to give recommendations. For a while, it will still be your responsibility to understand the capabilities you need by doing your own research. Some promising candidates are evolving out of the open-source think tank at this very moment. First and foremost, projects like OpenShift Origin, WildFly Swarm, Fabric8, and APIMan will help you put together most of the puzzle pieces in your microservices-based architecture.

Microservices and containers will change the way we build, maintain, operate, and integrate applications.  When architectured with discipline and a careful selection of the outer architecture, they will help applications become more portable and more adaptive. In the end, better application or service integration will be very different to today’s approaches: it will become a number-one key requirement for distributed microservices applications.

For more insights on microservices, JVM languages, and more integration trends, get your free copy of the DZone Guide to Enterprise Integration, 2015 Edition!