Integration 2018 – Inflection Point
Integration 2018 – Inflection Point
Zone Leader Thomas Jardinet reflects on integration trends in 2018, explaining why he's convinced that integration is the cornerstone of digital transformation.
Join the DZone community and get the full member experience.Join For Free
With Google acquiring API Management platform Apigee in 2016, and more recently Mulesoft purchasing Salesforce in 2018 for $ 6.5 billion, the world of integration that was thought to be dormant is in full swing. Integration, which was often perceived as a has been, shows how strategic it is. I remain personally convinced that integration is the cornerstone of digital transformation. In fact, on major transformation programs, integration can represent 50% of the costs. And whether we're talking about API Management, microservices, iPaaS, or even Data Virtualization, all these topics are related and part of a complete hybrid integration stack.
Why Such a Need?
This strong need comes from several strong movements:
The democratization of integration
The global IT architecture, more multi-channel and more data-oriented
The global move to the cloud has brought two major constraints:
The need to exchange cloud and on-premise
The need to have cloud to cloud exchange, or what is called the multi-cloud
Behind these innovations, we can see the issue of security, but the one we don't see perhaps enough is the need for "transparency" in integration. Indeed, without tools simplifying this integration topology, the integration developed will need to be to redone and rethought for each environment migration. This problem is largely answered by iPaaS and to a lesser extent by API Management.
The Democratization of Integration
With the development of collaborative SaaS tools available for free, tools like IFTTT, Zapier, and Microsoft flow have sprung about. Even if these tools perhaps make you smile in light of their "simple" character, we see a big trend emerging to offer integration to everyone. As a result, more and more end users are doing their own integrations.
Saying it another way, integration comes out of its cocoon of requiring specialists and is now supported by many projects. Indeed, the microservices architecture and the API Management solutions have brought with them the possibility for projects to realize their own integration. This is excellent because integration projects have a very different temporality than conventional projects. So it's much easier to delegate integration to project developers rather than having specialists to do that.
Evolution of IT
We have seen the rise of omni-channel architectures and the rise of data-oriented architecture. As much as the first pushes for a deeper, stricter integration with data accessible to all, the second pushes us to rethink tools and architectures. Indeed, the data being exploded (multiplication of types of databases, sharding data pushed by microservices), we are still in the process of maturing architecture at this point. The costly Data Lake projects are starting to end, and there is a need to simplify and make data access easy to accomplish. Behind these needs push solutions like Data Virtualization or Change Data Capture.
These are all new needs that lead us to consider that there is a Hybrid integration. Indeed the integration is thus now hybrid:
By its cloud and on-premise deployment,
From its target audience, from developer to Citizen Integrator,
And by its ability to address a service-oriented IS and a data-oriented IS, or the mix of both.
Description of Logical Bricks
The integration architecture can then be summarized by the following schema:
Yes, I know, it's pretty full. :) Let's start with the simplest and make our way to the most complicated.
We have, of course, the use of drivers for integration with the cloud, AI, SaaS, mobile, and IoT. Now, I'll allow myself a little digression on AI. "Integrating AI" means integrating AI with applications, as well as integrating data and services with AI. Let me explain that last point—AI requires a lot of data and information sources to learn, and must be able to search the data itself. But, at the moment, I doubt that HATEOAS and Swagger are enough on their own to integrate a maximum of data to learn. There will surely be progress at these levels to make life easier for AI.
Concerning people, we have the partners who are the "external" integrators of the organization and the "Citizen Integrators" who are the internal operationals—the integration specialists who do just that, contrary to the Ad Hoc Integrator who are the project developers.
All of this draws three major organizational needs, from "do it together" to "do it by specialist," with all the consequences you can imagine to put this in place.
Following all this, we must obviously choose architectural styles. Microservice architectures & SOA, which I put together because it is a question of service architectures, and SOA learned a lot from microservices.
The Data layer that could also be called "Data Centric IT", and finally the event-driven architecture that comes back on the scene, with some API Manager that facilitate their implementation.
In terms of solutions, I think it is useless to present iPaaS, API, and ESB. I added two personal favorites: Data virtualization and change data capture.
Data virtualization literally virtualizes data from different sources, and offers a unified view exposed in both JDBC and API Rest. The sources can be both an Excel file and a NoSQL database. We can mention Denodo and Tibco among classical publishers, and GraphQL in the open source world, even if GraphQL does not necessarily reach the same level of automation as publisher solutions.
The change data capture, which may make some architects want to pull their hair out (more hair pulling to come! :-) , consists of a bi-directional synchronization of two databases. It's done very well now, so much so that Salesforce Connect could be cited as a great change data capture solution to synchronize Salesforce and Heroku worlds. In the world of Big Data databases, we can mention Syncsort, among others.
On the left side, we have the deployment styles, which I already presented at the beginning of the article.
Of course, we must address the topics of BI and Big Data, but two trends are emerging gradually. I think of the iBPM, which in a way that a BPM to which we added analytical features.
The RPA is also a kind of BPM (attention architects, you can pull your hair out now), which proposes to "robotize" the human gestures on the interfaces. For example, you must constantly copy a form from one application to another application, and the RPA offers to do it for you. To this is coupled a BPM, but also a BRMS engine. This is potentially very useful for small projects, where an SOA redesign of an application would not make sense economically.
As we can see, integration is a subject that is still moving, with many new topics, and is not content to be just one single episode in the saga of ESB. It's a safe bet that the subject will continue to evolve, driven by the switch to multi-cloud, AI, and Big Data.
Opinions expressed by DZone contributors are their own.