Webinar Recap: Put Data at the Center of Your Enterprise Integration Strategy

Earlier this month, our Head of Product Marketing, Ross Garrett, led a webinar, Put Data at the Center of Your Enterprise Integration Strategy. He discusses some of the challenges with the current state of enterprise integration, creating a scale-free network, the enterprise technology landscape and convergence, and shifting to a data-hub approach.

Let's dive into each of these themes starting from the top:

Fundamental Flaw With Integration: Scale

We all know that the number of APIs has grown exponentially over the past decade. There are hundreds of thousands of public APIs, millions of private APIs, and at least tens of millions of microservices in production today. So why is integration broken? It is impossible for developers to understand, learn and develop against or integrate this number of API services.

Creating a Scale-Free Network

Let's rewind to 1994 when the World Wide Web was being developed by Jerry Lang at Stanford. This was effectively a curated, aggregated, hierarchical index of the web at the time. This was really the very beginnings of the web - HTTP was only just invented and the URI spec didn't really exist yet. This curated approach to the web seemed like a great idea at the time, but the web was starting to explode. There was an enormous number of new pages which made it very difficult to maintain this curated index of the web and in turn made it even more difficult to find anything on the web. Much like the how the number of web pages was exploding then, so is the number of APIs growing exponentially today.

Shifting gears to Larry Page and Sergey Brin who founded BackRub, which of course later became Google. They ultimately took a very different approach to the world of the web, a scale-free approach. The definition of a scale-free network is that a degree distribution follows a Power Law. It simply means that connections are not equally distributed and thus across this network, not all connections are created equal. This is how networks work. If we looked at the web back in '94 and '98, or the world of APIs as we know them today, they've reached this scale-free network.

This concept of the Power Law is sort of reverse hockey stick curve: a small number of nodes have many connections, and many nodes have few connections. If we look at the world of public APIs again, there's been this exponential growth. There's still a small number of providers like Salesforce or Amazon, and various others that have a huge amount of traffic, but there's a massive number of new APIs. It's growing exponentially every day as smaller, long-tail platforms start to expose APIs.

In a network, in the world of the web, there is this unequal distribution, but you have to serve the power of the entire network. In order to support web search in this network effect, we need to be able to reach all links rather than just those that have the most connections. Ultimately, that's how we need to think about integration. We want to be able to serve all of the available products and integrate all of the available products, not just those that have the most links. This manifests itself in the world of enterprise integration.

Enterprise Integration Today

The average enterprise has over 1000 internal cloud and on-prem applications - creating the challenge of combining them all together to sync and effectively share data. Typically we think about integrating the just big players (e.g Salesforce, Netsuite) but what about smaller apps that still provide value? There needs to be a more holistic integration approach. Within a network, it’s the data within the apps that holds the true value rather than the apps themselves.

If we are only able to link the largest apps together, then we're only being able to realize value from those large applications. Current enterprise integration models don’t support integration across all apps and, developers can’t possibly have an intimate understanding of all APIs (back to the exponential growth point). In this world today, most application integration is focused on interfaces and apps. This is a point-to-point model where I integrate application A, Salesforce, through application B, NetSuite. If you do that, you really can only scale to the biggest nodes in your network, the biggest applications, because you have to have an intimate understanding of each API and each interface for those apps.

Looking at the integration technology landscape, ESBs are by far the most popular way for most organizations to integrate apps together. ESBs focus on the application rather than the data. What if the most important thing in your integration changes your data model? That's where we need to change our thinking and focus instead on the data itself. If one API changes, it can cause this massive butterfly effect across your integration portfolio.

If we look at the traditional world of ESBs and iPaaS providers, that have just extended that pattern into the cloud, you're forced to create point-to-point integrations. It quickly becomes an unmanageable hairball of integration, where you're trying to connect explicitly one app to the next. It means that integration is brittle in the first place. If one small thing changes, such as a data object inside NetSuite, it could impact all of the integrations that stem from the NetSuite product. You have to make multiple changes just because a single piece of data has been modified. This brittle nature of integration is really because we focused on the wrong thing. We focused on solving for some point-to-point, tactical problem that exists between connecting two apps together rather than understanding fundamentally where value exists.

Shifting to a Data Hub

The world of application integration has a good understanding of the domain of each app, but not the domain of the data itself. We have to ask ourselves, are there clear owners? Are you routinely monitoring the quality of shared data? Are you able to handle exponential growth as more and more applications and devices connect to that data?

If we focused on the data instead, we move away from a point-to-point model which enables us to deliver a hub-like integration pattern. In this case, the piece of data that you're interested in acts as the center of that hub, and it can connect out to many applications. That offers far greater scale and provides the opportunity to reach smaller apps that hold valuable data. We're not trying to curate the world of services by focusing on the biggest applications, but rather bring together the power of the entire network and allow us to understand where the value exists, the data value itself, rather than the applications individually.

Start with what your organization cares about - invoices or products or customers? Then you'll have a better sense of where the value exists and how you should interact with those valuable pieces of data.

Convergence of Integration Domains

There are three distinct disciplines that have all existed separately: data integration, process integration, and application integration. They need to come together and integration at the hub is the solution. Organizations are spending massive amounts of money trying to find value that exists within their data warehouses, and they need to apply that same thinking across application integration. Looking again to current technologies like an iPaaS, or point to point integration, they don't really allow you to get an understanding of the data inside those apps or translate that into governing the data as it moves between applications or is constructed as a combination of applications. The action of data integration is really a way to combine data that exists in different sources. We need to look at process integration as part of our overall integration platform to understand how data is flowing through workflows.

We need to deliver this low-code/no-code environment for less technical users and business owners to get value while providing a way to govern and control that data. There needs to be a combination of more complex process integration tools and this low-code environment.

And then we come to the well-known environment, the world of application integration. There are more than a 1000 apps in a typical organization, but have three or four very large systems. How do you connect those with the hundreds of other things that surround them? We need to bring together the world of data integration, process integration, and application integration. If they remain as individual disciplines, then the cracks, or the scale apocalypse will get worse, because we're not able to connect the value that exists in the smallest applications in the network.

Data-Centric Integration

You should be able to make business decisions based on the multi-dimensional view across data objects. Data integration combined with application integration needs to be a strategic function so that you can start to interact with the data, the true value, that exists inside your organization as a unified, strategic function.

A data-hub approach connects all of the nodes empowering you to reach all of the value inside your organization. This isn't the old world of master data management, where a single IT department makes every decision for all business owners, but rather it's a structure. It's trying to understand collectively what data is actually important to the business. How do you go about finding that? It's about looking into the apps and services that your business uses and how your business actually delivers value to your customers. What are the things inside your organization that makes it great?

For example, if invoice data is key, create a data model/data hub that represents that invoice and understand how it maps out to the one of many applications. This would be invoice data model then sits at the center of multiple apps. Rather than trying to individually map each individual application on a point-to-point basis, you have a centralized way to interact with the data that's extracted or sent to each of those apps. That's far more scalable and more effective.

In the Real World

Ultimately integration is about data, not interfaces. A common example is that sales needs to understand customers, but the support organization needs to understand customers. Inside the sales team, they use Salesforce, and that has its own representation of a customer. The support team uses Jira, and it has its representation of a customer. How do we bring those two views together? How do we get a common way of interacting with the customer as a data object across these two domains? That's what a data hub is all about. You need to get a clear understanding of the data we care about and use that as the central point.

Pulling It All Together

As the number of applications increases, we can't possibly connect all of them individually. Consider how data and the value of the data inside those apps interacts, and how to leverage data at the center of your integration strategy rather than simply ignoring it.

Integration technology decisions should be based on that understanding rather than perpetuating this point-to-point model that was designed for the early '90s, not a world of where applications have reached the thousands. Make integration easier for developers. Data hubs will remove the need for developers to know anything about data structures inside each and every application.

If you start with defining the data you care about and interacting just with that, then you don't need to know that customer name and customer ID are all actually the same thing. You don't need to have any understanding of individual application interfaces. You simply need to understand the app, or the data that resides within each of those applications. That's a far more scalable world as the number of apps increases.

To learn more about how we support enterprise integration, check out our guide, IT Integration and Transformation. If you'd like to discuss how you can level up your enterprise integration strategy, contact one of our API experts below.