Don't Get Stranded on API Data Islands
Don't Get Stranded on API Data Islands
Every integration endpoint is different in how it expects to receive or return data from calls made via its API.
Join the DZone community and get the full member experience.Join For Free
I've seen many API specifications varying across the board from SOAP to REST with random others in between. While you can typically sum up the components of an API spec into category types; such as web service, authentication, resources, methods, events, search, and so forth, they are never the same in terms of data payload definition. Every endpoint is different in how it expects to receive or return data from calls made via its API. It is its own API island with its own language, so to speak.
So, if you have a SaaS application or you are looking to sync data between endpoints, the traditional way that many iPaaS platforms solve the issue of inconsistent data structures is to provide simple wrappers, or "connectors" — which convert the data into a format that can be read by another connector that is designed to work with it. This, in the world of API integration, is referred to as point-to-point connectivity. This works well and has been the model used to connect endpoint services together for years.
But does it scale? No.
Why? Because point-to-point is great when integrating to a small number of back-office services but falls short when your use case requires integrations to multiple endpoint services or if reusability of a sync application across multiple endpoints is required.
With point-to-point based solutions, each new endpoint needs to be connected directly to every other endpoint in the ecosystem. Each connector is specialized, custom, and programmed to connect to only one other endpoint rather than to map and transform data to a common object format that can be consumed by multiple other connectors.
That said, there are platforms that have the notion of a common object or format converter functions, which are more optimal than point-to-point, for sure. But typically these data mappings are one size fits all and are not generally adjustable or modifiable to your specific business needs. Thus, the point-to-point nature is still in play in this case, just lessened in severity.
Speaking of data mappings, many integration platforms ignore the fundamental value of integration - the data. By ignoring the data, it is impossible to create centralized or canonicalized data models that can be governed by your application's data needs. As a result, you are limited to what a generic connector provides in terms of data interactions and thus, you still have to create endpoint specific data mapping customizations. Again, back to the point-to-point connection issue.
Another issue with point-to-point based connectors is they typically do not expose a callable API resource that allows your SaaS application to directly interact with a common object definition, even if the common object is built into the given connector. In other words, even if the connector has the ability to map and transform data to a common format, it is not set up in a way that allows you to interact directly, via API, with that common data definition rather than the using the native API data definition provided by the endpoint. You are stuck trusting the connector to interact with endpoint service data on your behalf.
So what is the next generation of the above? What you need is a platform that provides, out of the box, the ability to:
- Define what data your application needs to interact with
- Bidirectionally transform that data definition as required to the endpoint's data specifications
- Interact with a callable API resource based on your specific data definition
The Virtual Data Resource (VDR)
Cloud Elements' Virtual Data Resources puts your data models at the center of your application ecosystem and enable you to manage the data you care about in the structure that is best for your application or business. VDRs provide a canonicalized view of your data objects while eliminating the need for point-to-point mapping of data to each and every new application.
I often use this analogy when I demo Cloud Elements' VDR:It is like you called up Salesforce's CEO and said, "Hey Marc Benioff, listen, I need you to change your API spec to match my application's specific data definitions - and I want that in the next 20 mins."
Yeah right. Marc is awesome, but it is not very likely he'd make that happen specifically for your app.
But, guess what? That is what a VDR gets you — a callable, REST resource that is crafted specifically to your application's data needs, across multiple endpoints - regardless of whether you have a SaaS app or are constructing a back office data sync.
If you are curious, even slightly technical, and have 8 minutes to spare, here is a demo of VDRs in action!
How It's Done
You can create your own VDRs that match your specific data needs easily in the Cloud Elements platform, either via our UI or via our REST APIs.
For the UI, you just click on "Build a new VDR":
You can view a catalog of VDR templates in the Cloud Elements UI, clone them, and make them yours! There are VDR templates that provide minimalist fields (e.g. /basic-contacts) for simple use cases, as well as, full-blown, super detailed, every field included (e.g. /contacts) VDR templates. Simply clone, modify, and use!
The Next Level
I'll leave you with this thought, at the end of the day, accessing and sharing data is the fundamental reason for integrating into endpoint services. The Virtual Data Resource provides the next level way to do that. We purposefully designed the VDR to provide a way to interact with the data islands across endpoints as defined by you and your application's data needs - wrapped up in callable REST resource. Your application can now easily interact with endpoint data islands as if each spoke a common API language, that was defined by you!
And you don't even need to ask the CEO of each endpoint company to modify their API specification - no need to trouble Marc Benioff, he is busy enough already.
If you're curious about how you can leverage VDRs to access, share, and sync your data across endpoints, try out our API integration platform for free for 30 days by clicking here. Enjoy, aloha!
Published at DZone with permission of David Honan , DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.