Don't Be Afraid of the Integration Iceberg

The Integration Iceberg. A fitting analogy representing integrations: it's what lurks below the surface that can sink your project and keep you from delivering on time and within budget.

When you are looking to integrate into or sync data between endpoint services like Salesforce, Hubspot, Netsuite, Shopify, MS Dynamics, etc - Googling their API documentation reveals the basics that are needed to get authenticated and push or pull data into each endpoint. That is the part that can be easily seen on the surface — the tip of the iceberg. What lurks below are all the other components that are not outwardly apparent, even to experienced developers.

Thus, at Cloud Elements, we contend, "Connecting is easy, integrating is hard."

To build a robust integration, whether it is to one or to many endpoints, there are components lurking below the surface that you will need to accommodate for. If you don't know what those components are and/or fail to plan for them, you may crash into them head-on. Oh, and it gets worse, each of the components depicted above that live below the iceberg typically vary by endpoint. Thus, even the "known" portions of the integration iceberg that are below the surface can be difficult to solution and overcome.

Authentication

Every endpoint service has authentication allowing your customers to connect and share data between their account at the endpoint and your application or workflow. While there are general authentication categories such as basic, token, Oauth, etc, each typically has nuances that you will need to accommodate for.

For example, Pinterest's API uses OAuth2, but doesn't utilize the typical Bearer Token that is more common with that type of authentication. Your integration team will need to be prepared for this, and create either custom code for each endpoint or some sort of authentication service to plug each new endpoint's authentication process into, normalizing them to fit your specific integration needs. More details in this blog.

Eventing and Polling

If your use case involves being notified when data changes occur in an endpoint service, e.g. a new contact is created in Netsuite, then you will need to equip each endpoint integration with the ability to receive and process events.

A small percentage of endpoints provide Webhooks natively, where your application or service is called when a change occurs within that endpoint. That is the best case scenario. However, for endpoints that do not support Webhooks, your integration team will need to produce a polling framework to "ask" the endpoint on a periodic basis if changes have happened since the last time you polled. This requires that the endpoint provides searching or filtering queries as a part of that poller so that only the changed data elements are returned.

Thus for the majority of the endpoints that you are integrating to, a polling framework is necessary AND you will need to also understand the specifics of how the endpoint allows your polling framework to query for data.

Lastly, once you get that in place, you will need to map the contents of the event data provided from the endpoint into what your application expects. More custom, per endpoint code. More details in this blog.

Error Handling

In general, modern endpoint APIs standardize on a set of error codes. But, that said, what that error code actually means, as well as, the text associated with the error message varies across endpoints. You will need to map the error codes provided by each endpoint to those that your application expects - custom for each endpoint.

To save you time, Cloud Elements has taken a list of error codes from our service providers and normalized them. If a response returns a message that includes a specific detail about the method call, we send our normalized error code along with any additional data included in the body of the message. Below is a sample message from a service provider that would be included with our normalized error code.

We provide the following normalized error codes:

Bulk Framework

Interacting data in bulk or batch mode isn't required for many integrations. However, if your use case is one where your application needs to move large amounts of data as part of "seeding" a new customer's data or periodically exchanging data into or out of the endpoints that you integrate with, you will need to add bulk capabilities.

As with most integration features, some endpoints support moving data in bulk by exposing asynchronous API resources natively. Unfortunately, most do not. Thus, your integration team will need to create a bulk data handling framework for those endpoints that don’t. And, likely, regardless of whether or not the endpoint’s API supports bulk, you will also have to manage chunking up large data files into manageable sizes, per endpoint service, depending on the limitations placed on large file transfers by for each service. What to know more? Good stuff in this blog.

Querying

It is more often than not the case where your integration will need to filter or query for data that exists in a given endpoint. For example, in the REST API world, there is a difference between a POST /contacts (create a new contact) and PATCH /contacts/{id} (update an existing contact). So, for a use case where a contact is added in your application and it is to be synced with your customer's application, you will need to query to see if that contact exists in order to determine whether to use POST or PATCH.

However, while many endpoints have query support, often resembling standard SQL, they all vary in capabilities - some are super powerful, some not so much. Your integration team will need to become familiar with each endpoint's specific query language - more custom code per endpoint integrated to.

Oh, I almost forgot, you will need to use your query framework both in your Bulk data framework and your Eventing framework mentioned earlier. More information regarding querying via APIs can be found in this blog.

Web Service Management

This one can be a real gotcha depending on the endpoint service. As background, there are two major API protocols used in modern API specifications: REST and SOAP.

One can argue which is better, but regardless of your opinion, you will need to code to either protocol or both, depending on the endpoint. In fact, I still see examples of cases were an endpoint is converting their API spec to REST, but not all the functionality is available yet as REST, so you will also need to code to their SOAP API resources. Dual protocols, all custom, a maintenance nightmare.

In some cases, it actually makes more sense to build conversion service framework to convert from REST to SOAP (and back) if your application is based on REST and you are integrating to several SOAP-based endpoint APIs.

Data Discovery

As is typical of the variances found in API specifications, some endpoints provide API resources that allow your application to discover or list what objects and metadata are available at that endpoint. This is particularly important if your integration use case interacts with customer specific custom fields for a given data object.

However, many endpoints don’t provide discovery API resources natively. So, guess what? You’ll have to build a framework that pulls in a sample of data and interpolates the results, looking for the fields that your application requires. More information on discovering data objects and fields in this blog.

Map and Transform

Every endpoint API is an island when it comes to the structure of the data expected. Part of integrating your application or workflow to each endpoint will require your integration team to custom map and transform the data in and out of each endpoint to the data structures used by our application. Yet another framework that you will have to build and maintain. Some cool additional info in this blog.

Workflows and Logic

If your use case is one where you are embedding any workflows or sync logic in your application, then likely you've already accommodated for that in your LOEs, etc. However, if you look at workflows from an API perspective, you may have to write custom code in your integration to overcome, you guessed it, endpoint specific data structure nuances.

For example, with Netsuite and Magento, address information associated with a contact is kept in an array of address, each one bearing a field that tells you the type, i.e. shipTo, soldTo, billTo, etc. You will need to know that gotcha as well and write custom code to accommodate.

Even the mighty Salesforce API has a nuance that requires logic. To retrieve all the fields associated with a contact, you will need to either know exactly all the fields beforehand, including custom fields or first do introspection to get all the field names for a contact, then use those when you make the call to get the contact information. Yet another nuance, even when integrating to Salesforce. Go figure.

Conclusion

Yes! Take much of the above risk out of your integrations and use the Cloud Elements' platform. We've done much of the work already to accommodate all of the above gotchas and normalize to a consistent API across over
You may be wondering, "Can I avoid crashing into the integration iceberg?" 150 endpoints, or what we call Elements. Allowing to you focus more on delivering new features or customer solutions, rather than fussing with the messy, endpoint specific integration nuances and associated maintenance nightmare. Your integration starts simple and stays simple.

If you're looking to dive deeper, explore our Definitive Guide to API Integration to flawlessly set up your next integration use case.