Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Managing an Integration: Developer to Integration Specialist

DZone's Guide to

Managing an Integration: Developer to Integration Specialist

Understanding of the three main concepts of integration abstraction will greatly aid in preparing to scope and starting an integration project.

· Integration Zone ·
Free Resource

The State of API Integration 2018: Get Cloud Elements’ report for the most comprehensive breakdown of the API integration industry’s past, present, and future.

Not every developer is an integration specialist, yet that doesn’t mean that this type of work will only be handled by those qualified to do it. Regardless of a developer’s experience with these complex projects, a methodical approach to project planning will be a guide to make sure the best technology decisions are being made, regardless of past experience.

There are three main concepts to understand at the start of an integration project: abstraction, metadata, and standardization. These concepts will prove an invaluable guide throughout the integration while lowering the total cost of ownership of the project.

Concept 1: Abstraction

At the start of a project, the challenge of bringing together data from two applications with very different structures can be daunting. Figuring out how to meet the current business requirements while building in flexibility to adapt as requirements change is another challenge. The concept of abstraction can help. This allows the project to be broken down into several manageable layers so that it is easier to understand the overall integration requirements before assessing the possible technology solutions. Because the project is now chunked out into smaller pieces, any change requests are much easier to parse, without disrupting the entire integration flow.

The three primary layers of an integration project are physical, logical, and view (see Figure 1 below).

1. Physical Layer

This is the lowest layer of abstraction and it defines how data is actually transferred from one application to another. As applications readily reveal the communication protocols and APIs that they support, this is the most logical place to start. The below questions about the applications to be integrated are a good guide for what considerations need to be made at this stage of the project:

  • Do the actual data transfer mechanisms or payloads differ between APIs?
  • Are the API payloads representing flat, relational, hierarchical, or binary data?
  • Do you need to handle pagination or batch processing, and what do those calls look like?
  • Do you have transactional operations you need to cache the results for?

2. Logical Layer

The next level of abstraction describes what relationship the data in one system has with the data in another system. The field mapping, relationships, transformations, and metadata about a system’s configuration is considered part of the logical layer. To understand the requirements at the logical layer, answer these questions:

  • What API endpoints or tables are you going to be working with? These are your entities.
  • What are the properties, fields, and data types included in each entity?
  • What are the available actions to get or set data on an entity?
  • Do the inputs and outputs match when you are performing operations?
  • Are you asking for data or receiving webhook messages?

3. View Layer

The last level of abstraction is what the purpose of the integration is, such as a description of what function the newly integrated applications will serve to a business user. This information will get exposed to other developers and application users to explain the key characteristics of the integration in order to effectively deploy, run, manage, and adapt the integration over time. For example, at the view layer a description will distinguish two connectors; one that is a Marketo connector that passes data using the SOAP mechanism and one that uses the RESTful mechanism.  To understand a project's requirements at the View layer, consider these questions:

  • Are you integrating a database, an FTP drive, or a web service?
  • For each instantiated integration, how will you handle authorization?
  • Do you need to call an Authentication endpoint, provide user credentials for Basic Auth, or do you need to programmatically work through 3-legged OAuth 2.0 for access?
  • Is there a specific port or protocol to use?
  • Do you need to be whitelisted?

Abstraction is more than arranging HTTP verbs and authentication methods into buckets or classes. It's more methodical than jumping right to the physical abstraction layer. It really does help you scope out the challenge of integrating two very different systems and gives you the requirements and a roadmap for how to approach the project.

Let’s consider examples of an integration between two SaaS applications:

Image title

Figure 1. Determining how you will integrate two very different applications starts with breaking the problem into three abstraction layers.

Example #1

The project is to setup a bi-directional integration between the Microsoft Dynamics 365 sales module and the HubSpot marketing automation system. The integration needs to offer recurring updates in batch operations. The process starts by evaluating each application on the three abstraction layers, resulting in:

Abstraction Layer

Application 1

Application 2

View

Dynamics 365 Sales/CRM (SOAP SDK), ADFS Auth

HubSpot (RESTful HTTP API), OAuth 2.0

Logical

Leads, Contacts, Accounts, Opportunities

Contacts, Companies, Deals

Physical

Flat/Relational data, XML payloads

Hierarchical data, JSON payloads

This reveals that the integration will need to make the two applications compatible on the logical layer by translating between the CRM’s Leads and Contacts, with HubSpot’s Contact entity.  This also gives shows how compatible these applications are on the physical layer by quickly presenting how the data is stored (flat/relational vs. hierarchical) and how the data is accessed and transferred (SOAP and XML vs. RESTful and JSON).  These become the key requirements for the integration project.

Example #2

In any integration project, the abstraction method will reveal the unique requirements of that specific project. Consider another example to create a bi-directional integration between the Dynamics 365 Operations module (or Dynamics AX) and Salesforce. In this case, the goal is to use event-driven messaging and transfer information on transactional operations between the applications. Here is what results using the abstraction method:

Abstraction Layer

Application 1

Application 2

View

Salesforce.com (SOAP), Basic or OAuth 2.0

Dynamics 365 Operations/AX (OData), Basic Auth

Logical

Accounts, Products, Opportunities, Opportunities Line Items

Customers, Items, Sales Order Headers, Sales Order Line Items

Physical

Flat/Relational data, XML payloads, has fields for Last Modified (when/by who)

Flat/Relational data, JSON payloads, does not have fields for Last Modified

This second example shows that the logical layer aligns nicely between these systems, however, data about when records were changed and by whom was not included in #2; that will pose problems for tracking net-change later on.

The abstraction method is a powerful way to start looking at integration projects.  It provides a structure to start analyzing the task of integrating very different applications. By breaking down integration concepts into smaller chunks, it becomes much easier and faster to pair the most complex and robust monolithic API with the latest-and-greatest microservices.

Concept 2: Metadata

Metadata is data about data and is important to integration, especially in the logical level of the abstracted integration. Metadata is the universal layer between the logical operations and field mappings (i.e., Upsert a HubSpot Contact), and the underlying API method(s) (i.e., an HTTP POST of a JSON body). More specifically, Metadata is information about the entities/objects, their supported actions, and fields available via API.

In order to create any logical mapping from one application to another, it’s valuable to have structural and descriptive information about the applications. Think of metadata as a card catalog to an API. Don't remember card catalogs? How about a Swagger specification/definition?

What to Define

When building an integration, it's helpful to define metadata for each of the following.

  • Entities. This should be a dictionary of the available endpoints (or objects) you plan to use. Be sure to describe any special properties (i.e., such as supports Bulk/Batch operations).
  • Actions. These are the operations you wish to support on your entities. Examples of actions are your common HTTP Verbs, CRUD operations, or special/virtual operations like Merge or Add-to-List. It’s a good idea to use universal terms (i.e., instead of “PUT” or “Merge,” use “Update”). 
  • Properties. These describe the fields and their data types. Remember that properties may be different depending upon requests and responses. Also, make sure to describe special properties like primary/foreign keys, indexes, required/not null, etc.
  • Relationships. Be sure to define any relationships that an entity has to another entity, and how that relationship is described (this could be a foreign key or a virtual property). Be sure to distinguish between parent, child, and complex relationships. 

After the metadata has been defined it will be easy to determine the types of operations that can be performed on it (i.e., Query, Update, Create), its input and output properties (i.e., firstName, lastName, email, and accountId), its declared relationships with foreign keys (i.e., my Contact relates 1:1 to an Account via an accountId property), and its special properties (i.e., supports Bulk).

Discovering Metadata

Enterprise software and many SaaS applications are often easily customizable by users. This makes defining the metadata definition more difficult. Fortunately, there are design patterns for defining instance-specific Metadata – or what I like to call, discovering metadata!

One route to describing the metadata dynamically is using a service, such as a Swagger definition, WSDL, or OData $metadata web service. If there’s no service available, Reflection can always be used on source queries. These are topics beyond the scope of this article and there are pros and cons to discovering vs. static metadata. Suffice to say that sometimes both will be used.

Robust metadata provides the means to understand how to map collections, and where functions and formulas are needed for transformations.

Concept 3: Standardization

Standardization is a concept to consider at the physical layer of the integration. Having a standard approach to integrating applications will pay off on future integration projects. Consider the API landscape with various protocols and architectures, further complicated by different payload types and data structures.

The standardization should be a methodical approach with interfaces including (but not limited to) authenticating, getting data, and setting data. This approach avoids the need to use a canonical or generic model.

Data Structures

There will likely be many different formatted payloads (i.e., XML, JSON) that need to be managed, however, those formats typically coincide with different data structures, which is more important to understand.

Here are some examples of different data structures.

Flat

Flat data contains one row for each primary record, but may contain duplicate primary records to provide detail on its related records, as shown below for this Sales Order:

OrderNum, OrdAmount, LineNum, ItemName, ItemAmount

ORD-100, $396.00, 1, XboxOne S, $300.00

ORD-100, $396.00, 2, Forza Horizon, $60.00

ORD-100, $396.00, 3, Tax, $36.00

Relational

Relational data associates records together, and typically includes foreign keys. Relational APIs allow you to perform inner and outer joins to understand 1:N, N:1, or N:N related records. SOAP APIs may pass payloads that contain relational data in a SOAP envelope or XML document, such as this Account and Contacts example:

<Account>
	<Id>Scrb123</Id>
	<Name>Scribe Software</Name>
	<Contact>
		<Name>Nate Keefe</Name>
		<CompanyId>Scrb123</CompanyId>
	</Contact>
	<Contact>
		<Name>Peter Chase</Name>
		<CompanyId>Scrb123</CompanyId>
	</Contact>
</Account>

Hierarchal 

Hierarchal structures represent an efficient way for API providers to send or receive all relevant data about an object. This is common practice in many RESTful APIs and webhooks. However, this structure can quickly create complexities for data transformations as the relationships can be nested and undefined. Here’s an example of an outgoing webhook containing event data:

{
    "messageId":"ABC",
    "message":
    [
        {
            "eventType":"Meeting",
            "eventCustomField":
                [
                    {
                        "name":"Event Room",
                        "value":"Merritt"
                    },
                    {
                        "name":"Locked",
                        "value":"False"
                    }
                ],
                    "eventTitle":"Strategic Integrations 2017"
            },
            {
                    "eventType":"Conference",
                    "eventTitle":"Integration Summit 2017"
            }
        ]
    }

Serialization

One solution to address the inconsistency of data structures and their varying payloads is to perform serialization (i.e., translate from one data structure to another) and deserialization (i.e., extract the structure from the data itself).

Serializing the data is not the same process as creating a common data model to fit a square peg into a round hole. By deserializing payloads, tasks for the various nuances in APIs – grouping, joining, merging, pagination, etc. – will be able to be grouped, regardless of the payload format. Serialization will also help unify error handling and exceptions across payloads.

For more context, consider some of the common manipulations you'll want to handle:

  • Map from a collection of Xs to a collection of Ys (turns an X into a Y).

  • Filter out elements of a collection (i.e., take an X and return false if it should be filtered).

  • Flatten collections (take an X and return a collection of Y).

There's a slew of tools and frameworks to help with serialization:

Without a common practice for dealing with varying payloads and handling their structures, there will be very little reusable work between integration projects. Understanding data structures will also help create integration requirements and user stories while not understanding these nuances may lead to unforeseen hurdles and project scope creep.

Conclusion

Today, there are more integration options than ever available to development teams. Despite the availability of easy to use tools and even no-coding tools, integration remains a complex field because of the intricacy and variety in applications. Understanding of the three main concepts of integration abstraction will greatly aid in preparing to scope and starting an integration project. Mastery of these concepts will lead to being able to build more robust integrations and lower total cost of ownership once the project is completed.

Your API is not enough. Learn why (and how) leading SaaS providers are turning their products into platforms with API integration in the ebook, Build Platforms, Not Products from Cloud Elements.

Topics:
integration ,abstraction ,metadata ,standardization

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}