Over a million developers have joined DZone.

Back to the 90s: Moving on from Flat Files to Automated Integration

DZone's Guide to

Back to the 90s: Moving on from Flat Files to Automated Integration

The use of flat file transfers is still prevalent, but thanks to Cloud Elements there's a better, easier way to handle this.

· Integration Zone ·
Free Resource

How to Transform Your Business in the Digital Age: Learn how organizations are re-architecting their integration strategy with data-driven app integration for true digital transformation.

The saying "I still think 1990 was 10 years ago," so perfectly reflects not only my current view of time, but also how many cloud applications, even modern ones, handle integrations. The use of flat file transfers as a supported method of syncing data - which was the most common way of doing so in the 1990s (and even in the early 2000s) - while proven, often requires quite a bit of manual intervention to get the data in the correct format before it can be loaded into the receiving service. Preparing flat files is also error prone - mistakes happen and when they do, they are often hard to undo.

There are plenty of reasons (code for excuses) as to why flat files are still used:

  1. There are so many services out there that it can seem impossible to create a reliable integration to each one - it simply costs too much and takes too long to integrate directly via APIs into each endpoint.
  2. Providing a flat file export and import capability in your product is a good, MVP way to get integrations going and get to market. It is zero or very low cost for your customers as they already understand how to make flat files work, and can use interns to support the process.
  3. Flat files make moving and handling data in bulk easier than "singular" API calls as many endpoints don't support bulk data transfers natively via their API.

For example, let's say your business handles the actual payments for your customers. Likely they are using something like Quickbooks, NetSuite, Freshbooks, Plaid or some other financial software to run their business. What they buy your service for is to facilitate making payments out to their vendors, thus they need to let your service know who and when to pay out.

The reverse is true too. For example, once payments are made within your service, your customers will need to extract the data from your service, and load it into their finance system. The fastest way to make this happen is to use a flat file transfer import/export mechanism. Even modern, prominent, successful payment services use flat files for transfers. In fact, there are several YouTube videos showing how to do this, mainly for importing payment status data into Quickbooks.

For the second use case where your customers would export payment status data from your service and load it into Quickbooks Online, there are preset templates that QBO provides to do such. It is quite a process to prepare the data and move it from an exported flat file to a format that QBO can ingest, but it works.

What if I Were Able to Show You a Better Way?

Let's start with the user experience that your customers expect. Regardless of whether you are syncing payments, invoices, employee records, contacts, opportunities - any object really - the following user experience is what your customers are looking for:

  1. Click to authenticate into external service, e.g. Quickbooks.

  2. If needed, a simple UI to adjust default data mappings to accommodate for custom data in your customer's specific endpoint payment data configuration.

  3. Set the start date for the data if manually retrieving data from endpoint (e.g. give me all the payments that are authorized in Quickbooks in the last 2 days).

  4. Activate transfer.

That's it, nice and automated. No need for YouTube videos to show how to pull and prep the data. You've successfully moved on from 1990's technology.

Even better, you really should provide an option to automate the pull of payments at a frequency of your customer's choosing - set it and forget it. Bam, welcome to 2018! This is what your modern customers will expect of your application - to already have this type of automation built in and ready to go.

Wait. Hold on. Right now you are thinking, " That is a fine user experience, but how can I overcome the barriers listed above and make it so that, as an MVP, my customers can sync data directly from within my application? " Answer: The Cloud Elements Platform.

Let's Knock Down the Barriers One at a Time:

First up, we support over 150 integrations, called Elements, which are super smart, ready to use connectors that have a uniform REST API interface. We make each endpoint look nearly the same from an integration perspective. We've done the hard work of learning the nuances of each endpoint and normalizing to a common, uniform REST API spec. You do the work once and then leverage it across many endpoints.

Not so scary now, is it?

Second, because of the smarts that are built into our Elements, much of the functionality needed to get to an MVP is already done, ready to use. For example, our Virtual Data Resources to map and transform data—easy to set up for each endpoint allowing your app to interact with only the data that it requires. Uniform REST APIs for retrieving and posting data in real-time. Let our platform's API resources shape the data for your customers automatically rather than manually doing so within flat files.

Third, regarding moving data in bulk. Once again, our Elements come to the rescue. Every Element in our catalog provides uniform bulk API resources - even if the endpoint doesn't provide them natively (like QBO). Poof, there goes the last objection on the list, moving data in quantity via APIs.

Oh, and for those of you who really want to hit it out of the park, let's automate! The Cloud Elements platform provides Formulas, which are configurable workflows that can be set to run automatically at a given frequency to sync your customers data, automatically and in near real time, with your application. Now your talking! Welcome to the next generation, welcome to 2018.

Anyone recall the hit song by Killing Joke? I think it was titled, "Living in the 80s" or something like that. Ahh, the 1980s, I think they were only like 15 years ago, right?

Need to get started? Get your free trial today!

Enjoy. Aloha.

Make your mark on the industry’s leading annual report. Fill out the State of API Integration 2019 Survey and receive $25 to the Cloud Elements store.

integration ,apis ,flat files ,automation

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}