Multi-Protocol File Transfer API Integrations in Python Using Zato
See how to set up Zato to do your transfers and processing for you.
Join the DZone community and get the full member experience.Join For Free
In many domains, the transfer of static and batch files is an important part of systems integrations and a large number of applications produce and expect data in the form of files rather than network-based APIs. In this article, we shall see how Zato makes multi-protocol integrations of this kind possible in a way that is secure, scalable and easy to extend in Python.
File transfer is often found in scenarios such as:
- Mass-transfer of invoices, e.g. an ERP may produce an hourly list of new ones to process
- Batch-transfer of patient data in healthcare systems
- Publication of updates to data such as stocks or exchange rates
- Secure exchange of orders with external business partners, e.g. using files encrypted with PGP
There are many more scenarios but what they all have in common is that in each such data flow there may be a single producer and multiple recipients of information, each of them preferring to use different technologies.
For instance, a single file saved to a Windows share may go to a REST endpoint, an IBM MQ or AMQP queue, an AWS S3 bucket, and to e-mail recipients, each of which may be an independent application or business entity and each may use a different data format, e.g. what is CSV initially may need to be transformed to JSON, XML, and PDF, depending on the receiver.
Let's see how Zato helps in the implementation of processes of this kind.
File Transfer Channels
The core Zato concept in file integrations is that of file transfer channels - reusable definitions pointing to producers and consumers of data along with accompanying details, such as a schedule, file patterns to use, what encoding the data is in, and more.
Zato will connect to the data source automatically and handle each new or modified file according to the channel's configuration, i.e. by parsing and distributing it to data recipients, allowing one to apply any transformations or enrichment as required, and optionally deleting the data file from the source location afterward.
In Zato web-admin, a file transfer channel may be defined as below.
Let's consider the following sample definition. In this particular case:
The data source is an SFTP connection definition called CRM
New or changed files are processed according to a schedule configured for a job called Transfer Invoices Job
Multiple input directories and glob file patterns can be used
Among of data recipients are REST outgoing connections — any file picked up from the source location will be sent to a REST endpoint directly as-is, without any programming needed
A file can be sent to a service — and the service may transform or enrich the file as needed, e.g. it may change its data format, invoke more APIs to obtain additional data, combine it all and send the result to one or more recipients
A file can also be delivered to a publish/subscribe topic — again, with multiple recipients and data transformations taking place before it is delivered to its final destination(s)
Note the flexibility and reusability - the same data source, schedule and recipients can be configured to take part in multiple file integration processes without influencing each other.
In fact, the same services and endpoints may independently participate in both file and online API integrations.
To showcase how file transfer works in practice, let's implement an example process:
- An external application periodically produces customer orders as CSV
- The CSV files are made available on SFTP that Zato can access
- They are to be checked once a minute
- Each file needs to be sent to a REST endpoint as-is, an AMQP queue as JSON and as an Excel e-mail attachment to selected people
Initial web-admin Configuration
We need to configure a few Zato objects for the new file transfer to use, check below for details of how to do it:
- An SFTP connection
- A scheduler's job
- A REST connection definition
- An AMQP connection definition
- An SMTP connection definition
Simply fill out a form such as the ones below, click OK and you are done. No server restarts are needed.
REST Recipients Are Special
Because so many new integrations are based around REST, recipients of this kind are treated specially by file transfer channels in that there is no programming required for files to be transferred to REST endpoints.
When you create a file transfer channel, simply assign one or more REST endpoints to it and Zato will send all the files to them as soon as they are picked up from the data source.
As to other recipient types, we will use the Python-based service below.
Let's quickly implement a service that will transform and send the files to AMQP and SMTP recipients.
The service will receive its input in an attribute called self.request.raw_request - we will transform it to the required data formats and send to both AMQP and SMTP.
Creating a New File Transfer Channel
Now, we can create a new file transfer channel. After clicking OK, it will start to work.
If there is anything that needs changing, click Edit in web-admin and enter new information, e.g. new directories to look for files in.
If anything changes in any of the pieces of configuration that the channel uses, such as AMQP, REST, or anything else, the channel will reconfigure automatically, without a need for a manual update.
Here we are, using Zato, we have just created a multi-protocol file transfer hub, integrating SFTP, AMQP and e-mail.
New protocols and recipients can be added at any time without disrupting existing file integrations — we can easily extend the service to transform and push the input files to SQL, Cassandra, and Redis, for instance.
And if new data sources are needed, just create new channels, start them in addition to the existing ones and they will begin to work immediately. It is just that easy!
Opinions expressed by DZone contributors are their own.