Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Managed File Transfer

DZone's Guide to

Managed File Transfer

Find out more about RoboMQ's Manage File Transfer offering, including how to configure it, how it works, how it handles errors, and more.

· IoT Zone
Free Resource

Address your IoT software testing needs – improve quality, security, safety, and compliance across the development lifecycle.

Managed File Transfer (MFT) is a reliable and guaranteed delivery file transfer service with all the management, error handling, and recovery features built in for a secure and encrypted file transfer. It supports managed file transfer across any of the file transfer protocols (be it FTP, SFTP, FTP over SSL, etc.) and various file systems (Windows, Linux/Unix, midrange systems, S3, object stores, cloud storage etc.). The file transfer is secure and encrypted and tracked through the transfer and in-transit processing providing a reliable enterprise-grade MFT platform for managing the secure transfer of data from the source location to one or more destinations through public, private, and hybrid networks.

Introduction

RoboMQ MFT service allows you to transfer files between file systems using a variety of file transfer protocols. The Managed File Transfer platform is built on the strength of RoboMQ Message Oriented Middleware(MOM), providing guarantee delivery reliable data transfer. It utilizes the RoboMQ broker for receiving and tracking events identifying each step of transfer and in-transit transformation.

In addition to regular MFT functionality, you get following silent advantages while using RoboMQ.
  1. Support for all file transfers.
  2. End-to-end tracking for the file transfers and transformations.

  3. Ability to break files into messages and integrating the data with APIa. This approach offers integration with non-file based APIs and systems.

  4. Microservices based approach for data transformation, enrichments, and validation
  5. Robust error handling, monitoring, alerts and Error Analytics

Setup and Configuration

The MFT service needs to talk to your RoboMQ tenant, therefore you will need to provide your vhost or tenant ID, username, password, and your preference for secure transfer over SSL.

To transfer files from one location to another, MFT service requires access to four folders — let's call them upload, inbound, outbound and destination (you can use any name you want):

├── destination
│   ├── archive
│   └── error
├── inbound
│   ├── archive
│   └── error
├── outbound
│   ├── archive
│   └── error
└── upload
    ├── archive
    └── error

Those folders don't need to be under the same directory. Actually, they can and often do reside on different servers, file systems, or S3 buckets.

Note that you will need to provide credentials of the accounts for accessing those file systems and make sure those accounts have read and write permissions on the above specific folders.

In case you need some files under those folders but don't want them to be transferred, you can configure a filter to ignore files with names that match a certain pattern.

At this point, the MFT service configured and is ready to work.

Workflow

The following picture depicts a typical MFT file transfer flow.

Work Flow Diagram Fig 1: High-level schematic of a file transfer flow

MFT has a file listener, which scans the upload folder on a configurable interval. Once an expected file is detected, it will be copied to one or more inbound folders (multiple MFT services can work together in case the file needs to be duplicated and processed at multiple destinations).

If the file requires specific validation or transformation, an optional MFT microservice can perform the transformation, validation, or data enrichment on the file in the inbound folder. After successfully processing the file, this component moves the file to the outbound folder.

Finally, another MFT microservice will pick it up from the outbound folder and put it into destination.

Note that just like with any integration with RoboMQ, there could be a whole chain of microservices performing a variety of functions including transformation, validation, data enrichment, or the application of any business logic while the file is in transit from source to the destination.

At each step of the processing, the original successfully processed files are archived in the corresponding archive folder.

Error Handling, Alerts, and Monitoring

Upon any failure at any point, the current file will be moved to corresponding error folder. At this point, the RoboMQ error handling process kicks in. It is robust and has multiple options for tactical and strategic handling of error cases.

  • An error message (includes file information) will be sent to an error queue, and a notification message (describes what is the error and how it happened) will be sent to a notification queue.

  • An SMS, an email, or a phone call could be triggered to the operations team or the responsible party based on setup of the severity rules and notification mechanism.

  • A ticket or a case can be created in any of the case management platform being used like ServiceNow, Jira, Salesforce, etc.

  • All the errors are saved into a real-time database, which provides an analytical dashboard view of the errors for long-term trends, root cause analysis, and corrective actions

Error Analytics Dashboard

Fig 2: Error Analytics dashboard for analysis and corrective actions.

Accelerate the delivery of high-quality software in the connected IoT era through an integrated analysis, testing, security, and analytics platform

Topics:
file transfer protocols ,enterprise

Published at DZone with permission of Victoria Gosalvez, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

THE DZONE NEWSLETTER

Dev Resources & Solutions Straight to Your Inbox

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

X

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}