A Low Code Approach for Serverless File Extraction and Transfer
In this article, we are going to create a simple serverless file extraction and its content transformation using Kumologica.
Join the DZone community and get the full member experience.Join For Free
Extraction of files from a remote FTP server and processing the file content to a destination system is a common use case in small, medium, or large enterprises. There are multiple ways by which an enterprise deals with this use case. Mostly they depend on some File transfer products or end up building a solution using VM's. In both cases, the total cost of ownership (TCO) of the solution for simple file extraction and processing is high when compared to a Serverless solution.
In this article, we are going to create a simple serverless file extraction and its content transformation using Kumologica. For those who are new to Kumologica, I would recommend going through our articles and YouTube videos to get an insight.
The lab team in ABC enterprise shares the lab results to the processing team regularly at midnight via the FTP server. The resulting file is a zip file containing multiple CSV files. Each CSV file has to be placed on an S3 folder individually so that the processing team can access them.
- Kumologica Designer — Download the designer for building the flow.
- FTP server access.
- AWS S3 — For Kumologica flow to send the CSV file.
- AWS Lambda — For running the Kumologica flow as lambda.
Install the ZIP node and FTP node from "Add more nodes" panel in Kumologica designer.
Let's start building the Extraction service flow that will be triggered nightly once every day.
1. Open Kumologica Designer, click the Home button and choose Create New Kumologica Project.
2. Enter name (for example ExtractService), select directory for the project.
3. Press Create Button.
4. Drag and drop the EventListener node to the canvas the provide the following configuration.
Display Name : Trigger 11:30 PM Provider : AWS Event Source : Amazon CloudWatch Events
5. Add a Logger node and provide the following configuration. Wire the logger node to the EventListener node.
Message : 'Triggered at : ' & $now() Log format : string
6. Add FTP node from the palette and provide the following configuration. Wire the node to the logger node.
Display Name : Read From FTP Hostname : your FTP hostname Port : your FTP port (default 21) Username : your FTP username Password : your FTP password Operation : Get
7. Drag and drop the ZIP node to the canvas and provide the following configuration. Wire the node with the FTP node.
Operation : Extract Content : msg.payload
8. Add a Set-Property node and configure the following. This is to store the extracted payload to a different object so that payload is not lost during any subsequent process in the flow. Wire the Set-property node with the ZIP node.
Operation : Set Target : msg.files Source : msg.payload
9. Add a ForEach node to the canvas in order to iterate the files which is retrieved after the extraction by the ZIP node. Provide the following configuration for the ForEach node. Wire the ForEach node with the Set-Property node.
Input Array : msg.files Await? : Checked
10. Drag and drop the S3 node from the palette and provide the following configuration. Ensure that the S3 bucket exists in your AWS account. Wire the S3 node with the ForEach node.
Display Name : SendToS3 Operation : PutObject Bucket : ResultsCSV Key : msg.payload.filename Content : msg.payload.content Content Type : text/csv
Each iterated object by the ForEach node will be part of the msg.payload object.
11. Now we will end the iteration by adding the ForEachEnd node. Wire the ForEachEnd node to the S3 node.
12. And finally, We will add the EventListener End node to complete the flow. Provide the following configuration and wire the EventListener End node with the ForEachEnd node.
Display Name : Success Status Code : 200 Content-Type : text/plain Payload : 'Completed'
Deploying to AWS
1. Select the AWS icon on the right-hand side vertical tab of Kumologica designer, select your AWS Profile.
Note: If you haven't mapped your local AWS profile with the designer then you may follow the below video to configure it.
Mapping the AWS Profile to Kumologica Designer
2. Click Connect. If successfully connected will show the rest of the configuration options.
3. Set the Memory to 512Mb and Timeout as 20 seconds.
4. Go to the “Trigger” section under the Cloud tab and select the Amazon CloudWatch Event and click on '+Event'. The default will be the API gateway trigger.
Click the ';-' button to remove the API gateway trigger.
4. Now provide the Rule ARN.
For getting the rule ARN you need to create a cloudwatch watch event with the CRON schedule for the nightly trigger of the flow.
Amazon EventBridge is the preferred way to manage your events. CloudWatch Events and EventBridge are the same...docs.aws.amazon.com.
5. Now click Deploy.
Check your AWS lambda service in your AWS account to see the extraction service function deployed successfully.
Opinions expressed by DZone contributors are their own.