Orchestrating Puppeteer From Lambda With Zero Lines of Code
In this article, we will look into the details of how to use the Kumologica Puppeteer node to capture website content.
Join the DZone community and get the full member experience.Join For Free
In this article, we will look into the details of how to use the Kumologica Puppeteer node to capture website content. For those who are not familiar with Kumologica, let me give a quick overview.
Kumologica is a free low-code development tool to build serverless integrations. Kumologica provides drag and drops visual designers to build your integration flows or micro-services that can be tested and deployed on to a serverless infrastructure (AWS Lambda).
Having the ability to take screenshots of dynamic website content and storing it for later reference is a regular need in integration. Kumologica supports this use case by providing the Puppeteer node.
We will look into the steps of how to go about capture the results of amazon.com website search, and storing it in an S3 bucket for future reference and analysis.
We will develop a single flow that will be deployed as an AWS lambda function.
Outline of the use case implementation is as below:
- The flow will receive the URL that we need to create a snapshot of in JSON format as part of the request.
- The flow will use the Puppeteer node to take a snapshot of the URL provided in PDF format.
- The flow will use the AWS S3 node to save the PDF to an AWS S3 bucket.
Follow the development process as outline below:
- Open Kumologica designer.
- On the popup, Click Create New Project.
- Provide the Project Name and Location to save your project and click Save.
- Drop from the palette an EventListener node onto the canvas with the EventSource as Amazon API gateway, verb as POST and resource end-point as /snapshots
- The puppeteer node is part of contrib nodes. Click on the button “Add more nodes” to add the Puppeteer node to the main palette. The newly added node will be available under the category “AUTOMATION”.
- From the palette, add Puppeteer node onto the canvas. Configure it as outlined below.
- The result of this node processing will be available as nodejs Buffer object at msg.payload.Dynamic-PDF.result
- From the palette, add AWS S3 node onto the canvas. We will use the AWS S3 node to store the PDF results onto an S3 bucket.
- Configure the AWS S3 bucket as outlined below.
NOTE: Content field of the S3 node supports dynamic JSONata based expressions. In JSONata, ‘-’ character has to be escaped with single quotes. If there is no ‘-’ character, it could be written with single quotes. eg: msg.payload.DynamicPDF.result
The resulting flow must be looking like this:
Let’s create our test case flow.
Click on the “Test” tab to switch to the test flow canvas.
- From the palette, add the TestCase node to the canvas. TestCase node is available under the category TESTING.
- Configure the TestCase node as outlined below:
- The TestCase node will target the message to the EventListener node. From the Target Node dropdown, select the EventListener node. We had named it “POST: /snapshots”.
- The event field will be populated with the sample event for EventListener type.
- Change the payload to the one below.
- The path is set to /snapshots and HTTP method to POST. This is the resource endpoint configuration provided in the EventListener node.
- In the body, we will be passing the URL that needs to be used against the key endpoint URL. During runtime, this value will be set into the message at msg.payload.endPointURL
- From the palette, add TestCaseEnd node to the canvas. TestCaseEnd node is available under the category TESTING.
The resulting test flow should be looking like below:
- Select the TEST tab from the right-side panel.
- Select the AmazonSearchPDF from the drop-down. This is the name that we gave to the TestCase node in the TEST tab.
- Click the Run button. Verify the results in the Amazon S3 bucket.
- Download the PDF created.
Thanks for reading (or watching the video tutorial). We hope you enjoyed this use case and appreciate how simple is to implement serverless integrations with Kumologica. We would love to hear your feedback and don’t hesitate to contact us if you have any questions.
To learn more about Kumologica check out our official documentation at kumologica.com and subscribe to our video on our YouTube channel.
Opinions expressed by DZone contributors are their own.
Part 3 of My OCP Journey: Practical Tips and Examples
A React Frontend With Go/Gin/Gorm Backend in One Project
What Is Envoy Proxy?
From On-Prem to SaaS