Splunk Logging in Lambda Using Low Code Approach

A centralized logging management solution like Splunk, Datadog, Sumologic, etc. enables organizations to collect, analyze, and display logs through a single pane of glass.

In this article, we will see how application logs can be sent to Splunk from lambda using the Kumologica Splunk node.

Kumologica is a free low-code development tool to build serverless integrations. You can learn more in this medium article.

Use Case

In this use case, we have a mobile app application that consumes a Mobile Channel API. Behind the mobile API, we have an Order FulFilment API. The Mobile API invokes the Order FulFilment API for completing the process. When the mobile app sends the very initial request it has a Tracking id. The Tracking id traverses from the Mobile Channel API to Order FulFilment API and back.

The message passed across both applications needs to be traced based on the Tracking id in Splunk.

Prerequisite

1. Kumologica designer installed in your machine. https://kumologica.com/download.html

2. Splunk cloud Trial account

Implementation

For the above-mentioned use case, we will be creating two API’s, a Mobile Channel API, and an Order FulFilment API.

Order fulfillment API

1. Open the Kumologica designer and Create a New project. And provide the name of the project as OrderFulFilmentService

2. Remove the default hello world flow.

3. Drag and drop EventListener Node from the pallet to the canvas and open the EvenListener node settings and configure the following.

Display Name : POST /order    

Event Source : Amazon API gateway     

Verb : POST     

URL : /order

4. Add Splunk Node from the Logging category.

By default, Splunk Node will not be available as part of the default Logging category.

To install the Splunk node, Go to Add More Nodes Option on the bottom of the pallet and click Install Splunk. Once Installation has completed a restart of the Designer will be required. Open the Splunk Node settings and configure the following.

x

Display Name: Log_Entry     

Host: hostname of your splunk trial account     

Port: 8088

Message: {“TrackingID” : msg.payload.TxnID,”ServiceName”:”Order_FullFillment”, “Data”: msg.payload }     

Splunk HEC Token: Splunk HTTP Event collector Token

5. Wire the EventListener Node to the Splunk Node.

6. Drag and drop the Datamapper node from the palette to the canvas.

We are using the Datamapper node to mimic the response after an Order processing.

Open the Datamapper settings and configure the following.

Display Name: Order_ProcessResponse

Mapping: 

x

{

 “TrackingID” : msg.payload.TxnID,

 “OrderID” : msg.payload.OrderID,

 “Status” : “Processed”

}

If you are not familiar with using the DataMapper node then would recommend going through the following article.

7. Wire the Splunk Node to the DataMapper Node.

8. Add the EventListener End node from the palette to the canvas. Open the settings and configure the following.

Display Name: Success     

Payload: msg.payload

With this, we have completed the development of our Order FulFilment API.

Now let’s deploy the API as lambda to your AWS account. If you are not familiar with the deployment of a Kumologica flow I would recommend having a walkthrough on the following medium article.

Note: Copy the service URL that you receive on the terminal after a successful deployment.

Mobile Channel API

1. Open the Kumologica designer and Create a New project. And provide the name of the project as MobileChannelAPIService

2. Remove the default hello world flow.

3. Drag and drop EventListener Node from the pallet to the canvas and open the EvenListener node settings and configure the following.

Display Name : POST /mobile

Event Source : Amazon API gateway

Verb : POST

URL : /mobile

4. Add Splunk Node from the Logging category.

By default, Splunk Node will not be available as part of the default Logging category.

To install the Splunk node, Go to Add More Nodes Option on the bottom of the Pallet and click Install Splunk. Once Installation has completed a restart of the Designer will be required. Open the Splunk Node settings and configure the following.

Display Name : Log_Entry     

Host : *hostname of your splunk trial account     

Port : 8088

Message : 

xxxxxxxxxx

{

 “TrackingID” : msg.payload.TxnID,

 ”ServiceName”:”Mobile_Channel”, 

 “Data”: msg.payload 

}

Splunk HEC Token : Splunk HTTP Event collector Token

5. Wire the EventListener node to the Splunk Node.

6. Add the HTTP Req node from the web category in the palette to the canvas. Open the settings and configure the following.

Display Name : Invoke_OrderFulFilmentService    

Method : POST     

Url : your order service url. Configure this once

Order FulFilment Service is deployed and you receive the url .

Return : a UTF-8 string     

Response Timeout : 120000ms (default)     

Authentication : None

7. Wire the Splunk Node to the HTTP Req node.

8. Add the EventListener End node from the palette to the canvas. Open the settings and configure the following.

Display Name : Success     

Payload : msg.payload

With this, we have completed the development of our Mobile Channel API.

Now let’s deploy both the services separately as two different Lambda in your AWS account.

Testing

For testing purposes, I am sending the following request as input to the Mobile API.

xxxxxxxxxx

 {

    “OrderID”: “APQ-00001”,

    “TxnID” : “0948939-AWHDH”,

    “Item” : {

    “Name” : “ABC Laptop”,

    “Type” : “Mini-5”,

    “Qty” : 1

    },

    “Address” :{

        “Unit” : “9”,

        “Street” : “TTK Street”,

        “Road” : “Tower Road”,

        “State” : “NSW”,

        “Country” : “Australia”,

        “PIN” : 94839

    }

   }

After you invoke the mobile channel API and when you search in Splunk with the TxnID value as the search parameter.

You would get the following log entries.

If you want to try out this application quickly you can import the Mobile Channel and Order FulFilment API flow from this project’s repo.

Summary

Thanks for reading and I hope you enjoyed this use case and appreciate how simple it is to send Splunk event from lambda using Kumologica. We would love to hear your feedback and don’t hesitate to contact us if you have any questions.

To learn more about Kumologica check out our official documentation at kumologica.com and subscribe to our youtube channel