Create Lambda Layers in AWS Lambda

For a while, I struggled when it came to making layers for my functions. I used to download them locally, zip them, upload them into S3 when their sizes were big, and then create a version. This process takes a long time, and chances are high that you will make a defective layer when you’re using Windows or Mac because of some layers that compile binaries when you download them.

I wrote an article back in 2019 on how to do that with the help of Docker, and it still gets read, which triggers the need to create a new, simple, and extremely fast way to use this amazing feature: Lambda Layers.

I will walk you through what it does and how you can have it in your account. Please note that this process is only (currently) for Python3.8. 

What It Does

This script consists of 3 main steps: create a new layer, update the existing one, and read what’s inside your latest layer version.

Create a New Layer

Because of the struggle I mentioned at first, this process is time-consuming when it comes to the manual, traditional way. Therefore, with providing some key information, the script will: create the directory structure, install the libraries with PIP, calculate the direct size to prevent exceeding layer limit, zip it, upload it into newly created S3 bucket (or existing if you have one), and finally, publish the new layer.

These steps are the minimum that you can do to create a new layer. Please note that some values are hardcoded. It can be easily made dynamic, but it's out of the scope (currently).

Update Existing Layer

Managing an existing layer could be difficult, as you will need to do many steps to maintain the existing libraries and add new ones. This script will get the referenced zip file of the latest version of your layer, download it, add to it, upload it into S3 again, and publish a new layer version. 

Read Layer Content

Because of the problem that log4j made a couple of weeks ago, and what threat existing resources made if they were affected, you will need to monitor your resources and update them accordingly. This action will get the latest layer version zip file, extract it, and use PIP to check what’s inside and which version you have. This could also help maintain supported libraries versions.

Enough talking! Let's dive in:

Preparation and Execution

We will go through the steps in order.

S3 Steps (If You Don’t Have One Already):

1. Login into your AWS account and go to S3.
2. Create a new S3 bucket, keep it in the same region you work in.
3. Set it up as you wish: no red lines are here.

Lambda Steps:

1. Go to the Lambda console and create a new function.

2.  Open the function -> Click on Configuration -> Click on Permissions -> click on Role Name.

3.  Click on Policy Name -> Edit policy.

4.  Paste this policy (edit resource as you wish) -> save it.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "lambda:ListFunctions",
                "lambda:ListLayerVersions",
                "lambda:ListLayers"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "lambda:GetLayerVersion",
                "lambda:DeleteLayerVersion",
                "lambda:AddLayerVersionPermission"
            ],
            "*"
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObjectAcl",
                "s3:GetObject",
                "logs:CreateLogStream",
                "lambda:PublishLayerVersion",
                "s3:ListBucket",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

5.  Go to GitHub and get the code -> copy the whole code in code.py and paste it into your function.

GitHub Repo

6. Click on Test and paste this JSON (all fields are required)

{
  "layer_name": "LAYER NAME",
  "s3_bucket": "YOUR S3 BUCKET",
  "libraries": ["LIBRARIES"],
  "action": "create_new"
}

NOTE - In the action key, there are 3 valid values: create_new, update, read_only

7.  Test the function to trigger it. Give it a try.

This is proof on this script execution:

Feel free to adjust the code as you wish, and let me know if you have any issues.

Conclusion:

This is a way of doing this procedure, and I’m pretty sure you have your own way too. Let me know: is it worth it? Do you know how to code in Node.js or Java and wish to see this script take another turn and provide another language's support? I’d be thrilled if that happened.

Stay Safe.