An AWS Lambda Advent-ure in Python
For many of us, Advent is a time to look forward; a countdown to Christmas; 24 days in December at the end of the year that we mark with special calendars. H...
Join the DZone community and get the full member experience.Join For Free
For many of us, Advent is a time to look forward; a countdown to Christmas; 24 days in December at the end of the year that we mark with special calendars. Hiding behind little windows in a festive picture are sketches, quotes, or perhaps even chocolates! I've built a simple AWS Lambda application to publish a custom Advent calendar with programming videos behind the windows.
I've published the code for this on my Github profile so you can clone it and run your own calendar.
There are three areas I want to focus on with this exercise,
- Programming: I'm picking Python...
- Deployment: let's use AWS Lambda to keep the future days secret.
- Infrastructure: we'll use the CDK to define our infrastructure in code.
Our calendar needs a data source. Rather than having to deploy a CMS, I'm using Google Sheets, so there are no programming skills required to update the contents of our calendar.
Our calendar application will fetch this data and generate an HTML page from it. You can see the published calendar here (apologies for the ugly URL).
If you want to simplify your deployment and avoid the Google Sheets integration, the code also supports loading the data from a bundled CSV file (which makes it a little faster too).
Let's step through the production code.
Since we're planning to use an AWS Lambda Function for this, we have a single entry point for our application. That's defined in
def handler(). It receives an event object and some context, both of which come from API Gateway in our solution.
Our handler does only 4 things. It:
- fetches the calendar data from Google Sheets,
- checks to see if it's in preview mode,
- builds the calendar webpage if the request wants HTML, or,
- returns the calendar data as JSON otherwise.
We call out to other modules that will do the work for us. One to fetch the Google Sheet data and the other to build the HTML page. Our handler function only deals with Lambda specific details.
get_sheet_data()wraps the Google Sheets API (which is available as a python module). That API requires credentials, which I've separately stored in AWS Secrets Manager. Fetching the credentials and creating the Google service takes a wee while, so the first run of our Lambda can take a couple of seconds. Subsequent runs are cached though, taking advantage of the warm start feature of the Lambda service.
You'll see here that I'm injecting a mapper to convert the data returned from the sheet into a data structure I can use to build the calendar. This allows me to potentially re-use this module elsewhere with differently shaped spreadsheets. It also makes testing a little easier as I can decouple fetching from re-shaping.
Looking at the google_credentials module, there's some copypasta code from AWS for extracting the credentials from SecretsManager.
fetch_credentials()checks the environment to see if it's being called from Lambda (on Lambda, the
AWS_EXECUTION_ENV value is set). If so, it uses Secrets Manager, otherwise it uses a local credentials file.
Moving on to the
My 'server-side' rendering is done using built-in features of Python, Templates, and f-strings. If this system were to require more complex HTML or styling, I would probably introduce a templating framework, but this only requires three separate components, the page itself, the list of windows, and the list of panels. Here's a snippet that shows loading a template from a file and using Python's Template class.
And here's a snippet of my code to build the windows
So that's our application. Let's look at the infrastructure to host it.
Our infrastructure is pretty simple. We have an AWS Lambda function that's hooked up to an API Gateway. We could have written the CloudFormation template for this in YAML pretty easily. But since we're Python developers, we'll write it using the CDK.
As an example, when I set up my integration between ApiGateway and Lambda, CDK expects the integration to be of type IHttpRouteIntegration. My integration is a LambdaProxyIntegration, which is one of those, but PyCharm doesn't see it that way, giving me a warning that the types don't match.
You do get some auto-complete features and parameter hints, but they aren't as helpful as what TypeScript and WebStorm would give you.
One other thing of interest in our CDK code; we're using context to inject variables (the Google Sheet Id and Range). If we were writing pure CloudFormation, we'd probably do this using CfnParameters, but the developers of the CDK recommend against this approach. By using context, we have access to the properties at synth time (rather than deploy time) so we can use the values in flow-control or other parts of our application.
We can provide context values in various ways like the cdk.json file or by command line parameters (see the CDK documentation for more details)
Our next challenge is how to package our code and get it deployed into our infrastructure.
I am probably not doing this in the most pythonic way.
There are recommended tools for packaging, deploying, and distributing Python applications. The most common distribution tool is setuptools. In this application, I have not used setuptools as there's no clear pathway for using it together with CDK to deploy to AWS Lambda.
Instead, I have used a simple Makefile to provide a number of build steps that allow me to deploy my application with a single command.
I'm using Pipenv to manage my dependencies. Pipenv has benefits on top of pip for deployment, as it allows me to differentiate between development dependencies (CDK and pytest packages, for example) and production dependencies (for example, the Google API client).
In my CDK code, I can specify a zip file as my source asset for my Lambda Function, and I can use my Makefile to build that package.
My 'build' step installs my production dependencies into a build directory. Then it copies my function code into that directory. Finally, it zips that directory. My CDK stack targets that zip file.
Putting that all together, by setting my google credentials into SecretsManager, creating a Google Sheet, and adding its Id to cdk.json, I can deploy my advent calendar by typing
make build deploy in my terminal.
Step 1, Release. Step 2, Fix.
As with any application, there are improvements that could be made.
The Sheets API setup causes the Lambda function to be quite slow to run initially (though subsequent calls are much faster). I could give up the ease of editing by adding a CSV file to my lambda (in fact, there's code for this already).
There's no DNS integration here, so that needs to be done separately. It's straightforward to add custom domains to API Gateway. Alternatively, an integration with CloudFront would let you add additional caching and integration with an existing website.
If you were so inclined, you could develop your own SPA web-app. The existing lambda code will work out-of-the-box with that approach; all you need to do is ensure your HTTP request headers ask for JSON.
python -m instil course.py
If building Serverless applications with Python is something you need to do, Instil offers training in Introductory Python, Advanced Python, and Python for Serverless. We'd be happy to discuss your specific needs. We deliver virtually to companies all over the world and are happy to customize our courses to tailor to your team's level and specific needs. Come and check us out to see if we can help you and your team.
Published at DZone with permission of Ryan Adams. See the original article here.
Opinions expressed by DZone contributors are their own.