Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Configuration Updates to Applications in AWS

DZone 's Guide to

Configuration Updates to Applications in AWS

Since configuration updates occur so frequently, applications shouldn't need to be rebuilt and redeployed with each update.

· Cloud Zone ·
Free Resource

I have been working on building API's on AWS Cloud for a few years now. More often than one can guess, there is a need to update the configuration of an app that is already deployed. Through the years, we have evolved the design of software applications to make it configurable to account for unforeseen circumstances. Configuration changes often require rebuild and redeployment of the application. This article will discuss one approach where we don't have to rebuild and redeploy for configuration updates.

To have real-time push events we would need:

  • A two-way communication protocol,

  • A health check on the connection established

  • A way to trigger notifications from a variety of configurations in AWS (like Secrets Manager, DynamoDB, S3, etc.).

With these requirements in mind, let's take a look at the architecture below

Architecture

Image title

Implementation

AWS Appsync GraphQL engine provides us a pub-sub framework with web sockets (two-way communication) over MQTT to push our configuration changes to applications. A generic GraphQL schema would be good enough. An example of the schema is given below

type Mutation {
    updateResource(id: ID!, data: String): Resource!
}

type Query {
    getResource(id: ID!, data: String): Resource!
}

type Resource {
id: ID!
data: String!
}

type Subscription {
updatedResource(id: ID): Resource
@aws_subscribe(mutations: ["updateResource"])
}

schema {
query: Query
mutation: Mutation
subscription: Subscription
}


Our applications can depend on a wide range of AWS services for configuration. The most common would be Secrets Manager and Parameter Store, and some would even consider S3 and Dynamodb. Cloudwatch events can provide a way to listen for triggers when updates to configuration sources are made. Some sources like DynamoDB and S3 can provide lambda triggers directly.

To make our understanding simple, let's say our configuration source is a secret ID in Secrets Manager.

To process the events received from triggers from configuration sources, we will have a lambda (event forwarder) which will process and retrieve the updated configuration and publish the events/data to an Appsync GraphQL endpoint. It would publish the event by invoking the mutation updateResourcewith an ID in a standard ARN format. For example, for Secrets Manager it would be "arn:aws:secretsmanager:::<secret_id>."

Applications that require updates in real-time would subscribe to a topic on the AppSync GraphQL endpoint. For example, to listen for changes to a particular secret ID, the ID would be "arn:aws:secretsmanager:::<secret_id>" as it is the same as the ID/topic the event forwarder the lambda would publish updates to.

Once the application subscribes, any changes to the configuration source, which in our case, means any changes to the subscribed secret ID in Secrets Manager, our application will get notified with its content. To keep it safe from special characters within the content, it would be wise to base64 encode it.

The source code of the above architecture is available in Github. Also to make it easier for applications to consume the GraphQL endpoint, we can use the pspring-aws module.

Here are the steps to get it working:

1. Update the variables that are required for appsync-listener in the Terraform variables file dev.vars

2. Deploy the appsync-listener on your AWS infrastructure.

3. Note the apiId/apiUrl/apiKey of the GraphQL API deployed (which is required for consumers).

An example of how to write a Python client to use Secrets Manager as a configuration source and receive real-time events is shown below. The below example uses pspring,pspring-aws,realtime-aws-secretsmngr, appsync-client and few other pypi Python modules for easier implementation. The same can be acheived using Java with any GraphQL client that supports subscriptions.

from pspring import *

import pspringaws

os.environ["region"]="<aws region>"
os.environ["apiId"]="<apiId>"

configProvider = pspringaws.RealTimeSecretsMgrConfigProvider(secretId=<secret-id-here>)

#initialize the configuration
Configuration.initialize([ configProvider ])

#Initialize the application context
ApplicationContext.initialize()

#Get the config object for the name space for current module
config = Configuration.getConfig(__name__)

from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello():
return "Hello "+config.getProperty("firstName")+" "+config.getProperty("lastName")+ "!!"

if __name__ == "__main__":
app.run()



This Flask application will receive real-time changes when the secret is updated in AWS Secrets Manager.

Limitations With AWS Lambda

Note that, although AWS Lambda supports threads, it suspends the spawned threads when no request is being processed. When GraphQL client creates a subscription, a thread is spawned to listen for HTTP messages. This listener thread gets suspended in AWS Lambda. One way to have updated configurations in Lambda without re-deployment is to check the during request process, for time validity. If the current configuration in Lambda is outdated, refetch them again when a request is processed.

Topics:
aws ,configuration ,realtime ,hot ,graphql ,ecs ,lambda ,cloud

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}