DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

The software you build is only as secure as the code that powers it. Learn how malicious code creeps into your software supply chain.

Apache Cassandra combines the benefits of major NoSQL databases to support data management needs not covered by traditional RDBMS vendors.

Generative AI has transformed nearly every industry. How can you leverage GenAI to improve your productivity and efficiency?

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workloads.

Related

  • Container Checkpointing in Kubernetes With a Custom API
  • Leveraging Seekable OCI: AWS Fargate for Containerized Microservices
  • Deploying Dockerized Applications on AWS Lambda: A Step-by-Step Guide
  • Strategic Deployments in AWS: Leveraging IaC for Cross-Account Efficiency

Trending

  • Infrastructure as Code (IaC) Beyond the Basics
  • After 9 Years, Microsoft Fulfills This Windows Feature Request
  • How to Convert XLS to XLSX in Java
  • Advancing Your Software Engineering Career in 2025
  1. DZone
  2. Software Design and Architecture
  3. Cloud Architecture
  4. Render AWS CloudFormation Templates With Docker

Render AWS CloudFormation Templates With Docker

Thinking about using AWS CloudFormation with Docker? Check out this article on how to make it happen!

By 
Dustin Collins user avatar
Dustin Collins
·
Jul. 08, 16 · Tutorial
Likes (1)
Comment
Save
Tweet
Share
4.8K Views

Join the DZone community and get the full member experience.

Join For Free

If your infrastructure runs on AWS and you're not yet using CloudFormation, you should give it a go. CloudFormation (from here on, "CF") is a powerful member of the AWS toolbox that allows you to declare every part of your infrastructure in JSON and "load" it into AWS, which then creates the resources your CF template describes. Come back to this post once you’ve read up on the basics. If you are a regular CF user, read on.

It was the best of formats, it was the worst of formats.

- Charles Dickens (sysadmin, no relation to the author)

Okay, so CF is incredibly powerful. Defining and parameterizing your infrastructure makes it much easier to launch the same resources in different contexts, cutting down on the care and feeding of your environments. Alas, CF has a downside — its templates are written in JSON. JSON is a great format, for machines. Hand-writing long JSON documents is error-prone, tough to verify (visually), and becomes restrictive when you want to reuse common definitions. For example, you can't include JSON files in other JSON files natively. Don't get me started on userdata scripts. This JSON workflow turns into a slog when creating, or updating, anything but the simplest template.

There has to be a better way.

There are several better ways, actually. Here are a few of the more popular tools that make working with CF templates easier.

  • cloudformation-ruby-dsl — Define your templates in Ruby, renders into JSON.
  • Terraform — Define your templates in HCL (HashiCorp format), allows you to plan changes before submitting them.
  • sparkleformation — Integrated CLI + Ruby DSL with lots of helper tools.
  • Troposphere — Define your templates in Python, renders into JSON.

I am using cloudformation-ruby-dsl (from here on, "cfndsl") currently, and I'll cover the workflow I've developed in this post. I use cfndsl now because it maps 1-1 with the best documentation on CloudFormation, the AWS docs. Every resource listed there, and all of its options, are the same with cfndsl. I have also used Terraform and Troposphere. Terraform is neat, but requires more mental gymnastics to look at the AWS docs and then translate into HCL. I like troposphere, for the same reasons I like cfndsl: simple and matches the official AWS docs. If you're in a Python shop, check it out. I have not used sparkleformation, but it looks robust (the docs are extensive).

Alright, so cfndsl it is.

This is Ruby, so I'm supposed to install the cloudformation-ruby-dsl gem. Fair enough, but I work on a lot of different projects, I like to keep them isolated. RVM is okay, but it has enough sharp edges for me to avoid. I also already use Docker daily, so I use it for this too.

My workflow uses three files:

  • template.rb — cfndsl Ruby template describing our CF stack
  • Dockerfile — Ruby 2.2 Docker image that installs the cfndsl gem
  • render.sh — Our dev script. Given a template file template.rb, renders it to stdout and template.rb.json

template.rb

#!/usr/bin/env ruby

require 'cloudformation-ruby-dsl/cfntemplate'

template do
    value :AWSTemplateFormatVersion => '2010-09-09'
    value :Description => 'Jenkins executor autoscaling group'

    parameter 'ImageId',
        :Description => 'Base AMI to launch from',
        :Type => 'String'
    parameter 'DesiredCapacity',
        :Description => 'Desired number of executors to launch',
        :Type => 'String',
        :Default => '1'
    parameter 'InstanceType',
        :Description => 'WebServer EC2 instance type',
        :Type => 'String',
        :Default => 'm3.medium',
        :AllowedValues => %w(m3.medium m3.large m3.xlarge),
        :ConstraintDescription => 'must be a valid EC2 instance type.'

    resource 'ASG', :Type => 'AWS::AutoScaling::AutoScalingGroup', :Properties => {
        :AvailabilityZones => ['us-east-1a'],
        :HealthCheckType => 'EC2',
        :LaunchConfigurationName => ref('LaunchConfig'),
        :DesiredCapacity => ref('DesiredCapacity'),
        :MinSize => 1,
        :MaxSize => 5,
        :Tags => [
        {:Key => 'Name', :Value => 'executor', :PropagateAtLaunch => 'true'}
      ]
    }
    resource 'LaunchConfig', :Type => 'AWS::AutoScaling::LaunchConfiguration', :Properties => {
        :ImageId => ref('ImageId'),
        :InstanceType => ref('InstanceType'),
        :KeyName => 'jenkins-user',
        :SecurityGroups => ['jenkins-executor'],
        :BlockDeviceMappings => [{
            :DeviceName => "/dev/sda1",
            :Ebs => {:VolumeSize => "120"}
        }],
    :UserData => base64(interpolate(file('userdata.sh')))
    }
end.exec!


This is a CloudFormation template written in the DSL defined by cfndsl. This template defines a stack of Jenkins executors in an autoscaling group. userdata.sh is just a bash script in the same directory that runs on the EC2 instances on first boot. cfndsl provides methods like base64, interpolate and file that make it easier to work with CF. Near the bottom of the template, base64(interpolate(file('userdata.sh'))) reads userdata.sh to a string, interpolates it with any variables referenced, and base64-encodes the result (this is the format CF requires for userdata). parameter and resource are methods as well.

The main point here is that cfndsl gives me the full power of Ruby, so I can transclude templates, use maps, loop, pass the template to helper functions to "DSL the DSL" and more. See the full list of functions available, as well as a full template example on the project's GitHub page. This file is executable - chmod +x template.rb.

Dockerfile

FROM ruby:2.2-alpine

ENV CF_RUBYDSL_VERSION 1.2.1

RUN gem install cloudformation-ruby-dsl -v $CF_RUBYDSL_VERSION

WORKDIR /app


Pretty simple Dockerfile: Uses a Ruby 2.2 baseimage, installs a specific version of the cfndsl gem and sets a working directory. The image base is Alpine, much smaller than the ruby:2.2 image. Note that I am not ADDing or COPYing my template into the image itself. Instead, I mount my dev directory directly into the container at path /app. Mounting files into the container means I don’t have to docker build ... and create a new image every time I edit my template.

render.sh

#!/bin/bash -e

TEMPLATE=$1

docker build -q --rm -t cloudformation .
docker run --rm \
    -v $PWD:/app \
    cloudformation \
    ./$TEMPLATE expand | tee $TEMPLATE.json


This bash script takes one argument, the cfndsl template file that I want to render. It then readies the image, starts a container that mounts my dev directory, and renders the template to stdout and <template>.rb.json. Since these files are mounted, the JSON file immediately appears up on my Docker host. This file is executable - chmod +x render.sh.

With those files in place, I run:

$ ./render.sh template.rb
sha256:554c9ed9cf934bc3bcf4dde11401e62280a85825178b09717fd692443822c806
{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "Jenkins executor autoscaling group",
    "Parameters": {
        "ImageId": {
            "Description": "Base AMI to launch from",
            "Type": "String"
            },
        "DesiredCapacity": {
            "Description": "Desired number of executors to launch",
            "Type": "String",
            "Default": "1"
            },
        "InstanceType": {
            "Description": "WebServer EC2 instance type",
            "Type": "String",
            "Default": "m3.medium",
            "AllowedValues": [
                "m3.medium",
                "m3.large",
                "m3.xlarge"
                ],
          "ConstraintDescription": "must be a valid EC2 instance type."
          }
      },
    "Resources": {
      "ASG": {
          "Type": "AWS::AutoScaling::AutoScalingGroup",
          "Properties": {
              "AvailabilityZones": [
                  "us-east-1a"
                  ],
               "HealthCheckType": "EC2",
          "LaunchConfigurationName": {
              "Ref": "LaunchConfig"
              },
          "DesiredCapacity": {
              "Ref": "DesiredCapacity"
              },
          "MinSize": 1,
          "MaxSize": 5,
          "Tags": [
                {
                    "Key": "Name",
                    "Value": "executor",
                    "PropagateAtLaunch": "true"
                    }
                ]
            }
        },
  "LaunchConfig": {
      "Type": "AWS::AutoScaling::LaunchConfiguration",
      "Properties": {
        "ImageId": {
          "Ref": "ImageId"
        },
        "InstanceType": {
          "Ref": "InstanceType"
        },
        "KeyName": "jenkins-user",
        "SecurityGroups": [
          "jenkins-executor"
        ],
        "BlockDeviceMappings": [
          {
            "DeviceName": "/dev/sda1",
            "Ebs": {
              "VolumeSize": "120"
            }
          }
        ],
        "UserData": {
          "Fn::Base64": {
            "Fn::Join": [
              "",
              [
                "#!/bin/bash -e\n",
                "\n",
                "DOCKER_VERSION='1.11.2-0~trusty'\n",
                "\n",
                "echo \"Installing Docker\"\n",
                "apt-get update\n",
                "apt-get install -y apt-transport-https ca-certificates\n",
                "apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D\n",
                "echo \"deb https://apt.dockerproject.org/repo ubuntu-trusty main\" | tee /etc/apt/sources.list.d/docker.list\n",
                "apt-get update\n",
                "apt-get purge lxc-docker || true\n",
                "apt-get install -y linux-image-extra-$(uname -r) apparmor\n",
                "apt-get install -y docker-engine=$DOCKER_VERSION\n",
                "usermod -aG docker ubuntu\n"
              ]
            ]
          }
        }
      }
    }
  }
}


The resulting template is also written to template.rb.json in my current directory. This file can be uploaded, using the AWS CLI or console, directly to CloudFormation to create, or update a stack. The edit/render/upload cycle doesn't take long. I've been using this workflow for about a year and am very happy with it.

That's all for now, thanks for reading! I didn't cover some of the more advanced workflows cfndsl enables in this post, but if you'd like me to cover a specific topic let me know. Advanced topics include:

  • template transclusion.
  • rendering variables and references into userdata.
  • pseudo-parameters.
  • maps and tables.
  • cfndsl CLI usage.
AWS Template Docker (software)

Published at DZone with permission of Dustin Collins, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Container Checkpointing in Kubernetes With a Custom API
  • Leveraging Seekable OCI: AWS Fargate for Containerized Microservices
  • Deploying Dockerized Applications on AWS Lambda: A Step-by-Step Guide
  • Strategic Deployments in AWS: Leveraging IaC for Cross-Account Efficiency

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!