CloudFormation Update: YAML, Cross-Stack References, and Simplified Substitution

In mid-September, AWS released a big update to CloudFormation.

The update contained:

• YAML support: You can now write your CloudFormation templates in YAML.
• Cross stack references: You can now export values from one stack and use them in another.
• Simplified substitution: You can more easily embed variables in strings.

After one month of using the new features, I want to share my learnings with you.

YAML Support

During the last four weeks, I discovered three main advantages of using YAML over JSON to describe my CloudFormation templates:

1. Support for multi-line strings.
2. It is possible to use comments within a template.
3. YAML is more compact than JSON.

Let me explain them in more detail.

Support for Multi-Line Strings

The UserData property usually consists of many lines. In JSON, there was no elegant way to express this. Instead, you used the Fn::Join function of CloudFormation.

{
  "Resources": {
    "MyLC": {
      "Type": "AWS::AutoScaling::LaunchConfiguration",
      "Properties":
        "UserData": {"Fn::Join": ["\n"], [
          "#!/bin/bash -x",
          "/opt/aws/bin/cfn-init -v --stack my-stack --resource MyLC --region eu-west-1",
          "/opt/aws/bin/cfn-signal -e $? --stack my-stac --resource MyASG --region eu-west-1"
        ]}
      }
    }
  }
}

In YAML, this looks much nicer:

---
Resources:
  MyLC:
    Type: 'AWS::AutoScaling::LaunchConfiguration'
    Properties:
      UserData: !Base64 |
        #!/bin/bash -x
        /opt/aws/bin/cfn-init -v --stack my-stack --resource MyLC --region eu-west-1
        /opt/aws/bin/cfn-signal -e $? --stack my-stack --resource MyASG --region eu-west-1

In YAML, you can also add comments.

---
Resources:
  MyLC:
    Type: 'AWS::AutoScaling::LaunchConfiguration'
    Properties:
      # This how a comment looks like
      UserData: !Base64 |
        #!/bin/bash -x
        /opt/aws/bin/cfn-init -v --stack my-stack --resource MyLC --region eu-west-1
        /opt/aws/bin/cfn-signal -e $? --stack my-stack --resource MyASG --region eu-west-1

Nothing fancy, but very helpful.

YAML Is More Compact Than JSON

I converted all our Free Templates for AWS CloudFormation from JSON to YAML. See how the number of lines changed:

On average, the templates get smaller.

Ouch

What I do not like is that there is more than one way to represent strings:

• Sometimes you need quotes; sometimes they are optional.
• Sometimes you need single quotes; sometimes you can use double quotes.

The rules seem to be more complicated than just using double quotes all the time like in JSON. I finally ended up with using single quotes all the time unless the string contains only [a-zA-Z0-9]. However, that is just my personal style. It ends up in a total mess if multiple people work on a single template.

Cross Stack References

Instead of putting everything in a single template, it can make sense to split them up. One very common example is the VPC. You create one CloudFormation stack that contains your VPC. Each application that you run is also a stack, but they depend on the VPC stack. Before Cross Stack References you could solve this problem with Parameters.

Application Template

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Parameters": {
    "Subnet": {
      "Description": "Use Subnet output from vpc stack.",
      "Type": "AWS::EC2::Subnet::Id"
    }
  },
  "Resources": {
    "VirtualMachine": {
      "Type": "AWS::EC2::Instance",
      "Properties": {
        "SubnetId": {"Ref": "Subnet"},
      }
    }
  }
}

With Cross Stack References, you can export values in one stack and import them in another stack. Let’s see how this works.

VPC Template

---
AWSTemplateFormatVersion: '2010-09-09'
Resources:
  VPC:
    Type: 'AWS::EC2::VPC'
    Properties:
      CidrBlock: !Sub '10.0.0.0/16'
  SubnetAPublic:
    Type: 'AWS::EC2::Subnet'
    Properties:
      AvailabilityZone: !Select [0, !GetAZs '']
      CidrBlock: '10.0.0.0/20'
      VpcId: !Ref VPC

Outputs:
  Subnet:
    Description: 'Subnet.'
    Value: !Ref SubnetAPublic
    Export:
      Name: 'vpc-subnet'

The VPC stack now exports the subnet ID under the name vpc-subnet.

Application Template

---
AWSTemplateFormatVersion: '2010-09-09'
Resources:
  VirtualMachine:
    Type: 'AWS::EC2::Instance'
    Properties:
      SubnetId: !ImportValue 'vpc-subnet'

The application stack imports the subnet id.

With Cross Stack References you can pass data from one stack to another.

Ouch

It is not possible to export and import a list of values at the moment. You can export a comma separated string, but you are not able to import that string as a list.

Keep in mind that you can not change exported values.

Let’s again have a look at the UserData property. It usually not only consists of many lines, but it also references some resources. In JSON, I used Fn::Join and Ref for this like crazy:

{
  "Resources": {
    "MyLC": {
      "Type": "AWS::AutoScaling::LaunchConfiguration",
      "Properties":
        "UserData": {"Fn::Join": [""], [
          "#!/bin/bash -x\n",
          "/opt/aws/bin/cfn-init -v --stack ", {"Ref": "AWS::StackName"}, " --resource ", {"Ref": "MyLC"}, " --region ", {"Ref": "AWS::Region"}, "\n",
          "/opt/aws/bin/cfn-signal -e $? --stack ", {"Ref": "AWS::StackName"}, " --resource ", {"Ref": "MyASG"}, " --region ", {"Ref": "AWS::Region"}, "\n"
        ]}
      }
    }
  }
}

In YAML, this looks cleaner:

---
Resources:
  MyLC:
    Type: 'AWS::AutoScaling::LaunchConfiguration'
    Properties:
      UserData:
        'Fn:Base64': !Sub |
          #!/bin/bash -x
          /opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource ${MyLC} --region ${AWS::Region}
          /opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} --resource ${MyASG} --region ${AWS::Region}

So the magic is the Fn::Sub function or shorter !Sub. It substitutes references inside ${} automatically with the value.

Ouch

Unfortunately, it is not possible to write !Base64 !Sub |. The documentation tells us:

If you use the short form and immediately include another function in the valueToEncode parameter, use the full function name for at least one of the functions. For example, the following syntax is invalid:

!Base64 !Sub string

Instead, use the full function name for at least one of the functions, as shown in the following examples:

'Fn::Base64':
  !Sub string

I hope this is fixed shortly.

The CloudFormation update makes our lives easier:

• We can now write the template in YAML which makes them shorter.
• We can add comments and deal with multi-line strings without struggles.
• The simplified substitution removes complex Fn::Joins from our templates.
• We can pass data between stacks with Cross Stack References.

As always, new features have rough edges:

• It is not possible to write !Base64 !Sub |. Instead, we need to write: 'Fn::Base64': !Sub |
• It is not possible to export and import a list values with Cross Stack References.
• YAML and strings are complicated. Many options and many ways to do express the same in different ways.

All in all, I can recommend the new YAML format.

I started with an automatic conversion from JSON to YAML with:

ruby -ryaml -rjson -e 'puts YAML.dump(JSON.parse(STDIN.read))' < template.json > template.yaml

Then, that hard manual work begins when you want to use simplified substitution and YAML multi-line support.