Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

AWS Velocity Series: EC2-Based App CI/CD Pipeline

DZone's Guide to

AWS Velocity Series: EC2-Based App CI/CD Pipeline

EC2-based apps are highly available, are scalable, offer frictionless deployment, are secure, and utilize CloudWatch Logs. They're a lot of work, but they're worth it.

· DevOps Zone
Free Resource

“Automated Testing: The Glue That Holds DevOps Together” to learn about the key role automated testing plays in a DevOps workflow, brought to you in partnership with Sauce Labs.

Most of our clients use AWS to reduce time-to-market following an agile approach. But AWS is only one part of the solution. In this article series, I show you how we help our clients to improve velocity: the time from idea to production.

EC2-Based App CI/CD Pipeline

In the previous article, you learned how to use CloudFormation to describe a production-ready infrastructure for an EC2 based app. In this article you will learn to:

  • Automate the creation of an AMI that contains the app with Packer.
  • Deploy a CloudFormation stack based infrastructure/ec2.yml with AWS CodePipeline.
  • Run the Acceptance tests on AWS CodeBuild against the infrastructure created in the previous step.
  • Deploy another CloudFormation stack for the production environment.

You can follow step-by-step or get the full source code here.

The pipeline is based on the CI/CD Pipeline as Code part of this series.

Additionally, the EC2-based app pipeline contains:

  • Add a BuildAMI action to the Build stage.
  • Add an Acceptance stage.
  • Add a Production stage.

Copy the deploy/pipeline.yml file to deploy/pipeline_ec2.yml to get the starting point right. If you don’t have the deploy/pipeline.yml file, you can download it from here.

BuildAMI Action

EC2 instances start from an image (AMI) that contains the operating system including all the files that are needed. You can create your own AMI as well. Usually, you take one of the available AMIs like the Amazon Linux, make your modifications, and then create a new image from that. This whole procedure can also be automated with a tool called Packer. You will now see how you can run Packer in CodeBuild to create a new AMI that contains the app.

Packer itself needs configuration files. Create a file infrastructure/packer.json with the following content that builds a new AMI based on ami-c51e3eb6 (Amazon Linux) and a Bash script that you will create later:

{
  "variables": {
    "ami_name": "{{env `CODEBUILD_BUILD_ID`}}"
  },
  "builders": [{
    "type": "amazon-ebs",
    "region": "eu-west-1",
    "source_ami": "ami-c51e3eb6",
    "instance_type": "t2.micro",
    "ssh_username": "ec2-user",
    "ami_name": "{{user `ami_name` | clean_ami_name}}",
    "ami_regions": ["REGION"]
  }],
  "provisioners": [{
    "type": "file",
    "source": "app",
    "destination": "/tmp"
  }, {
    "type": "shell",
    "script": "infrastructure/packer.sh"
  }]
}

Packer is configured to run a Bash script to provision the AMI and to upload the app folder. Create a file infrastructure/packer.sh with the following content to:

  • Install the latest patches.
  • Install Node.js 6.x.
  • Install the CloudWatch Logs agent.
  • Install forever, a tool to run Node.js script in the background.
#!/bin/bash -ex
# this script runs as ec2-user
  
sudo yum -y update
curl --silent --location https://rpm.nodesource.com/setup_6.x | sudo bash -
sudo yum -y install nodejs awslogs
sudo npm install -g forever@0.15.3
sudo mv /tmp/app /opt

The script also moves the application files to the right place.

To integrate Packer into the pipeline, add the following resources to the Resources section of deploy/pipeline_ec2.yml to create a CodeBuild project to run Packer with the above configuration. Packer also needs a bunch of IAM permissions which are also added.

# Packer needs a set of access rights as defined in https://www.packer.io/docs/builders/amazon.html
AMICodeBuildRole:
  DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
  Type: 'AWS::IAM::Role'
  Properties:
    AssumeRolePolicyDocument:
      Version: '2012-10-17'
      Statement:
      - Effect: Allow
        Principal:
          Service:
          - 'codebuild.amazonaws.com'
        Action:
        - 'sts:AssumeRole'
    Policies:
    - PolicyName: ServiceRole
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Sid: CloudWatchLogsPolicy
          Effect: Allow
          Action: 
          - 'logs:CreateLogGroup'
          - 'logs:CreateLogStream'
          - 'logs:PutLogEvents'
          Resource: '*'
        - Sid: CodeCommitPolicy
          Effect: Allow
          Action: 'codecommit:GitPull'
          Resource: '*'
        - Sid: S3GetObjectPolicy
          Effect: Allow
          Action: 
          - 's3:GetObject'
          - 's3:GetObjectVersion'
          Resource: '*'
        - Sid: S3PutObjectPolicy
          Effect: 'Allow'
          Action: 's3:PutObject'
          Resource: '*'
    - PolicyName: EC2
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Sid: CloudFormation
          Effect: Allow
          Action: 
          - 'ec2:AttachVolume'
          - 'ec2:AuthorizeSecurityGroupIngress'
          - 'ec2:CopyImage'
          - 'ec2:CreateImage'
          - 'ec2:CreateKeypair'
          - 'ec2:CreateSecurityGroup'
          - 'ec2:CreateSnapshot'
          - 'ec2:CreateTags'
          - 'ec2:CreateVolume'
          - 'ec2:DeleteKeypair'
          - 'ec2:DeleteSecurityGroup'
          - 'ec2:DeleteSnapshot'
          - 'ec2:DeleteVolume'
          - 'ec2:DeregisterImage'
          - 'ec2:DescribeImageAttribute'
          - 'ec2:DescribeImages'
          - 'ec2:DescribeInstances'
          - 'ec2:DescribeRegions'
          - 'ec2:DescribeSecurityGroups'
          - 'ec2:DescribeSnapshots'
          - 'ec2:DescribeSubnets'
          - 'ec2:DescribeTags'
          - 'ec2:DescribeVolumes'
          - 'ec2:DetachVolume'
          - 'ec2:GetPasswordData'
          - 'ec2:ModifyImageAttribute'
          - 'ec2:ModifyInstanceAttribute'
          - 'ec2:ModifySnapshotAttribute'
          - 'ec2:RegisterImage'
          - 'ec2:RunInstances'
          - 'ec2:StopInstances'
          - 'ec2:TerminateInstances'
          Resource: '*'
# This IAM User is only temporarily necessary until https://github.com/mitchellh/packer/pull/4613 is fixed!
AMICodeBuildUser:
  DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
  Type: 'AWS::IAM::User'
  Properties:
    Policies:
    - PolicyName: Packer
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Sid: EC2
          Effect: Allow
          Action:
          - 'ec2:AttachVolume'
          - 'ec2:AuthorizeSecurityGroupIngress'
          - 'ec2:CopyImage'
          - 'ec2:CreateImage'
          - 'ec2:CreateKeypair'
          - 'ec2:CreateSecurityGroup'
          - 'ec2:CreateSnapshot'
          - 'ec2:CreateTags'
          - 'ec2:CreateVolume'
          - 'ec2:DeleteKeypair'
          - 'ec2:DeleteSecurityGroup'
          - 'ec2:DeleteSnapshot'
          - 'ec2:DeleteVolume'
          - 'ec2:DeregisterImage'
          - 'ec2:DescribeImageAttribute'
          - 'ec2:DescribeImages'
          - 'ec2:DescribeInstances'
          - 'ec2:DescribeRegions'
          - 'ec2:DescribeSecurityGroups'
          - 'ec2:DescribeSnapshots'
          - 'ec2:DescribeSubnets'
          - 'ec2:DescribeTags'
          - 'ec2:DescribeVolumes'
          - 'ec2:DetachVolume'
          - 'ec2:GetPasswordData'
          - 'ec2:ModifyImageAttribute'
          - 'ec2:ModifyInstanceAttribute'
          - 'ec2:ModifySnapshotAttribute'
          - 'ec2:RegisterImage'
          - 'ec2:RunInstances'
          - 'ec2:StopInstances'
          - 'ec2:TerminateInstances'
          Resource: '*'
# This IAM Access Key is only temporarily necessary until https://github.com/mitchellh/packer/pull/4613 is fixed!
AMICodeBuildUserAccessKey:
  DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
  Type: 'AWS::IAM::AccessKey'
  Properties:
    UserName: !Ref AMICodeBuildUser
AMIProject:
  DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
  Type: 'AWS::CodeBuild::Project'
  Properties:
    Artifacts:
      Type: CODEPIPELINE
    Environment:
      ComputeType: 'BUILD_GENERAL1_SMALL'
      Image: 'amazonlinux:2016.09'
      Type: 'LINUX_CONTAINER'
      EnvironmentVariables: # pass in the AWS credentials as environment variables is only temporarily necessary until https://github.com/mitchellh/packer/pull/4613 is fixed!
      - Name: 'AWS_ACCESS_KEY_ID'
        Value: !Ref AMICodeBuildUserAccessKey
      - Name: 'AWS_SECRET_ACCESS_KEY'
        Value: !GetAtt 'AMICodeBuildUserAccessKey.SecretAccessKey'
    Name: !Sub '${AWS::StackName}-ami'
    ServiceRole: !GetAtt 'AMICodeBuildRole.Arn'
    Source:
      Type: CODEPIPELINE
      BuildSpec: !Sub |
        version: 0.1
        phases:
          install: # install Packer
            commands:
            - 'yum -y install unzip'
            - 'curl -s -m 60 -o /opt/packer.zip https://releases.hashicorp.com/packer/0.12.3/packer_0.12.3_linux_amd64.zip'
            - 'unzip /opt/packer.zip -d /opt'
          pre_build: # replace the REGION placeholder with the stack's region
            commands:
            - 'sed -i "s:REGION:${AWS::Region}:g" infrastructure/packer.json'
          build: # run packer
            commands:
            - '/opt/packer -machine-readable build infrastructure/packer.json | tee infrastructure/packer.txt'
          post_build: # extract the AMI id into a JSON file
            commands:
            - 'echo "{" > infrastructure/ami.json'
            - 'cat infrastructure/packer.txt | grep '',amazon-ebs,artifact,0,id,'' | awk -F'','' ''{print $6}'' | sed ''s/%!(PACKER_COMMA)/\''$''\n/g'' | awk -F'':'' ''{print "\"image\": \""$2"\","}'' | sed ''$ s/.$//'' >> infrastructure/ami.json'
            - 'echo "}" >> infrastructure/ami.json'
        artifacts:
          files:
          - 'infrastructure/packer.txt'
          - 'infrastructure/ami.json'
    TimeoutInMinutes: 10

You also need to make a small modification to the exiting BuildSpec in the AppProject resource to include the packer files into the App artifact. This is a hack because CodeBuild does not support multiple input artifacts at the moment. Change the artifacts section to:

artifacts:
  files:
  - 'app/**/*'
  - 'infrastructure/packer.json' # this is a hack because we can not pass multiple Arifacts as an input to CodeBuild at the moment
  - 'infrastructure/packer.sh' # this is a hack because we can not pass multiple Arifacts as an input to CodeBuild at the moment

Now the CodeBuild project needs to be called in the pipeline, therefore change the Pipeline resource in the file deploy/pipeline_ec2.yml and add a new build action:

Pipeline:
  Type: 'AWS::CodePipeline::Pipeline'
  Properties:
    # [...]
      - Name: BuildAndTestAcceptance
        ActionTypeId:
          Category: Build
          Owner: AWS
          Provider: CodeBuild
          Version: 1
        Configuration:
          ProjectName: !Ref AcceptanceProject
        InputArtifacts:
        - Name: Source
        OutputArtifacts:
        - Name: Acceptance
        RunOrder: 1
      # NEW STUFF!
      - Name: BuildAMI
        ActionTypeId:
          Category: Build
          Owner: AWS
          Provider: CodeBuild
          Version: 1
        Configuration:
          ProjectName: !Ref AMIProject
        InputArtifacts:
        - Name: App
        OutputArtifacts:
        - Name: AMI
        RunOrder: 2

Now, a new AMI is automatically created whenever the pipeline runs. The AMI will include the app and the latest patches.

It’s time to deploy the app to the acceptance stage and to see if the app works.

Acceptance Stage

The acceptance stage consists of a CloudFormation stack based on infrastructure/ec2.yml and the execution of the acceptance tests. To create the CloudFormation stack, you first have to provide a few parameters. Create a file infrastructure/ec2.json with the following content:

{
  "Parameters": {
    "ImageId": {"Fn::GetParam": ["AMI", "infrastructure/ami.json", "image"]},
    "ParentVPCStack": "vpc-2azs",
    "ParentSSHBastionStack": "",
    "KeyName": "",
    "AdminEmail": "your@email.com"
  }
}

If you don’t have a VPC stack based on our free templates for AWS CloudFormation, create a VPC stack first. Make sure to change the ParentVPCStack parameter in the infrastructure/ec2.yml accordingly. Also, change the value of the AdminEmail parameter. The other values can stay as they are. Look at the ImageId parameter value. This is the way of getting a value out of a JSON artifact file in CoePipeline.

To run the acceptance tests, you also need another CodeBuild project. Add the following resources to the Resources section of deploy/pipeline_ec2.yml:

ExtendedCodeBuildRole:
  DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
  Type: 'AWS::IAM::Role'
  Properties:
    AssumeRolePolicyDocument:
      Version: '2012-10-17'
      Statement:
      - Effect: Allow
        Principal:
          Service:
          - 'codebuild.amazonaws.com'
        Action:
        - 'sts:AssumeRole'
    Policies:
    - PolicyName: ServiceRole
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Sid: CloudWatchLogsPolicy
          Effect: Allow
          Action: 
          - 'logs:CreateLogGroup'
          - 'logs:CreateLogStream'
          - 'logs:PutLogEvents'
          Resource: '*'
        - Sid: CodeCommitPolicy
          Effect: Allow
          Action: 'codecommit:GitPull'
          Resource: '*'
        - Sid: S3GetObjectPolicy
          Effect: Allow
          Action: 
          - 's3:GetObject'
          - 's3:GetObjectVersion'
          Resource: '*'
        - Sid: S3PutObjectPolicy
          Effect: 'Allow'
          Action: 's3:PutObject'
          Resource: '*'
    - PolicyName: CloudFormation
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Sid: CloudFormation
          Effect: Allow
          Action: 
          - 'cloudformation:DescribeStacks'
          Resource: !Sub 'arn:aws:cloudformation:${AWS::Region}:${AWS::AccountId}:stack/${AWS::StackName}-acceptance/*'
RunAcceptanceProject:
  DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
  Type: 'AWS::CodeBuild::Project'
  Properties:
    Artifacts:
      Type: CODEPIPELINE
    Environment:
      ComputeType: 'BUILD_GENERAL1_SMALL'
      Image: 'aws/codebuild/nodejs:6.3.1'
      Type: 'LINUX_CONTAINER'
    Name: !Sub '${AWS::StackName}-run-acceptance'
    ServiceRole: !GetAtt 'ExtendedCodeBuildRole.Arn'
    Source:
      Type: CODEPIPELINE
      BuildSpec: !Sub |
        version: 0.1
        phases:
          build: # execute acceptance tests against the acceptance stack
            commands:
            - 'cd acceptance/ && ENDPOINT=`aws cloudformation describe-stacks --stack-name ${AWS::StackName}-acceptance --query "Stacks[0].Outputs[?OutputKey==''URL''].OutputValue" --output text` ./node_modules/jasmine-node/bin/jasmine-node .'
    TimeoutInMinutes: 10

Now the CodeBuild project needs to be called in the pipeline, therefore change Pipeline resource in the file deploy/pipeline_ec2.yml to:

  1. Deploy the CloudFormation stack suffixed with -acceptance
  2. Run the acceptance tests.
Pipeline:
  Type: 'AWS::CodePipeline::Pipeline'
  Properties:
    # [...]
      - Name: BuildAMI
        ActionTypeId:
          Category: Build
          Owner: AWS
          Provider: CodeBuild
          Version: 1
        Configuration:
          ProjectName: !Ref AMIProject
        InputArtifacts:
        - Name: App
        OutputArtifacts:
        - Name: AMI
        RunOrder: 2
    # NEW STUFF!
    - Name: Acceptance
      Actions:
      - Name: Deploy
        ActionTypeId:
          Category: Deploy
          Owner: AWS
          Provider: CloudFormation
          Version: 1
        Configuration:
          ActionMode: CREATE_UPDATE
          Capabilities: CAPABILITY_IAM
          RoleArn: !GetAtt 'CloudFormationRole.Arn'
          StackName: !Sub '${AWS::StackName}-acceptance'
          TemplatePath: 'Source::infrastructure/ec2.yml'
          TemplateConfiguration: 'Source::infrastructure/ec2.json'
        InputArtifacts:
        - Name: Source
        - Name: AMI
        RunOrder: 1
      - Name: Test
        ActionTypeId:
          Category: Build
          Owner: AWS
          Provider: CodeBuild
          Version: 1
        Configuration:
          ProjectName: !Ref RunAcceptanceProject
        InputArtifacts:
        - Name: Acceptance
        RunOrder: 2

The acceptance stage is now ready.

Production Stage

The production stage is pretty simple, just one CloudFormation stack. Change the Pipeline resource in the file deploy/pipeline_ec2.yml to add a new stage that looks familiar to the acceptance stage:

Pipeline:
  Type: 'AWS::CodePipeline::Pipeline'
  Properties:
    # [...]
      - Name: Test
        ActionTypeId:
          Category: Build
          Owner: AWS
          Provider: CodeBuild
          Version: 1
        Configuration:
          ProjectName: !Ref RunAcceptanceProject
        InputArtifacts:
        - Name: Acceptance
        RunOrder: 2
    # NEW STUFF!  
    - Name: Production
      Actions:
      - Name: Deploy
        ActionTypeId:
          Category: Deploy
          Owner: AWS
          Provider: CloudFormation
          Version: 1
        Configuration:
          ActionMode: CREATE_UPDATE
          Capabilities: CAPABILITY_IAM
          RoleArn: !GetAtt 'CloudFormationRole.Arn'
          StackName: !Sub '${AWS::StackName}-production'
          TemplatePath: 'Source::infrastructure/ec2.yml'
          TemplateConfiguration: 'Source::infrastructure/ec2.json'
        InputArtifacts:
        - Name: Source
        - Name: AMI
        RunOrder: 1

Now the AMI containing the application is deployed to production with confidence and without disturbing the users. Try it and run the pipeline!

Summary

Let’s use my production ready-definition to summarize how each point is implemented.

Highly Available

The load balancer (which is HA) sits in front of a fleet of EC2 instances managed by the Auto Scaling Group for maximum availability. In the case of an unhealthy instance, the Auto Scaling Group will replace that instance.

Scalable

If the CPU utilization gets over 70%, a Cloud Watch Alarm triggers a Scaling Policy to add new instances automatically.

Frictionless Deployment

To deploy a new version of the app, a new AMI is created. This AMI is then rolled out to the acceptance environment by updating the CloudFormation stack with the new ImageId parameter. CloudFormation and the Auto Scaling Group perform a rolling update to avoid the application being down during deployment. If the application can not be started the Rolling Update fails and CloudFormation rolls back.

Secure

During AMI creation, the latest patches are applied. You must ensure that the pipeline runs often enough to keep up with new patches. Besides that, Security Groups control network traffic to the EC2 instances. When following the bastion host approach, you get maximum security. The EC2 instance is only allowed to send logs to CloudWatch Logs by following the least privileges approach.

Operations

All logs are stored in CloudWatch Logs, important metrics are monitored and alarms are defined.

If you now have the impression that running an app on EC2 is a lot of work, you are right. In the next two articles, you will learn about other options with fewer responsibilities.

Learn about the importance of automated testing as part of a healthy DevOps practice, brought to you in partnership with Sauce Labs.

Topics:
devops ,aws ,continuous integration ,continuous delivery ,pipeline ,ec2 ,app development

Published at DZone with permission of Michael Wittig, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}