AWS Velocity Series: Containerized ECS-Based App Infrastructure

Most of our clients use AWS to reduce time-to-market following an Agile approach — but AWS is only one part of the solution. In this article series, I show you how we help our clients improve velocity: the time from idea to production.

Containerized ECS-Based App Infrastructure

EC2 Container Service (ECS) is a highly scalable, fast container management service that makes it easy to run, stop, and manage Docker containers on a cluster of Amazon EC2 instances. To run an application on ECS, you need the following components:

• Docker image.
• ECS cluster: EC2 instances running Docker and the ECS agent.
• ECS service: Managed Docker containers on the ECS cluster.

ECS provides the ability to run a production-ready application on EC2 with reduced responsibilities and increased deployment speed compared to the EC2 based approach. By production-ready, I mean:

• Highly available: No single point of failure.
• Scalable: Increase or decrease the number of instances/containers based on load.
• Frictionless deployment: Deliver new versions of your application automatically without downtime.
• Secure: Patching operating systems and libraries frequently, follow the least privilege principle in all areas.
• Operations: Provide tools like logging, monitoring and alerting to recognize and debug problems.

The overall architecture will consist of a load balancer, forwarding requests to containers running on multiple EC2 instances distributed among different availability zones (data centers).

This diagram was created with Cloudcraft.

Let’s start with the needed infrastructure: the ECS cluster and the service.

ECS Cluster

The ECS cluster is a fleet of EC2 instances with the ECS agent and Docker installed. The ECS cluster is responsible for scheduling the work (containers) to the EC2 instances.

This diagram was created with Cloudcraft.

Your Docker containers will run on those EC2 instances. You don’t need to care about the ECS cluster too much. We provide you a free and production-ready CloudFormation template. Please set up the ECS cluster now if you want to setup the scenario. AWS charges will likely occur! The ECS cluster needs to run in a VPC, so if you don’t have a VPC stack based on our free templates for AWS CloudFormation create a VPC stack first.

This first step was easy. Now, you learn to use the ECS cluster with ECS services.

ECS Service

The ECS service is responsible for launching Docker containers in the cluster. The service also makes sure that failed containers are replaced, and it also takes care about performing rolling updates for you. You also need a load balancer to route traffic to the containers. The following ECS service template is based on free and production-ready CloudFormation template.

Load Balancer

You can follow step by step or get the full source code here.

Create a file infrastructure/ecs.yml. The first part of the file contains the load balancer. To fully describe an application load bBalancer, you need:

• A security group that allows traffic on port 80.
• The application load balancer itself.
• A target group, which is a group of EC2 instances. Each of them runs containers that can receive traffic from the load balancer.
• A listener, which wires the load balancer together with the target group and defines the listening port.

Watch out for comments with more detailed information in the code.

AWSTemplateFormatVersion: '2010-09-09'
Description: 'ECS: service that runs on an ECS cluster based on ecs/cluster.yaml and uses a dedicated ALB, a cloudonaut.io template'
Parameters:
  # You can reuse a VPC for multiple applications. In this case, we use one of our Free Templates for AWS CloudFormation (https://github.com/widdix/aws-cf-templates/tree/master/vpc).
  ParentVPCStack:
    Description: 'Stack name of parent VPC stack based on vpc/vpc-*azs.yaml template.'
    Type: String
  ParentClusterStack:
    Description: 'Stack name of parent Cluster stack based on ecs/cluster.yaml template.'
    Type: String
Resources:
  # Allow traffic from the load balancer to the EC2 instances in the cluster. This is only necessary because we reuse the cluster template in the way it is!
  SecurityGroupInALB:
    Type: 'AWS::EC2::SecurityGroupIngress'
    Properties:
      GroupId:
        'Fn::ImportValue': !Sub '${ParentClusterStack}-SecurityGroup'
      IpProtocol: tcp
      FromPort: 0
      ToPort: 65535
      SourceSecurityGroupId: !Ref ALBSecurityGroup
  # The load balancer accepts HTTP traffic. Therefore the firewall must allow incoming traffic on port 80.
  ALBSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupDescription: 'ecs-cluster-alb'
      VpcId:
        'Fn::ImportValue': !Sub '${ParentVPCStack}-VPC'
      SecurityGroupIngress:
      - CidrIp: '0.0.0.0/0'
        FromPort: 80
        ToPort: 80
        IpProtocol: tcp
  # The load balancer needs to run in public subnets because our users should be able to access the app from the Internet.
  LoadBalancer:
    Type: 'AWS::ElasticLoadBalancingV2::LoadBalancer'
    Properties:
      Scheme: 'internet-facing'
      SecurityGroups:
      - !Ref ALBSecurityGroup
      Subnets: 
      - 'Fn::ImportValue': !Sub '${ParentVPCStack}-SubnetAPublic'
      - 'Fn::ImportValue': !Sub '${ParentVPCStack}-SubnetBPublic'
  # A target group groups a bunch of backend instances that receive traffic from the load balancer. the health check ensures that only working backends are used.
  DefaultTargetGroup:
    Type: 'AWS::ElasticLoadBalancingV2::TargetGroup'
    Properties:
      HealthCheckIntervalSeconds: 15
      HealthCheckPath: '/1'
      HealthCheckProtocol: HTTP
      HealthCheckTimeoutSeconds: 10
      HealthyThresholdCount: 2
      Matcher:
        HttpCode: '200'
      Port: 80
      Protocol: HTTP
      UnhealthyThresholdCount: 4
      VpcId:
        'Fn::ImportValue': !Sub '${ParentVPCStack}-VPC'
  # The load balancer should listen on port 80 for HTTP traffic
  HttpListener:
    Type: 'AWS::ElasticLoadBalancingV2::Listener'
    Properties:
      DefaultActions:
      - TargetGroupArn: !Ref DefaultTargetGroup
        Type: forward
      LoadBalancerArn: !Ref LoadBalancer
      Port: 80
      Protocol: HTTP
# A CloudFormation stack can return information that is needed by other stacks or scripts.
Outputs:
  DNSName:
    Description: 'The DNS name for the ECS cluster/service load balancer.'
    Value: !GetAtt 'LoadBalancer.DNSName'
    Export:
      Name: !Sub '${AWS::StackName}-DNSName'
  URL:
    Description: 'URL to the ECS service.'
    Value: !Sub 'http://${LoadBalancer.DNSName}'
    Export:
      Name: !Sub '${AWS::StackName}-URL'

But how do you get notified if something goes wrong? Let’s add a parameter to the Parameters section to make the receiver configurable:

AdminEmail:
  Description: 'The email address of the admin who receives alerts.'
  Type: String

Alerts are triggered by a CloudWatch Alarm which can send an alert to an SNS topic. You can subscribe to this topic via an email address to receive the alerts. Let’s create an SNS topic and two alarms in the Resourcessection:

# A SNS topic is used to send alerts via Email to the value of the AdminEmail parameter 
  Alerts:
    Type: 'AWS::SNS::Topic'
    Properties:
      Subscription:
      - Endpoint: !Ref AdminEmail
        Protocol: email
# This alarm is triggered, if the load balancer responds with 5XX status codes
  LoadBalancer5XXAlarm:
    Type: 'AWS::CloudWatch::Alarm'
    Properties:
      EvaluationPeriods: 1
      Statistic: Sum
      Threshold: 0
      AlarmDescription: 'Load balancer responds with 5XX status codes.'
      Period: 60
      AlarmActions:
      - !Ref Alerts
      Namespace: 'AWS/ApplicationELB'
      Dimensions:
      - Name: LoadBalancer
        Value: !GetAtt 'LoadBalancer.LoadBalancerFullName'
      ComparisonOperator: GreaterThanThreshold
      MetricName: HTTPCode_ELB_5XX_Count
  # This alarm is triggered, if the backend responds with 5XX status codes
  LoadBalancerTargetGroup5XXAlarm:
    Type: 'AWS::CloudWatch::Alarm'
    Properties:
      EvaluationPeriods: 1
      Statistic: Sum
      Threshold: 0
      AlarmDescription: 'Load balancer target responds with 5XX status codes.'
      Period: 60
      AlarmActions:
      - !Ref Alerts
      Namespace: 'AWS/ApplicationELB'
      Dimensions:
      - Name: LoadBalancer
        Value: !GetAtt 'LoadBalancer.LoadBalancerFullName'
      ComparisonOperator: GreaterThanThreshold
      MetricName: HTTPCode_Target_5XX_Count

Let’s recap what you implemented: A load balancer with a firewall rule that allows traffic on port 80. In the case of 5XX status codes, you will receive an email. But the load balancer alone is not enough. Now, it’s time to add the ECS service.

I already talked about the ECS service. It will take care of your containers. To be more precise, it will take care of your tasks that run in the ECS cluster. One task can contain one or multiple Docker containers. Three resources are needed:

• A Task Definition that describes the Docker containers (similar to a Docker Compose file, or a Kubernetes Deployment).
• An IAM Role for your container, so you don’t need to pass in static credentials when you want to interact with AWS from within your container. If you don’t want to make AWS API calls from your container the role is not needed.
• The ECS Service itself.

To make things parameterizable, you also need to add a few parameters to the Parameters section:

# Where does this Docker image comes from? It will be created in the pipeline!
Image:
  Description: 'The image to use for a container, which is passed directly to the Docker daemon. You can use images in the Docker Hub registry or specify other repositories (repository-url/image:tag).'
  Type: String
DesiredCount:
  Description: 'The number of simultaneous tasks, which you specify by using the TaskDefinition property, that you want to run on the cluster.'
  Type: Number
  Default: 2
  ConstraintDescription: 'Must be >= 1'
  MinValue: 1
MaxCapacity:
  Description: 'The maximum number of simultaneous tasks, that you want to run on the cluster.'
  Type: Number
  Default: 4
  ConstraintDescription: 'Must be >= 1'
  MinValue: 1
MinCapacity:
  Description: 'The minimum number of simultaneous tasks, that you want to run on the cluster.'
  Type: Number
  Default: 2
  ConstraintDescription: 'Must be >= 1'
  MinValue: 1

Now, you can describe the resources in the CloudFormation template.

TaskDefinition:
  Type: 'AWS::ECS::TaskDefinition'
  Properties: 
    Family: !Ref 'AWS::StackName'
    NetworkMode: bridge
    ContainerDefinitions:
    - Name: main
      Image: !Ref Image # This is where the Docker image is configured
      Memory: 128
      PortMappings:
      - ContainerPort: 3000 # The image exposes the app on port 3000
        Protocol: tcp
      Essential: true
      LogConfiguration:
        LogDriver: awslogs
        Options:
          'awslogs-region': !Ref 'AWS::Region'
          'awslogs-group':
            'Fn::ImportValue': !Sub '${ParentClusterStack}-LogGroup'
          'awslogs-stream-prefix': !Ref 'AWS::StackName'
# The role is using the managed policy AmazonEC2ContainerServiceRole
ServiceRole:
  Type: 'AWS::IAM::Role'
  Properties:
    ManagedPolicyArns:
    - 'arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole'
    AssumeRolePolicyDocument:
      Version: '2008-10-17'
      Statement:
      - Effect: Allow
        Principal:
          Service: 'ecs.amazonaws.com'
        Action: 'sts:AssumeRole'
Service:
  Type: 'AWS::ECS::Service'
  DependsOn: HttpListener
  Properties:
    Cluster:
      'Fn::ImportValue': !Sub '${ParentClusterStack}-Cluster'
    DeploymentConfiguration: # This is the configuration for the rolling update
      MaximumPercent: 200
      MinimumHealthyPercent: 50
    DesiredCount: !Ref DesiredCount
    LoadBalancers:
    - ContainerName: main
      ContainerPort: 3000 # The image exposes the app on port 3000
      TargetGroupArn: !Ref DefaultTargetGroup
    Role: !GetAtt 'ServiceRole.Arn'
    TaskDefinition: !Ref TaskDefinition

Let’s recap what you implemented: A Task definition to define the containers that are managed by the service, an IAM role that is accessible inside the containers, and the ECS service that used the task definition to launch tasks (a bunch of containers) in the cluster. Logs from the containers are already shipped to CloudWatch Logs by the awslogs log driver and are visible in the Log Group that is part of the ECS cluster template.

Two things are missing:

1. Scalability of containers.
2. Alerting if containers have issues.

Let’s tackle those issues step by step.

Auto-Scaling of Containers

If you auto-scale the number of containers, your ECS cluster must be able to auto-scale, as well. If you use our free template on GitHub, the cluster will auto-scale, as well.

Auto-scaling works similar compared to the EC2-based approach. To scale based on the load, you need to add:

• Scaling policies to define what should happen if the system should scale up and down.
• CloudWatch alarms to trigger a scaling policy based on a metric such as CPU utilization.
• Additionally, you need a so-called Scalable Target. The Scalable Target can be an ECS service, but it could also be a Spot Fleet or an EMR Instance Group.

Again, you have to add those resources to the Resources section of your template:

ScalableTargetRole: # based on http://docs.aws.amazon.com/AmazonECS/latest/developerguide/autoscale_IAM_role.html
  Type: 'AWS::IAM::Role'
  Properties:
    AssumeRolePolicyDocument:
      Version: '2012-10-17'
      Statement:
      - Effect: Allow
        Principal:
          Service: 'application-autoscaling.amazonaws.com'
        Action: 'sts:AssumeRole'
    Path: '/'
    Policies:
    - PolicyName: ecs
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Action:
          - 'ecs:DescribeServices'
          - 'ecs:UpdateService'
          Resource: '*'
    - PolicyName: cloudwatch
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Action:
          - 'cloudwatch:DescribeAlarms'
          Resource: '*'
ScalableTarget:
  Type: 'AWS::ApplicationAutoScaling::ScalableTarget'
  Properties:
    MaxCapacity: !Ref MaxCapacity
    MinCapacity: !Ref MinCapacity
    ResourceId: !Sub
    - 'service/${Cluster}/${Service}'
    - Cluster:
        'Fn::ImportValue': !Sub '${ParentClusterStack}-Cluster'
      Service: !GetAtt 'Service.Name'
    RoleARN: !GetAtt 'ScalableTargetRole.Arn'
    ScalableDimension: 'ecs:service:DesiredCount'
    ServiceNamespace: ecs
ScaleUpPolicy:
  Type: 'AWS::ApplicationAutoScaling::ScalingPolicy'
  Properties:
    PolicyName: !Sub '${AWS::StackName}-scale-up'
    PolicyType: StepScaling
    ScalingTargetId: !Ref ScalableTarget
    StepScalingPolicyConfiguration: 
      AdjustmentType: PercentChangeInCapacity
      Cooldown: 300
      MinAdjustmentMagnitude: 1
      StepAdjustments:
      - MetricIntervalLowerBound: 0
        ScalingAdjustment: 25
ScaleDownPolicy:
  Type: 'AWS::ApplicationAutoScaling::ScalingPolicy'
  Properties:
    PolicyName: !Sub '${AWS::StackName}-scale-down'
    PolicyType: StepScaling
    ScalingTargetId: !Ref ScalableTarget
    StepScalingPolicyConfiguration: 
      AdjustmentType: PercentChangeInCapacity
      Cooldown: 300
      MinAdjustmentMagnitude: 1
      StepAdjustments:
      - MetricIntervalLowerBound: 0
        ScalingAdjustment: -25
CPUUtilizationHighAlarm:
  Type: 'AWS::CloudWatch::Alarm'
  Properties:
    AlarmDescription: 'Service is running out of CPU'
    Namespace: 'AWS/ECS'
    Dimensions:
    - Name: ClusterName
      Value:
        'Fn::ImportValue': !Sub '${ParentClusterStack}-Cluster'
    - Name: ServiceName
      Value: !GetAtt 'Service.Name'
    MetricName: CPUUtilization
    ComparisonOperator: GreaterThanThreshold
    Statistic: Average
    Period: 60
    EvaluationPeriods: 1
    Threshold: 70
    AlarmActions:
    - !Ref ScaleUpPolicy
CPUUtilizationLowAlarm:
  Type: 'AWS::CloudWatch::Alarm'
  Properties:
    AlarmDescription: 'Service is wasting CPU'
    Namespace: 'AWS/ECS'
    Dimensions:
    - Name: ClusterName
      Value:
        'Fn::ImportValue': !Sub '${ParentClusterStack}-Cluster'
    - Name: ServiceName
      Value: !GetAtt 'Service.Name'
    MetricName: CPUUtilization
    ComparisonOperator: LessThanThreshold
    Statistic: Average
    Period: 60
    EvaluationPeriods: 1
    Threshold: 30
    AlarmActions:
    - !Ref ScaleDownPolicy

The number of tasks is now increased if the CPU utilization of the service goes above 70%, while the number of tasks is decreased if the CPU utilization falls below 30%.

Monitoring

Last but not least, you have to add CloudWatch Alarms to get alerted if something is wrong with your service. In the Resources section:

# Sends an alert if the average CPU load of the past 5 minutes is higher than 85%
CPUTooHighAlarm:
  Type: 'AWS::CloudWatch::Alarm'
  Properties:
    AlarmDescription: 'Service is running out of CPU'
    Namespace: 'AWS/ECS'
    Dimensions:
    - Name: ClusterName
      Value:
        'Fn::ImportValue': !Sub '${ParentClusterStack}-Cluster'
    - Name: ServiceName
      Value: !GetAtt 'Service.Name'
    MetricName: CPUUtilization
    ComparisonOperator: GreaterThanThreshold
    Statistic: Average
    Period: 300
    EvaluationPeriods: 1
    Threshold: 85
    AlarmActions:
    - !Ref Alerts

The infrastructure is ready now. In the next week, you will learn how to setup the CI/CD pipeline to deploy the ECS-based app.