{{announcement.body}}
{{announcement.title}}

How to Attach an AWS EBS Storage Volume to Your Docker Container

DZone 's Guide to

How to Attach an AWS EBS Storage Volume to Your Docker Container

In this article, see how to attach an AWS EBS storage volume to your Docker container.

· Cloud Zone ·
Free Resource

In an ideal world, Docker containers should be ephemeral without any reliance on external storage. In the microservice world, this is achievable when services are connecting to external databases, queues, and other services.

Sometimes though, we need persistent storage, when we're running services such as Jenkins, Prometheus, or Postgres. Fortunately, there's a straightforward way to set this up now for our ECS Clusters using Docker volume drivers. In this article, you'll learn how to attach EBS volumes to your ECS Tasks, which detach and reattach automatically when your ECS Task gets restarted.

Overview of Volumes in ECS

By default, when you run an ECS Task it's going to have an area of storage on the host that's running it. This host is known as the ECS Container Instance, and is in actual fact an EC2 instance. This is fine for temporary data, but as soon as our ECS Task restarts we lose the data. What we need is a way to connect to external storage, such as AWS EBS or AWS EFS.

With Docker volume plugins (also known as volume drivers), such as REX-Ray, we can now achieve this. The REX-Ray plugin can configure AWS services, such as creating volumes and attaching volumes to EC2 instances.

As you can see in the diagram below, if we have an ECS Task running on an EC2 Instance, then the volume (e.g. EBS) needs to be attached to that instance:

virtual private cloud

REX-Ray takes care of all of this for us, and also specifically can manage:

  • creating the volume if it doesn't already exist, including configuring volume type and size
  • making sure our Docker container/ECS Task is mounted with the volume
  • detaching re-attaching the volume when the ECS Task moves from one EC2 instance to another

ECS launch types

ECS has the EC2 and Fargate launch types. With EC2 you are responsible for provisioning the underlying EC2 instances on which your ECS Tasks will be deployed. With Fargate, you just have to specify the CPU and memory requirements, then AWS provisions everything needed to run your ECS Task.

It's worth noting that you can only use persistent storage with the EC2 launch type, not with Fargate. That's why in this article we will only be considering the EC2 launch type.

Setting up a Persistent Docker Volume: A Working Example

You can follow along with this example, where we'll:

  1. create an ECS Cluster built on top of 2 EC2 instances. The REX-Ray docker plugin will be installed on both of the instances.
  2. create an ECS Task definition for the Postgres database. The task definition will include the Docker volume configuration required to use the REX-Ray volume driver to attach a new EBS volume.
  3. launch the ECS Service for our ECS Task, which will deploy to one of our EC2 instances
  4. connect to our Postgres container, and create some data in a new database
  5. move the ECS Task from one EC2 instance to the other, which will restart the task
  6. connect to Postgres again, and see that data has persisted

You'll need access to the AWS Console and AWS CLI to complete this example.

Provisioning an ECS Cluster

First up, we're going to create an ECS Cluster built on two ECS Container Instances (EC2 instances), provisioned by an AutoScalingGroup. The CloudFormation template below contains everything you need.

Since it's a rather large template, in particular, pay attention to the following parts which are specific to the fact that we're using volumes:

  1. When each of our ECS Container Instances is launched, docker plugin install rexray/ebs is run to install the required REX-Ray plugin (see UserData in ContainerInstances).
  2. The IAM Role attached to our EC2 instances has permissions which include ec2:CreateVolume, ec2:DeleteVolume, and ec2:DetachVolume. This allows the REX-Ray volume driver to manage the EBS volumes (see EC2Role).
Shell
 




x
209


 
1
AWSTemplateFormatVersion: "2010-09-09"
2
Parameters:
3
  VPCID:
4
    Type: String
5
  SubnetId:
6
    Type: String
7
  InstanceType:
8
    Type: String
9
    Default: t2.small
10
  ECSAMI:
11
    Description: AMI ID
12
    Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
13
    Default: "/aws/service/ecs/optimized-ami/amazon-linux/recommended/image_id"
14
  KeyName:
15
    Type: String
16
  AllowedCIDRIp:
17
    Type: String
18
    Default: 0.0.0.0/0
19
Resources:
20
  ECSCluster:
21
    Type: AWS::ECS::Cluster
22
    Properties:
23
      ClusterName: docker-volume-demo
24
  ECSAutoScalingGroup:
25
    Type: AWS::AutoScaling::AutoScalingGroup
26
    Properties:
27
      AvailabilityZones:
28
        - Fn::Select:
29
            - 0
30
            - Fn::GetAZs:
31
                Ref: AWS::Region
32
      VPCZoneIdentifier:
33
        - Ref: SubnetId
34
      LaunchConfigurationName:
35
        Ref: ContainerInstances
36
      MinSize: 2
37
      MaxSize: 2
38
      DesiredCapacity: 2
39
      Tags:
40
        - Key: Name
41
          Value: ECS host
42
          PropagateAtLaunch: true
43
    CreationPolicy:
44
      ResourceSignal:
45
        Timeout: PT15M
46
    UpdatePolicy:
47
      AutoScalingRollingUpdate:
48
        MinInstancesInService: 1
49
        MaxBatchSize: 1
50
        PauseTime: PT15M
51
        WaitOnResourceSignals: true
52
        SuspendProcesses:
53
          - HealthCheck
54
          - ReplaceUnhealthy
55
          - AZRebalance
56
          - AlarmNotification
57
          - ScheduledActions
58
  InstanceSecurityGroup:
59
    Type: AWS::EC2::SecurityGroup
60
    Properties:
61
      VpcId:
62
        Ref: VPCID
63
      GroupDescription: Enable SSH access via port 22
64
      SecurityGroupIngress:
65
        - IpProtocol: tcp
66
          FromPort: 22
67
          ToPort: 22
68
          CidrIp: !Ref AllowedCIDRIp
69
        - IpProtocol: tcp
70
          FromPort: 5432
71
          ToPort: 5432
72
          CidrIp: !Ref AllowedCIDRIp
73
  ContainerInstances:
74
    Type: AWS::AutoScaling::LaunchConfiguration
75
    Properties:
76
      ImageId:
77
        Ref: ECSAMI
78
      InstanceType:
79
        Ref: InstanceType
80
      IamInstanceProfile:
81
        Ref: EC2InstanceProfile
82
      KeyName:
83
        Ref: KeyName
84
      AssociatePublicIpAddress: true
85
      SecurityGroups:
86
        - Ref: InstanceSecurityGroup
87
      UserData:
88
        Fn::Base64:
89
          Fn::Sub: "#!/bin/bash\nyum install -y aws-cfn-bootstrap\n/opt/aws/bin/cfn-init
90
            -v --region ${AWS::Region} --stack ${AWS::StackName} --resource ContainerInstances\n/opt/aws/bin/cfn-signal
91
            -e $? --region ${AWS::Region} --stack ${AWS::StackName} --resource ECSAutoScalingGroup\n\nexec
92
            2>>/var/log/ecs/ecs-agent-install.log\nset -x\nuntil curl -s http://localhost:51678/v1/metadata\ndo\n
93
            \  sleep 1\ndone\ndocker plugin install rexray/ebs REXRAY_PREEMPT=true
94
            EBS_REGION=us-west-2 --grant-all-permissions\nstop ecs \nstart ecs\n"
95
    Metadata:
96
      AWS::CloudFormation::Init:
97
        config:
98
          packages:
99
            yum:
100
              aws-cli: []
101
              jq: []
102
              ecs-init: []
103
          commands:
104
            01_add_instance_to_cluster:
105
              command:
106
                Fn::Sub: echo ECS_CLUSTER=${ECSCluster} >> /etc/ecs/ecs.config
107
            02_start_ecs_agent:
108
              command: start ecs
109
          files:
110
            "/etc/cfn/cfn-hup.conf":
111
              mode: 256
112
              owner: root
113
              group: root
114
              content:
115
                Fn::Sub: |
116
                  [main]
117
                  stack=${AWS::StackId}
118
                  region=${AWS::Region}
119
            "/etc/cfn/hooks.d/cfn-auto-reloader.conf":
120
              content:
121
                Fn::Sub: |
122
                  [cfn-auto-reloader-hook]
123
                  triggers=post.update
124
                  path=Resources.ContainerInstances.Metadata.AWS::CloudFormation::Init
125
                  action=/opt/aws/bin/cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource ContainerInstances
126
          services:
127
            sysvinit:
128
              cfn-hup:
129
                enabled: true
130
                ensureRunning: true
131
                files:
132
                  - /etc/cfn/cfn-hup.conf
133
                  - /etc/cfn/hooks.d/cfn-auto-reloader.conf
134
  EC2Role:
135
    Type: AWS::IAM::Role
136
    Properties:
137
      Path: /
138
      AssumeRolePolicyDocument: |
139
        {
140
          "Statement": [{
141
              "Action": "sts:AssumeRole",
142
              "Effect": "Allow",
143
              "Principal": {
144
                "Service": "ec2.amazonaws.com"
145
              }
146
          }]
147
        }
148
      Policies:
149
        - PolicyName: ECSforEC2InstanceRolePolicy
150
          PolicyDocument: |
151
            {
152
              "Version": "2012-10-17",
153
              "Statement": [
154
                {
155
                  "Effect": "Allow",
156
                  "Action": [
157
                    "ecs:CreateCluster",
158
                    "ecs:DeregisterContainerInstance",
159
                    "ecs:DiscoverPollEndpoint",
160
                    "ecs:Poll",
161
                    "ecs:RegisterContainerInstance",
162
                    "ecs:StartTelemetrySession",
163
                    "ecs:Submit*",
164
                    "ecr:GetAuthorizationToken",
165
                    "ecr:BatchCheckLayerAvailability",
166
                    "ecr:GetDownloadUrlForLayer",
167
                    "ecr:BatchGetImage",
168
                    "logs:CreateLogStream",
169
                    "logs:PutLogEvents"
170
                  ],
171
                  "Resource": "*"
172
                }
173
              ]
174
            }
175
        - PolicyName: RexrayPolicy
176
          PolicyDocument: |
177
            {
178
              "Version": "2012-10-17",
179
              "Statement": [{
180
                "Effect": "Allow",
181
                "Action": [
182
                  "ec2:AttachVolume",
183
                  "ec2:CreateVolume",
184
                  "ec2:CreateSnapshot",
185
                  "ec2:CreateTags",
186
                  "ec2:DeleteVolume",
187
                  "ec2:DeleteSnapshot",
188
                  "ec2:DescribeAvailabilityZones",
189
                  "ec2:DescribeInstances",
190
                  "ec2:DescribeVolumes",
191
                  "ec2:DescribeVolumeAttribute",
192
                  "ec2:DescribeVolumeStatus",
193
                  "ec2:DescribeSnapshots",
194
                  "ec2:CopySnapshot",
195
                  "ec2:DescribeSnapshotAttribute",
196
                  "ec2:DetachVolume",
197
                  "ec2:ModifySnapshotAttribute",
198
                  "ec2:ModifyVolumeAttribute",
199
                  "ec2:DescribeTags"
200
                ],
201
                "Resource": "*"
202
              }]
203
            }
204
  EC2InstanceProfile:
205
    Type: AWS::IAM::InstanceProfile
206
    Properties:
207
      Path: "/"
208
      Roles:
209
        - Ref: EC2Role



Save the CloudFormation into a file ecs-cluster.yml, then run the following AWS CLI command: 

Shell
 




xxxxxxxxxx
1


 
1
$ aws cloudformation create-stack --stack-name docker-volume --parameters ParameterKey=VPCID,ParameterValue=<default-vpc-id> ParameterKey=SubnetId,ParameterValue=<public-subnet-id> ParameterKey=KeyName,ParameterValue=<key-pair-name>  --template-body file://./ecs-cluster.yml --capabilities CAPABILITY_IAM



Make sure to add the parameters values specific to your setup:

  • VPCID: you can use the default VPC
  • SubnetId: you need to select a public subnet, so you can always use one of the default subnets
  • KeyName: this needs to be an existing keypair. Required so you can SSH into the ECS Container Instances later.

In the AWS Console go to Services > CloudFormation After some time you'll see your stack reach the UPDATE_COMPLETE status. This may take up to 10 minutes.

stacks


Head over to Services > ECS, and you'll see you've got a new ECS Cluster called docker-volume-demo. Click on the name and you'll see you don't have any services or tasks yet, but go to ECS Instances and you'll see details of your two EC2 instances: 

cluster docker-volume-demo

Provisioning an ECS Task and Service

Now that our ECS Cluster is setup, we just need to deploy an ECS Task and ECS Service. Remember that the ECS Task can be thought of as a Docker container, whereas the ECS Service manages the ECS tasks, including ensuring enough replicas are running and setting up networking.

You can add the following template to the end of your ecs-cluster.yml file. Specifically, it's worth noting the following sections, specific to volumes:

  1. In Taskdefinition our ContainerDefinitions has MountPoints defined. /var/lib/postgresql/data is where Postgres stores it's data, and in this case, it will be mounted into the rexray-vol volume.
  2. In the TaskdefinitionVolumes section, we have a volume named rexray-vol. Here we're saying we want an AWS EBS volume to be auto-provisioned of type gp2 with size 5Gb. The type and size are specific to the REX-Ray driver we're using and are passed to the underlying docker volume create command.
Shell
 




xxxxxxxxxx
1
56


 
1
Taskdefinition:
2
    Type: AWS::ECS::TaskDefinition
3
    Properties:
4
      Family: postgres
5
      Cpu: 512
6
      Memory: 512
7
      NetworkMode: awsvpc
8
      RequiresCompatibilities:
9
        - EC2
10
      ContainerDefinitions:
11
        - Name: postgres
12
          Image: postgres
13
          Essential: true
14
          MountPoints:
15
            - SourceVolume: rexray-vol
16
              ContainerPath: /var/lib/postgresql/data
17
          PortMappings:
18
            - ContainerPort: 3306
19
              Protocol: tcp
20
          LogConfiguration:
21
            LogDriver: awslogs
22
            Options:
23
              awslogs-group: !Ref LogGroup
24
              awslogs-create-group: true
25
              awslogs-region: !Ref AWS::Region
26
              awslogs-stream-prefix: ecs
27
      Volumes:
28
        - Name: rexray-vol
29
          DockerVolumeConfiguration:
30
            Autoprovision: true
31
            Scope: shared
32
            Driver: rexray/ebs
33
            DriverOpts:
34
              volumetype: gp2
35
              size: 5
36
  Service:
37
    Type: AWS::ECS::Service
38
    Properties:
39
      Cluster: !Ref ECSCluster
40
      ServiceName: postgres
41
      DesiredCount: 1
42
      TaskDefinition: !Ref Taskdefinition
43
      LaunchType: EC2
44
      DeploymentConfiguration:
45
        MaximumPercent: 100
46
        MinimumHealthyPercent: 0
47
      NetworkConfiguration:
48
        AwsvpcConfiguration:
49
          SecurityGroups:
50
            - !Ref InstanceSecurityGroup
51
          Subnets:
52
            - !Ref SubnetId
53
  LogGroup:
54
    Type: AWS::Logs::LogGroup
55
    Properties:
56
      LogGroupName: postgres



Let's run the AWS CLI update-stack command to update our existing CloudFormation stack. You can use all the same parameters as you used in the create-stack command: 

Shell
 




xxxxxxxxxx
1


1
$ aws cloudformation update-stack --stack-name docker-volume --parameters ParameterKey=VPCID,ParameterValue=<default-vpc-id> ParameterKey=SubnetId,ParameterValue=<public-subnet-id> ParameterKey=KeyName,ParameterValue=<key-pair-name>  --template-body file://./ecs-cluster.yml --capabilities CAPABILITY_IAM



Once your CloudFormation stack update has completed, check out your cluster again in the AWS Console: 

We now have an active service, with one running Postgres ECS Task. ✅

Connecting to the Postgres Container

Since our Postgres container doesn't have a public IP and isn't connected to a load balancer, we'll have to connect via an SSH tunnel. This way we can have a Postgres client on our local machine, with a connection to our Postgres container routed via the ECS Container Instance on which it's deployed:

To set this up you need the private IP address of the ECS Task, which you can find on the task details page of the AWS Console under Network:

We'll also need the public IP of one of the ECS Container Instances, which you can grab by clicking on the container instance id on the same the task details page.

Now run:

Shell
 




xxxxxxxxxx
1


1
ssh -N -L 5432:<task-private-ip>:5432  ec2-user@<container-instance-public-ip>



This will setup the tunnel and continue running in the foreground. Note that if you already have Postgres installed on your local machine, you may have to choose a port other than 5432.

Create Database Data

To create some data on the EBS volume, we're going to create a Postgres database and add some test data. To do that, you can either use the psql command line tool or follow along with steps below which use pgAdmin, which is free to download.

pgAdmin data setup

Once you've installed pgAdmin, starting it will open up a page in your browser. Right click on Servers and select Create > Server. Enter a server name:

Click on the Connection tab, enter localhost as the Host name, then click Save:

If prompted, the default password is Postgres.

Right click on the new dockervolume server, and select Create > Database. Enter a database name, then click Save:

Click on the new database, then select Tools > Query Tool, and we can start running some SQL.

Execute the following SQL (shortcut to execute is F5) which will create a table with some healthy test data:

Shell
 




xxxxxxxxxx
1


 
1
CREATE TABLE vegetables (vegetable_name text, colour text);
2
 
          
3
INSERT INTO vegetables VALUES ('carrot', 'orange');



Drain the Instance

We're going to change the container instance state to DRAINING, which will force ECS to deploy our task onto the other container instance. If we can still access the database data once the ECS Task moves over, then that proves it's successfully persisted in the EBS volume.

To make sure we're draining the correct container instance, in ECS grab the container instance id that the task is currently running in:

You'll need the full ARN of the container instance, which you can get with this AWS CLI command and picking the matching result:

Shell
 




xxxxxxxxxx
1


1
$ aws ecs list-container-instances --cluster docker-volume-demo
2
{
3
    "containerInstanceArns": [
4
        "arn:aws:ecs:eu-west-1:299404798587:container-instance/02e78e33-f3cc-4121-ad2b-4e039cb610b9",
5
        "arn:aws:ecs:eu-west-1:299404798587:container-instance/214ad5c1-d3c1-41aa-b11f-7afcac542939"
6
    ]
7
}



Now we have the ARN, it's time to run the following update-container-instances-state command to change the state to DRAINING

Shell
 




xxxxxxxxxx
1


1
aws ecs update-container-instances-state --cluster docker-volume-demo --container-instances <container-instance-arn> --status DRAINING



Once that's happened, head over to ECS Instances in the AWS Console and you'll see the instance is in the DRAINING state:

Head on over to Tasks and eventually, you'll see a new task coming up on the remaining ACTIVE container instance.

Wait for it's status to reach RUNNING.

Verify that the Database Has Come Back

Now that our ECS Task has moved over to the other container instance, we can validate that the data has persisted by running an SQL SELECT query.

First though, your old SSH tunnel will now have a connection error. You'll need to grab the new private IP address from the ECS Task details page, then run the ssh command again:

Shell
 




xxxxxxxxxx
1


 
1
$ ssh -N -L 5432:<task-private-ip>:5432  ec2-user@<container-instance-public-ip>



Back in pgAdmin, disconnect and reconnect your dockervolume Server. Then run the following SELECT query on the dockervolume database: 

Shell
 




xxxxxxxxxx
1


 
1
SELECT * FROM vegetables;



You'll see we still have the same data. Awesome!

Cleanup

You can remove the CloudFormation stack with the following command:

Shell
 




xxxxxxxxxx
1


 
1
aws cloudformation delete-stack --stack-name docker-volume


Note that this won't delete the EBS volume, which was created automatically by REX-Ray outside of CloudFormation. Find the correct volume id with the following command: 

Shell
 




xxxxxxxxxx
1
23


 
1
$ aws ec2 describe-volumes --filter Name=tag:Name,Values=rexray-vol
2
{
3
    "Volumes": [
4
        {
5
            "Attachments": [],
6
            "AvailabilityZone": "eu-west-1a",
7
            "CreateTime": "2020-01-25T18:17:00.927Z",
8
            "Encrypted": false,
9
            "Size": 5,
10
            "SnapshotId": "",
11
            "State": "available",
12
            "VolumeId": "vol-08670b6c65571df51",
13
            "Iops": 100,
14
            "Tags": [
15
                {
16
                    "Key": "Name",
17
                    "Value": "rexray-vol"
18
                }
19
            ],
20
            "VolumeType": "gp2"
21
        }
22
    ]
23
}


Now run aws ec2 delete-volume --volume-id <volume-id>

Final Words

You should now understand that with the correct configuration, ECS Tasks can easily be setup to connect to AWS EBS volumes. The REX-Ray Docker volume driver does the hard work for us, and AWS ECS easily integrates with it to make sure that volumes are always attached to the correct EC2 host.

Please remember that this CloudFormation stack was designed as a simple example, and should not be used in production. For example, the Postgres instance should ideally not be exposed over the internet, and the ECS Container Instances should be deployed in a private subnet.

REX-Ray can also be configured to use AWS Elastic File System (EFS) too. If you have a requirement to access a volume from multiple ECS Tasks at the same time, you'll want to check out this option.

Now where next? Well it's really up to you. You could possibly try deploying a few more applications running on a Docker container on your own and let us know your comments. 


This article was originally published on https://appfleet.com/ 

Topics:
amazon aws, cloud, cloud native, devops, docker, tutorial

Published at DZone with permission of Sudip Sengupta . See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}