6 Tips for AWS Command Line Ninjas

The AWS Command Line Interface (CLI) allows you to manage AWS services. Using the CLI from your terminal interactively allows you to half-automate tasks and frees you from logging into the AWS Management Console. In addition, integrating the CLI into shell scripts allows you to automate your infrastructure and the configuration of EC2 instances during the boot process.

This article covers typical hurdles when using the AWS CLI.

Command Completion

When using the CLI interactively within your terminal, command completion is a killer feature you should not miss. When enabled, command completion allows you to use the TAB key to complete commands. This will speed up your CLI usage significant.

The following steps are needed to enable command completion for bash on OS X:

echo "complete -C aws_completer aws" > ~/.bash_profile  
source ~/.bash_profile  

The official documentation contains general instructions for other shells as well.

Filtering Results of Requests on Server-Side

By default, the CLI uses a page size of 1,000 and retrieves all available items. If you need to request items from a list of more than 1,000 items, or if you want to speed up your commands, it is a good idea to filter the results of your request on the server side.

Many describe-* and list-* commands support server-side filtering: --filter. For example, it is possible to filter EC2 instances by instance type:

$ aws ec2 describe-instances --filter Name=instance-type,Values=t2.nano

Filtering Output on Client-Side

Another useful feature of the CLI is filtering the output of any command on the client side: --query. The JMESPath query language is used for filtering.

The following example lists all VPCs within a region and filters the results by using a --query.

$ aws ec2 describe-vpcs --query "Vpcs[?VpcId == 'vpc-aaa22bbb'].CidrBlock"
[
    "94.194.0.0/16"
]

You might need the CIDR of a VPC as a variable in your shell script. The following example shows how to achieve that. Formatting the output as text by adding the parameter --output text removes " character from the JSON result.

#!/bin/bash
CIDR=$(aws ec2 describe-vpcs --query "Vpcs[?VpcId == 'vpc-aaa22bbb'].CidrBlock" --output text)  
echo $CIDR  

Wait for...

When writing shell scripts by using the CLI, there will be the need to wait for a specific condition from time to time. For example, after initiating an EBS snapshot your script might need to wait until the snapshot was completed. Waiting can be achieved with a polling loop and a describe-* command. But there is a simpler solution built into the CLI for this: aws <service> wait <condition>.

The following example contains a wait command that will block the script until the snapshot has been completed.

#!/bin/bash
echo "Waiting for EBS snapshot"  
aws ec2 wait snapshot-completed --snapshot-ids snap-aabbccdd  
echo "EBS snapshot completed"  

Assuming an IAM Role

The CLI supports assuming an IAM role. Very handy if you need to switch between multiple AWS accounts with the help of cross-account roles.

All you need to do is to configure two profiles in ~/.aws/config: an IAM user and an IAM role profile.

[profile iam-user]
output = json  
region = eu-west-1

[profile iam-role]
role_arn = arn:aws:iam::<ACCOUNT_ID>:role/<IAM_ROLE>  
source_profile = iam-user  
output = json  
region = eu-west-1  

Only the IAM user needs security credentials stored in ~/.aws/credentials.

[iam-user]
aws_access_key_id = ***  
aws_secret_access_key = ***  

Afterwards, you are able to assume the IAM role by adding --profile iam-role to your CLI commands.

Fine-Tuning S3 config

The AWS CLI includes transfer commands for S3: cp, sync, mv, and rm. You are able to fine-tune these commands with special configuration.

For example, if you need to sync a large number of small files to S3, increasing the following values added to your ~/.aws/config config file will speed up the sync process.

[profile default]
...
s3 =  
  max_concurrent_requests = 100
  max_queue_size = 10000
  use_accelerate_endpoint = true

The official documentation contains detailed information about additional S3 configuration values.

Feedback

Is anything missing? Looking forward to your feedback! @andreaswittig or andreas@widdix.de.