CloudWatch Disk Space Monitoring For Elastic Beanstalk Instances

If you’re responsible for developing and deploying web applications within your AWS environment for your organization, then it’s likely you may have heard of AWS Elastic Beanstalk already. The service’s automation and simplification make it ideal for deploying, provisioning, monitoring and scaling efficient environments to run developed applications in.

By leveraging Amazon’s orchestration service AWS Elastic Beanstalk, we can quickly deploy and manage all the resources for applications that we wish to upload to the AWS cloud. Elastic Beanstalk is a great developer tool that’s easy to get started with and comes packed with several other benefits including:

• Autoscaling options

• Improved developer productivity

• Customization options

• Cost-effective capabilities

With AWS Elastic Beanstalk, we can spin up our entire application environment without having to configure any of the resources manually. Rather, AWS Elastic Beanstalk takes responsibility for deploying the correct infrastructure to run the uploaded code. Plus, there are no separate costs for running Elastic Beanstalk in AWS; you only pay for the resources you use to run your application, for example, the storage you use in Amazon S3. Also, the cost is not fixed; it can vary according to the number of EC2 instances, the size of your S3 bucket, and how you’ve configured your database instances.

CloudWatch Alarms For Elastic Beanstalk Instances

By default, AWS Elastic Beanstalk only shares the  EnvironmentHealth metrics with CloudWatch. To enable any other metrics that are essential for monitoring Beanstalk, you will have to configure enhanced monitoring. AWS provides detailed technical documentation on how to enable enhanced monitoring using the Beanstalk console, CLI, or configuration files. You just have to select which metrics you want to be collected by instance or environment.

Once you’ve set up enhanced monitoring, Elastic Beanstalk will publish a host of metrics to CloudWatch. Setting up alarms on these to look for any abnormalities and limits is a great way to spot issues in your AWS environments. In this post, we will cover how to monitor the disk space of Elastic Beanstalk instances with the help of Cloudwatch by creating alarms for those metrics.

To monitor the "disk," "swap," and "memory" spaces for our example Linux instances, we just need to deploy certain scripts. With the help of these scripts, we can monitor whatever requirements we wish.

.Ebextensions:

If you haven’t used .ebextensions before; it’s important to know that the .ebextensions folder will run within your Elastic Beanstalk environment creation. Here we create a script that monitors the disk space metric which will be kept inside the .ebextension folder. For example, cloudwatch.config. See the full script below for more details:

Packages:

xxxxxxxxxx

  yum:

    perl-DateTime: []

    perl-Sys-Syslog: []

    perl-LWP-Protocol-https: []

    perl-Switch: []

    perl-URI: []

    perl-Bundle-LWP: []

Source Material:

https://aws-cloudwatch.s3.amazonaws.com/downloads/CloudWatchMonitoringScripts-1.2.1.zip

Container_commands:

x

 01-setupcron:

    command: |

      echo '*/5 * * * * root perl /opt/cloudwatch/aws-scripts-mon/mon-put-instance-data.pl `{"Fn::GetOptionSetting" : { "OptionName" : "CloudWatchMetrics", "DefaultValue" : "--mem-util --disk-space-util --disk-path=/" }}` >> /var/log/cwpump.log 2>&1' > /etc/cron.d/cwpump

  02-changeperm:

    command: chmod 644 /etc/cron.d/cwpump

  03-changeperm:

    command: chmod u+x /opt/cloudwatch/aws-scripts-mon/mon-put-instance-data.pl

Option_settings:

x

  "aws:autoscaling:launchconfiguration" :

    IamInstanceProfile : "aws-elasticbeanstalk-ec2-role"

  "aws:elasticbeanstalk:customoption" :

    CloudWatchMetrics : "--disk-space-util --disk-space-used --disk-space-avail --disk-path=/"

Creating Alarms For Disk Space Monitoring

To create Cloudwatch Alarms for disk space monitoring, put this below script in a .ebextension folder and deploy it on the app. Save the script below as alarms.config.

Packages:

xxxxxxxxxx

yum:

    jq: []

x

  yum:

    perl-DateTime: []

    perl-Sys-Syslog: []

    perl-LWP-Protocol-https: []

    perl-Switch: []

    perl-URI: []

    perl-Bundle-LWP: []

Files:

xxxxxxxxxx

 "/tmp/alarams.sh" :

      mode: "000777"

      content : |

        #!/bin/bash

        RESULT="`wget -qO- http://169.254.169.254/latest/meta-data/instance-id`"

  Region="`curl --silent http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r .region`"

 EB_ENV=$(/opt/elasticbeanstalk/bin/get-config environment | jq -r 'Test')

 aws cloudwatch put-metric-alarm --region=$Region --alarm-name DiskSpaceUtilization6-$RESULT-$EB_ENV --alarm-description "Alarm when DiskSpaceUtilization exceeds 50 percent" --metric-name DiskSpaceUtilization --namespace System/Linux --statistic Average --period 300 --threshold 50 --comparison-operator GreaterThanThreshold  --dimensions Name=Filesystem,Value=/dev/xvda1 Name=MountPath,Value=/ Name=InstanceId,Value=$RESULT --evaluation-periods 2 --alarm-actions arn:aws:sns:us-east-1:XXXXXXXXXXX:example.name --unit Percent

Container_commands:

x

  01_alarams: 

    command: "/tmp/alarams.sh"

Publishing CloudWatch Metrics

In the above script, we set the threshold as 50%. However, you can modify that setting as per your individual requirements. You can also set up to receive alarms by email. You just need to create an SNS topic and subscribe that to your email ID and input your SNS ARN into the script.

In order to publish custom Amazon CloudWatch metrics, the instances in your environment need permission to use CloudWatch. You can grant permissions to your environment’s instances by adding them to the environment’s instance profile. You can add permissions to the instance profile before or after deploying your application.

Also, the IAM role which you’re using for your Elastic Beanstalk application has to have full access to Cloudwatch Alarms. Only then can it create disk space monitoring metrics and alarms for the instances.

Now to monitor disk space and create an alarm for when it hits our 50% threshold, we create a folder called .ebextension in the code environment and place the above two scripts in that folder, then zip them and upload them to Beanstalk.

If you run your app successfully, you’ll see the following output:

Beanstalk CloudWatch metrics can be viewed normally through the Metrics section of CloudWatch, but it is also possible to use the Monitoring tab in the Beanstalk console. For alarms, go to the Alarms tab under CloudWatch and you will find your alarm created there, as per the image below:

Use the method described to take the manual work out of monitoring Beanstalk and run this automatic setup to effortlessly watch for anomalies with your disk space within AWS Elastic Beanstalk without any overly complex configuration.

Further Reading

AWS Elastic Load Balancing (ELB) and Monitoring