The Definitive Guide to AWS Log Analytics Using ELK: Part II
The Definitive Guide to AWS Log Analytics Using ELK: Part II
Here is Part II of this step-by-step guide to retrieving log data from all cloud layers and then visualizing and correlating these events to give a clear picture of one’s entire AWS infrastructure.
Join the DZone community and get the full member experience.Join For Free
Sensu is an open source monitoring event pipeline. Try it today.
In Part I, we covered why you should be looking at your logs, why ELK and Logz.io, analyzing application and infrastructure logs, and monitoring your system performance with ELK. This time, we will cover monitoring EBL logs, AWS CloudTrail logs, AWS VPC flow logs, Cloudfront logs, and S3 access logs.
Monitoring ELB Logs
What Are ELB Log Files?
ELB is Amazon Web Services’ EC2 load balancer. The ELB logs are a collection of all of the traffic running through the ELB. This data includes from where the ELB was accessed, which internal machines were accessed, the identity of the requester (e.g., the operating system and browser) and additional metrics such as processing time and traffic volume.
How Can I Use ELB Log Files?
There are many uses for ELB logs, but the main reasons are to check the operational health of the ELB and it’s efficient operation. In the context of operational health, you might want to determine if your traffic is being equally distributed amongst all internal servers. For operational efficiency, you might want to identify the volume of access that you are getting from different locations in the world. You can visit the ELK Labs and search for “ELB” to find different visualizations, dashboards, and alerts.
How Can I Ship ELB Log Files?
ELB logs can be saved into a S3 bucket by making a very simple configuration in your EC2 console. Once the files are in the S3 bucket, you can configure read-only access to that bucket here.
Security – AWS CloudTrail logs
What Are CloudTrail Log Files?
CloudTrail logs is a logging mechanism of Amazon Web Services’ EC2, which records all of the changes done in an environment. This is a very powerful and robust tool that gives a different set of events for each EC2 object that can be leveraged according to the desired use. EC2 log events include, among other things, access to the EC2 account and changes to security groups as well as activation and termination of machines and services.
How Can I Use CloudTrail Log Files?
CloudTrail logs are very powerful and have many uses. One of the main uses revolves around auditing and security. For example, we monitor access and receive internal alerts on suspicious activity in our environment. Two important things to remember: Keep track of any changes being done to security groups and VPC access levels, and monitor your machines and services to ensure that they are being used properly by the proper people. You can visit the ELK Labs and can search for “CloudTrail” to find different visualizations, dashboards, and alerts.
How Can I Ship CloudTrail Log Files?
CloudTrail logs are easy to configure because they ship to S3 buckets. As opposed to some EC2 services, CloudTrail logs can be collected from all different regions and availability zones into a single S3 bucket. Once the files are in the S3 bucket, you can configure read-only access to that bucket here.
AWS VPC Flow Logs
What Are VPC Flow Logs?
VPC flow logs provide the ability to log all of the traffic that happens within an AWS VPC (Virtual Private Cloud). The information captured includes information about allowed and denied traffic (based on security group and network ACL rules). It also includes source and destination IP addresses, ports, IANA protocol numbers, packet and byte counts, time intervals during which flows were observed, and actions (ACCEPT or REJECT).
How Can I Use the VPC Logs?
VPC flow logs can be turned on for a specific VPC, VPC subnet, or an Elastic Network Interface (ENI). Most common uses are around the operability of the VPC. You can visualize rejection rates to identify configuration issues or system misuses, correlate flow increases in traffic to load in other parts of systems, and verify that only specific sets of servers are being accessed and belong to the VPC. You can also make sure the right ports are being accessed from the right servers and receive alerts whenever certain ports are being accessed. You can visit ELK Labs and search for “VPC” to find different visualizations, dashboards, and alerts.
How Can I Ship VPC Logs?
Once enabled, VPC flow logs are stored in Cloudwatch logs, and you can extract them to a third-party log analytics service via several methods. The two most common methods are to direct them to a Kinesis stream and dump them to S3 using a Lambda function. At Logz.io, we recommend using a third-party open source tool to dump cloudwatch logs to S3. You can read more about the different methods here.
What Are CloudFront Access Logs?
CloudFront is AWS’s CDN, and CloundFront logs include information in W3C Extended Format (http://www.w3.org/TR/WD-logfile.html) and report all access to all objects by the CDN.
How Can I Use CloudFront Logs?
CloudFront logs are used mainly for analysis and verification of the operational efficiency of the CDN. You can see error rates through the CDN, from where is the CDN being accessed, and what percentage of traffic is being served by the CDN. These logs, though very verbose, can reveal a lot about the responsiveness of your website as customers navigate it. You can visit ELK Labs at https://app.logz.io/#/labs and search for “CloudFront” to find different visualizations, dashboards, and alerts.
How Can I Ship Cloudfront Logs?
Once enabled, CloudFront will write data to your S3 bucket every hour or so. You can then pull the CloudFront logs to Logz.io by pointing to the relevant S3 Bucket. Go here for additional assistance and to see examples on how to configure access.
S3 Access Logs
What Are S3 Access Logs?
S3 access logs record events for every access of an S3 Bucket. Access data includes the identities of the entities accessing the bucket, the identities of buckets and their owners, and metrics on access time and turnaround time as well as the response codes that are returned.
How Can I Use S3 Access Logs?
Monitoring S3 access logs is a key part of securing AWS environments. You can determine from where and how buckets are being accessed and receive alerts on illegal access of your buckets. You can also leverage the information to receive performance metrics and analyses on such access to ensure that overall application response times are being properly monitored.
How Can I Ship S3 Access Logs?
Once enabled, S3 access logs are written to a S3 bucket of your choice. You can then pull the S3 access logs to Logz.io by pointing to the relevant S3 Bucket. Go here for additional assistance and to see examples of configuring access.
ELK is a very powerful platform and can provide tremendous value when you invest the effort to generate a holistic view of your environment. When running on AWS, the majority of infrastructure logs can be added with a single click of the button to Logz.io’s ELK Cloud platform. In a manner of minutes, you’ll be able to leverage the auto-generated dashboards and alerts.
There are many uses for AWS logs that range from performing audits to maintaining security — and all uses can be supported with S3 access and CloudTrail logs and then monitored with CloudFront and VPC flow logs. Make sure to check out ELK Labs for the marketplace for auto-generated dashboards and alerts.
Published at DZone with permission of Samuel Scott , DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.