The Definitive Guide to AWS Log Analytics Using ELK: Part I
This is a step-by-step guide to retrieving log data from all cloud layers and then visualizing and correlating these events to give a clear picture of one’s entire AWS infrastructure. Be sure to keep an eye out for Part II.
Join the DZone community and get the full member experience.Join For Free
this is designed to help developers, devops engineers, and operations teams that run and manage applications on top of aws to effectively analyze their log data to get visibility into application layers, operating system layers, and different aws services. this is a step-by-step guide to retrieving log data from all cloud layers and then visualizing and correlating these events to give a clear picture of one’s entire aws infrastructure.
why should you look at your logs?
cloud applications are inherently more distributed and built out of a series of components that need to operate together to deliver a service to the end user successfully. analyzing logs becomes imperative in cloud environments because the practice allows relevant teams to see how all of the building blocks of a cloud application are orchestrated independently and in correlation with the rest of the components.
why elk (elasticsearch, logstash, and kibana)?
elk is the most common log analytics platform in the world. it is used by companies including netflix, linkedin, facebook, google, microsoft, and cisco. elk is an open source stack of three libraries (elasticsearch, logstash, and kibana) that parse, index, and visualize log data (and, yes, it’s free).
so instead of going through the challenging task of building a production-ready elk stack internally, users can signup and start working in a matter of minutes. in addition, logz.io’s elk as a service includes alerts, multi-user, and role-based access, and unlimited scalability. on top of providing an enterprise-grade elk platform as a service, logz.io employs unique machine-learning algorithms to automatically surface critical log events before they impact operations, providing users with unprecedented operational visibility into their systems.
- how to deploy elk in production
- lessons learned from elasticsearch cluster disconnects
- how to use elk to monitor performance
- how to integrate aws cloudtrail and the elk stack
analyzing application logs
why should i analyze my application logs?
application logs are fundamental to any troubleshooting process. this has always been true — even for mainframe applications and those that are not cloud-based. with the pace at which instances are spawned and decommissioned, the only way to troubleshoot an issue is to first aggregate all of the application logs from all of the layers of an application. this enables you to follow transactions across all layers within an application’s code.
how do i ship application logs?
there are dozens of ways to ship application logs. the best method to use depends on the type of application, the format of the logs, and the operating system. for example, java applications running on linux servers can use logstash or logstash -forwarder (a version that is lightweight and includes encryption) or ship it directly from the application layer using a log4j appender via https/http. you can read more in our essay on the 6 must-dos in modern log management .
analyzing infrastructure logs
what are infrastructure logs?
we consider everything which is not the proprietary application code itself to be an infrastructure log. these include system logs, database logs, web servers logs, network device logs, security device logs, and countless others.
why should i analyze infrastructure logs?
infrastructure logs can shed light on problems in the code that is running or supporting your application. performance issues can be caused by overutilized or broken databases or web servers, so it is crucial to analyze those log files especially when correlated with the application logs. while troubleshooting performance issues, we’ve seen many cases in which the root cause was a linux kernel issue. overlooking such low-level logs can make forensics processes long and fruitless. read more about why it’s important to ship os logs in our essay on lessons learned from elasticsearch cluster disconnects .
how do i ship infrastructure logs?
shipping infrastructure logs is usually done with open-source agents such as rsyslog , logstash , logstash forwarder, or nxlog that read the relevant operating system files such as access logs, kern.log, and database events. you can read here about more methods to ship logs here .
monitoring system performance with elk
one of the challenges organizations face when troubleshooting performance issues is that they are looking at one dashboard that shows performance metrics and another to troubleshoot issues and analyze logs. in many cases, it’s possible to use a single dashboard to that shows both the performance metrics and the visualized log data that is being generated by all of the components of your system. in many cases, performance issues are related to events in application stacks that are recorded in log files. collecting system performance metrics and shipping them as log entries then enables quick correlations between performance issues and their respective events in the logs.
how do i ship performance metrics?
to use elk to monitor your platform’s performance, run probes on each host to collect system performance metrics. software service operations teams can then visualize the data with the kibana part of elk and use the resulting charts to present their results.
for example, we encapsulated collectl in a docker container to have a docker image that covered all of our data collecting and shipping needs. read more and get a download on our site: how to use elk to monitor platform performance .
be sure to catch us next time in part ii when we take a look at monitoring ebl logs. we will also cover aws cloudtrail logs, aws vpc flow logs, cloudfront logs, and s3 access logs.
Published at DZone with permission of Samuel Scott, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.