Today there is no shortage of interesting information being produced from machines and software applications alike giving us more data to analyze than ever before. The Big Data problem is generally focused on making sense of this data; and last week’s announcement of Amazon’s Machine Learning Service highlights the need to make analytics of Big Data more accessible. My take is that this new AWS service will go a long way in helping to achieve this. However, usage of this service will require some basic understanding of machine learning, data mining and data analytics. Look around you… probably not too many people in the office fall into this category… So what about for the rest of us?
At Logentries we are all about doing the hard work so you don’t have to. Our mission is to make the power of log data accessible to anyone. Thus, today we are announcing a new Logentries service providing automated log data analytics reports produced and verified by the Logentries Data Insights Teams. The reports are designed to provide valuable insights into log data without the need for users to perform complex analytics. The first of these reports will monitor and analyze AWS CloudTrail log data with an overview metrics report including:
- System activity over time (Day, Week, Totals)
- Most common security-related actions
- Most active users
- Instance launch and termination over time
- Top successful user access by region
- Top unauthorized user access by regions
Why is There a Need for This Service?
Well in many cases the signal to noise ratio for large volumes of data can be very low – this is particularly the case when it comes to log data…
For example, check out our analysis from last year where we looked at Heroku logs and found that for a troubleshooting use case, less that 0.018% of the log data being produced was useful for identifying platform errors. Don’t get me wrong, Heroku logs are very useful for troubleshooting and system monitoring (in fact the guys are Heroku have done a great job when it comes to logging best practices) but the point is that logs by their nature are often extremely verbose. They can capture every important interaction in your system, and because they are so verbose, often only a small percentage of the overall logs are useful for your given use case – say troubleshooting or error tracking. For another use case – say security – a small percentage of the overall log data is generally useful; and for another – say feature usage tracking another small percentage of the overall data is useful. So while it can make sense to log verbosely, as it gives you the ability to use your logs for a range of different use cases and provides you with deep system understanding with wide coverage, extracting the signal from all the noise can indeed be challenging without the right tools and skills.
Logentries has been designed to be easy to use and provides out of the box intelligence for common components (which makes log analysis for common systems easier than with any other logging provider), but someone on your team will still need to know what to look for, you will need to spend time perusing your logs, and will require a basic understanding for the data set you are analyzing.
So to make your life even easier our new Logentries Analytics service will provide automated log data analytics reports produced and verified by the Logentries Data Insights Teams.
Our Insights Team is made up of guys and gals with PhDs and MScs in data analytics, anomaly detection, performance engineering and security analysis. Their job is to separate the signal from the noise and to draft, validate and serve up weekly reports for our users across a range of different areas (including system usage analytics, error tracking, security and compliance analysis, operations and uptime tracking…).
Our users already send us all their log data, we have the expertise and the Phds … the real question is why wouldn’t we provide this service. It makes our users lives easier, separates the signal from the noise and provides deep insights across a wide range of user cases.