java,high-perf,saas,performance,logging,log,analyzers,comparison guide,splunk,splunk storm
The 7 Log Management Tools Java Developers Should Know
Splunk, Sumo Logic, LogStash, GrayLog, Loggly, PaperTrails - did I miss someone? I’m pretty sure I did. Logs are like fossil fuels - we’ve been wanting to get rid of them for the past 20 years, but we’re not quite there yet. Well, if that's the case I want a BMW!
To deal with the growth of log data a host of log management & analysis tools have been built over the last few years to help developers and operations make sense of the growing data. I thought it’d be interesting to look at our options and what are each tools’ selling point, from a developer’s standpoint.
Splunk vs. Sumo Logic vs. LogStash vs. GrayLog vs. Loggly vs. PaperTrails vs. Splunk>Storm
ProsSplunk is probably the most feature rich solution in the space. It’s got hundreds of apps (I counted 537) to make sense of almost every format of log data, from security to business analytics to infrastructure monitoring. Splunk’s search and charting tools are feature rich to the point that there’s probably no set of data you can’t get to through its UI or APIs.
ConsSplunk has two major cons. The first, that is more subjective, is that it’s an on-premise solution which means that setup costs in terms of money and complexity are high. To deploy in a high-scale environment you will need to install and configure a dedicated cluster. As a developer, it’s usually something you can't or don’t want to do as your first choice. Splunk’s second con is that it’s expensive. To support a real-world application you’re looking at tens of thousands of dollars, which most likely means you’ll need sign offs from high-ups in your organization, and the process is going to be slow. If you’ve got a new app and you want something fast that you can quickly spin up and ramp as things progress - keep reading. Some more enterprise log analyzers can be found here.
SaaS Log Analyzers
ProsSL is chock-full of features to reduce, search and chart mass amounts of data. Out of all the SaaS log analyzers, it’s probably the most feature rich. Also, being a SaaS offering it inherently means setup and ongoing operation are easier. One of Sumo Logic’s main points of attraction is the ability to establish baselines and to actively notify you when key metrics change after an event such as a new version rollout or a breach attempt.
ConsThis one is shared across all SaaS log analyzers, which is you need to get the data to the service to actually do something with it. This means that you’ll be looking at possible GBs (or more) uploaded from your servers. This can create issues on multiple fronts -
- As a developer, if you're logging sensitive or PII you need to make sure it’s redacted.
- There may be a lag between the time data is logged and the time it’s visible to to the service.
- There’s additional overhead on your machines transmitting GBs of data, which really depends on your logging throughput.
ProsWhereas Sumo Logic has a strong enterprise and security focus, Loggly is geared more towards helping devops find and fix operational problems. This makes it very developer-friendly. Things like creating custom performance and devops dashboards are super-easy to do. Pricing is also transparent, which makes start of use easier.
ConsDon't expect Loggly to scale into a full blown infrastructure, security or analytics solution. If you need forensics or infrastructure monitoring you’re in the wrong place. This is a tools mainly for devops to parse data coming from your app servers. Anything beyond that you’ll have to build yourself. tailing your log in the cloud, and you won't be too far off.
ProsPT is what it is. A simple way to look at log files from multiple machines in a singular view in the cloud. The UX itself is very similar to looking at a log on your machine, and so are the search commands. It aims to do something simple and useful, and does it elegantly. It’s also very affordable.
ConsPT is mostly text based. Looking for any advanced integrations, predictive or reporting capabilities? You're barking up the wrong tree.
ProsStorm lets you experiment with Splunk without having to install the actual software on-premise, and contains much of the features available in the full version.
ConsThis isn't really a commercial offering, and you're limited in the amount of data you can send. It seems to be more of an online limited version of Splunk meant to help people test out the product without having to deploy first. A new service called Splunk Cloud is aimed at providing a full-blown Splunk SaaS experience.
Open Source Analyzers
ProsBeing an open-source solution means you're inherently getting a lot of a control and a very good price. Logstash uses three mature and powerful components, all heavily maintained, to create a very robust and extensible package. For an open-source solution it’s also very easy to install and start using. We use Logstash and love it.
Takipi for Logs
While this post is not about Takipi, I thought there’s one feature it has which you might find relevant to all of this. The biggest disadvantage in all log analyzers and log files in general, is that the right data has to be put there by you first. From a dev perspective, it means that if an exception isn’t logged, or the variable data you need to understand why it happened isn't there, no log file or analyzer in the world can help you. Production debugging sucks. One of the things we’ve added to Takipi is the ability to jump into a recorded debugging session straight from a log file error. This means that for every log error you can see the actual source code and variable values at the moment of error. You can learn more about it here. This is one post where I would love to hear from you guys about your experiences with some of the tools mentioned (and some that I didn’t). I’m sure there are things you would disagree with or would like to correct me on - so go ahead, the comment section is below and I would love to hear from you.