Programs have generated logs for as long as they have existed — but with the advent of modern server and application software, logging has become a part of the IT management and monitoring process. Servers and applications can generate log data on a variety of processes, from simple announcements about healthy checks to detailed information on running processes.
It’d be interesting to look at these tools from a developer’s point of view. In this article, we shall review five of the most popular log management standalone tools of today. They are listed in a random order, not in the order of performance or capabilities.
Fluentd is an open-source data collector that lets you unify the data collection and consumption for a better use and understanding of data. Fluentd’s performance has been proven in the field: its largest user currently collects logs from 5,000+ servers, 5 TB of daily data, handling 50,000 messages/sec at peak time. Yes, the main reason to use it is performance. Fluentd has a flexible plugin system that allows the community to extend its functionality.
Apache 2.0 License project.
More than 300 community-contributed plugins.
Requires very little system resource.
Store data in multiple systems.
Doesn’t provide any built-in visualizations.
LogPacker provides you with two solutions: Standalone and Cloud. The Standalone version illustrates the network with Agents and Servers in the Cluster. There can be any amount of servers in this network. Agents can be installed or integrated into a various amount of platforms such as Unix, Windows, Android, iOs, WP, and even into the website’s JS, while the Server accepts logs from all Agents and saves them into any storage. The main advantage is that LogPacker works just from the box and can find and send to the Cluster all possible logs on the server, grouped and aggregated.
All LogPacker services are written in Go and created for high performance. The Agent instance usually spends around 30-40 MB of memory on default server installation.
Built-in support for more than 100 log sources types.
Multiple storage providers.
Alerting and reporting system by Email, Slack or SMS.
Easy installation from packages.
Wide platforms support (Unix, Windows, Mobile, JS).
REST API to build custom solution based on saved data.
Events aggregation and security.
Disk-based Cloud dashboard.
The standalone version has no built-in web interface.
The free version has a limit up to five servers in Cluster.
Logstash is a part of ELK stack, which includes ElasticSearch (clustered search and storage system) and Kibana (web frontend for ElasticSearch). With 200 plugins and counting, Logstash can connect to a variety of sources and stream data at scale to a central analytics system. Built with extensibility in mind, Logstash provides an API for rapid plugin development by the community. With recent improvements of the plugin ecosystem, contributors are able to publish new plugins at any time.
Free and open source.
Great integration with other Elastic products.
Extended functionality via plugins.
Filters are code.
Does not come bundled with a UI.
Filters can be difficult to write.
No native alerts.
Graylog is an open-source packager that claims to perform the same functions as Splunk. Graylog is written in Java and its web interface is written in Ruby on Rails. Graylog does not have the ability to read directly from syslog files. Instead, you need to send your messages directly to Graylog and it’s less convenient. You can perform searches of your data just as in Splunk and with similar search functions. Alerting is also a possibility in Graylog, but the alert emails were less than informative on their own, providing only a reference to search results contained on Graylog’s web interface.
Free and open source.
Streams allow identifying events in real-time and perform actions.
Server-side functionality can be extended via plug-ins.
Intuitive search interface.
Graylog only has support for syslog and GELF.
Logagent-js includes a command line tool with default settings for Logsene as the Elasticsearch backend for storing the shipped logs. Logagent-js detects the log format automatically using the built-in pattern definitions (and also lets you provide your own, custom patterns).
Lightweight and easy to install (via NPM).
Free and open source.
No native UI.
Only JSON and ES as output options.