The ELK Stack (Elasticsearch, Logstash, and Kibana) is the world’s most popular open source log analysis platform. With 500,000 downloads a month, ELK is quickly overtaking existing proprietary solutions and becoming THE first choice for companies shopping for log analysis and management solutions.
Setting up the ELK Stack is a complicated task that requires a lot of expertise, so many companies these days are instead looking for “hosted ELK,” or “ELK as a service,” or “cloud-hosted ELK” solutions whenever possible.
Cloud-hosted ELK solutions all seek to offer an easy and scalable way to use the stack, but they differ in many ways. Recognizing the growing role that ELK is playing in the world of log analytics, we thought it would be interesting to compare the leading solutions by examining the criteria that is crucial to check when contemplating which ELK logging platform to use.
What is ELK?
The ELK Stack is comprised of three separate yet homogenous open-source products: Elasticsearch, Logstash, and Kibana. Elasticsearch, probably the more well-known of the three components, is the search engine that powers the stack. Based on Apache Lucene, Elasticsearch can be used to perform full-text and other complex searches. Logstash processes the data before sending it to Elasticsearch for indexing and storage. Kibana is the visualization tool with which you can view the log messages and create graphs and visualizations.
Why is ELK so Popular?
The ELK Stack is popular for a number of reasons. First, IT companies are moving more and more of their architecture—including logging systems—to open source technologies. Second, existing logging platforms are too expensive for small and medium-sized companies. ELK, for example, might not have all of the features of Splunk, but it does not need that feature-richness. ELK is a simple but robust log analysis platform that costs a fraction of the price. The community being created around ELK is another reason for the growing popularity.
As the leading cloud service provider, it was just a matter of time until AWS introduced its hosted solution for ELK.
Introduced in October last year, this service enables AWS users to setup and configure an Elasticsearch cluster from the AWS Management Console. The service then provisions all the resources for the cluster and launches it.
Monitoring, backups and security is handled by integrations with complimentary AWS services. IAM policies (Identity and Access Management) controls access to Elasticsearch, automatic and manual snapshots store data on S3, and CloudWatch and CloudTrail provide the monitoring and auditing services.
Scalability, which is quite the issue in large production environments, is not automated. Users are expected to monitor the cluster using either CloudWatch, or any other weapon of choice, and manually add resources as the need arises. In case of node failure, the failing node is automatically detected and replaced.
The process of log parsing and mapping needs to be handled manaully by the user, with full support for Logstash.
The service currently ships with Elasticsearch 1.5.2 and it is not clear how upgrades to newer versions are handled and what happens with the data during the transition period.
Pricing for the service is pay as you go and according to EC2 policies. A nice option is to use the Free Tier for development -- with 750 hours per month for a single-AZ t2.micro instance and 10GB a month of optional EBS storage, but you will quickly surpass these limits with a reasonably-sized production logging system. Keeping tabs on your AWS costs is a challenge and will most likely necessitate using a cloud cost management service.
Elastic—the commercial entity behind the ELK Stack—recently released its own cloud platform called Elastic Cloud, based on AWS infrastructure.
Elastic offers a free trial option for 14 days, with 1GB memory and 16GB for storage. Creating an account is easy, after which you are given access to a management dashboard. From this dashboard you can then configure your new cluster -- its size, region, the number of data centers you want to use, the version of Elasticsearch you wish to use, number of shards and more.
For security and managing user access, you are provided with Elastic Shield (you cannot access Kibana without first enabling and configuring Shield). You also are given access to other Elastic services: Marvel, Sense, Timelion, and Graph.
Backing up your logs is taken care of by snapshots that are taken every 30 minutes and stored on S3.
Like with AWS, scaling is done manually by the user, as is the logging pipeline (aggregation, parsing and mapping).
As for pricing, this of course depends on the cluster setup and your needs. A medium sized cluster for production use, with 64GB of reserved memory and 1TB of reserved storage, will cost around $2600/month.
This platform offers a somewhat different approach by providing a more complete ELK service. Logz.io has done a good job of publicizing the platform’s architecture, so some information is available on how they process the data coming into the system.
The platform uses Kafka as a message queue for all of the incoming messages, including those from Logstash (queuing is a common best practice for logging systems, and is used so as not to overload Elasticsearch). As part of the service, Logz.io provides auto parsing, auto sharding, and auto mapping for logs, so the ingestion pipeline is all taken care of automatically.
There are various log shipping integrations available, so users can quickly integrate their existing environments and establish a shipping pipeline. Generic shippers such as Filebeat, Logstash and rsyslog are all supported, together with native support for AWS, Docker and specific languages.
As with Elastic and AWS, logs can be archived to S3.
Logz.io offers role based access for security and basic user management features. An important feature to point out is the ability to create alerts, based on saved searches, for events you want to be notified about, either via email or a chatting program (e.g. Slack).
Another nice feature is ELK Apps -- a collection of pre-made Kibana visualizations for various types of logs (e.g. Apache, AWS, Docker).
There is a free trial available for 14 days restricted to shipping up to 1GB a day. The pro plan costs $89 a month, and includes 1GB a day of log shipping with 14 days retention and S3 archiving.
Logsene is Sematext’s ELK-as-a-Service offering, and compliments the other analytical tools and services Sematext offers.
Creating a new Logsene application is easy enough, and once created, you can either use the Sematext UI to manage and analyze the logs or the integrated Kibana 4 (you can also hook into your own Kibana or Grafana.) As with Logz.io, Sematext offer a wide variety of integrations with standard logging methods and platforms, but again—parsing of logs and the ingestion pipeline is up to the user.
A feature worth mentioning is Live Tail for live viewing of incoming messages -- very useful for monitoring your environment for errors taking place in real time. Sematext also offers an alerting mechanism that integrates with chatting programs so you can get notifications about events.
For monitoring performance metrics in your environment, you can use Sematext’s SPM offering within the same UI, which is useful for identifying correlations in the data.
Sematext includes full role-based access control with owner, admin and user roles, and also provides an on-premise option for users who can’t ship logs to the cloud.
The pro pricing plan costing $60 a month caters for 1GB a day, 7 days retention, and S3 archiving.
To sum things up, we’ve provided a comparison table below that will help you see the big picture. Different challenges necessitate different solutions. It all depends on your specific requirements.
What is clear is that if you’re looking for a log analytics solution based on ELK as an end-to-end service, the solution offered by Logz.io is the most functional and most compliant offering. If you’re looking for a searching solution only, Elastic provide an excellent solution. AWS offers a low-cost solution but requires a substantial amount of work to integrate with other AWS services and establishing the log pipeline. Sematext offer a good solution, and we especially liked the concept of live tail, but found the pricing a bit expensive.