Over a million developers have joined DZone.

Why ELK Stack

· Big Data Zone

Learn how you can maximize big data in the cloud with Apache Hadoop. Download this eBook now. Brought to you in partnership with Hortonworks.

Problem statement:

You have different applications feeding into each other. Debugging an issue requires logging into each individual box to look at the logs. With small number of apps/boxes it's not an issue, but it quickly becomes tedious as the number of apps/boxes increase!


It would be awesome to have all of your logs aggregated into one place so you can see the process flow and perform queries against the logs from all applications from one place.

Enter ELK stack. It will also improve your heart health because no need to log into 100 different boxes to follow the logs!

What is ELK stack?

ELK stands for Elasticsearch, Logstash and Kibana.

Brief definitions:

Logstash: It is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs. It is fully free and fully open source.

Elasticsearch: Elasticsearch is a search server based on Lucene. It provides a distributed, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents.

Kibana: A nifty tool to visualize logs and timestamped data.

How do we use it?

Here's one possible (and recommended) architecture diagram that helps visualize the flow of log information:

As shown in the diagram: App 1...App n are generating logs. Logstash shipper* is reading the logs from a log file and sending the logs to a redis database. Logstash in the ELK stack is reading off of the redis database, feeding that information on to elasticsearch. Kibana is a configrable web dashboard that is querying elastic search for log information and presenting it to the user. Logstash of course can be configured to read logs/register events from a myriad of sources and send log events to multiple different sources. We will cover Logstash in more depth in a different post.

You can follow this post to learn how to install ELK stack on your server.

Hortonworks DataFlow is an integrated platform that makes data ingestion fast, easy, and secure. Download the white paper now.  Brought to you in partnership with Hortonworks


Published at DZone with permission of Ravi Hasija, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

The best of DZone straight to your inbox.

Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}