Hunting the ELK (Stack): Data Monitoring to Visualization
Hunting the ELK (Stack): Data Monitoring to Visualization
Everything you need to master your big data workflow.
Join the DZone community and get the full member experience.
Join For Free
Made up of Elastisearch, "a search and analytics engine," Logstash, "a server-side data processing pipeline that "ingests data from multiple sources simultaneously, transforms it, and then sends it to a 'stash'," (according to Elastic's official site) and Kibana, a robust visualization tool, the ELK stack has quickly become one of the premier tools available to developers for data processing, management, and visualization.
Whether you're just starting out with any of the three technologies, or you're a seasoned veteran, we've compiled the best that our community has to offer for basic questions about getting started to complex tutorials for real-time data management.
Before we begin, we'd like need to thank those who were a part of this article. DZone has and continues to be a community powered by contributors like you who are eager and passionate to share what they know with the rest of the world.
Let's get started!
Elastisearch
Getting Started
If you're looking to get up and running with Elastisearch, look no further than Elastisearch Setup and Configuration by Guarev Rai Mazra, as he walks readers through basic concepts behind the framework, installation, and configuration with Java.
For further understanding concerning Elastisearch, check out Elastisearch 101 by Lucas Saldana. In the article, readers will go further in-depth on the fundamentals of Elastisearch, including indexing and searching, data analysis, and querying. (For a more in-depth look at querying data, see An Introduction to Elastisearch by Hassan Rahhal.)
Follow along in Data Analytics Made Easier With Elastisearch, as Mitul Makadia explains to readers why they should pick the framework for their data analytics needs.
Get an overview of Veronika Rovnik's first-hand with Elastisearch and the Elastic Stack in Reporting and Analysis With Elastisearch.
Elastisearch vs the Rest
Yigal compares Solr and Elastisearch in Solr vs Elastisearch: Who's the Leading Open Source Search Engine? Find out which framework is best for your project's needs.
In What Is Elastisearch? (And Why You Need to Be Using it), Mitul Makadia breaks down the advantages that Elastisearch offers compared to its competitors.
Follow along with Vincent Royer, as he explains how Elassandra can be a potential replacement for Elastisearch when working with Kubernetes logs, as the tool can give users the benefits of powerful scaling and low downtime.
Spring Boot and Elastisearch
In this Sring Boot and Elastiesearch Tutorial, MVB, Rajeesh Bhojwani discuss how to use a Spring-Data-Elastisearch project to connect with the Elastisearch engine by using the Transport Client library in order to perform CRUD operations.
Elastisearch Query Cheatsheets
Tim Ojo, in one of our most popular posts to date, lays out 23 useful Elastisearch queries that readers can bookmark for the next time they need to work with Elastisearch in a pinch.
In this Game-of-Thrones-themed tutorial, Sohan Ganapathy explains how parent and child relationships function within Elastisearch and how to perform joins on data within those relationships.
Reporting and Analysis With Elastisearch
Follow along with Veronika Rovnik, as she discusses her experiences working with Elasticsearch, the Elastic Stack, and a few complimentary dev tools for the big data platform in Reporting and Analysis With Elastisearch.
In this two-part series by Ayush Jain, the developer covers the frameworks that make up the ELK stack (Elastisearch, Logstash, and Kibana) and how they all work together. Parts one and two can be found here and here, respectively.
Get Elastisearch up and running on Kubernetes, as Itamar Syn-Hersko explains the structure of both Elastisearch and Kibana and then shows readers how to deploy Elastisearch on K8s.
Elastisearch Clusters
In Elastisearch Tutorial: Creating an Elastisearch Cluster, Daniel Burman walks readers through setting up an Elastisearch cluster and offers them some operational tips and best practices to get started.
Follow along with Burak Atlas, as he explains how to configure settings for Elasticsearch clusters in order to improve queries latency.
Elastisearch Performance
Follow along with Burak Atlas, as he explains how to optimize Elastisearch
Learn how to perform bulk inserts with Elastisearch's REST high-level client in Sujith Menon's most recent article.
Logstash
Getting Started
Get started with Logstash in Installing Logstash by perennial ELK-stack-contributor, Gaurav Rai Mazra, as he explains the tool's basic architecture and how to install it.
If you're a Java programmer coding Microservices and working with the ELK stack, this tutorial by Nicolas Frankel is perfect for you. Learn how to use grok and Logstash's dissect filter to parse Spring Cloud tracing logs.
Logstash Alternatives
Looking for other options for Logstash? Look no further than Radu Gheorghe's article, Five Alternatives to Logstash, as he breaks down advantages and disadvantages of Logstash, as well as its open source competitors.
In Filebeat vs Logstash — The Evolution of a Log Shipper by Daniel Berman, readers can get an in-depth comparison of the two technologies and use cases for when each is optimal.
Migrating Data
In this article by Leona Zhang, learn how to migrate data clusters in ElisticSearch with Logstash for situations like backing up data during a system upgrade.
Follow along with Shriram Untawale, as he shows readers how to migrate MySQL data to Elastisearch using Lohstash.
Logstash Debugging and Tips
Having some trouble with your configuration file? Check out this tutorial by Daniel Berman, as he walks readers through issues he's previously faced and how to fix them.
In 10 Things to Consider When Parsing With Logstash by Bipin Patwardhan, the developer walks readers through pain points he's encountered in the past when writing Logstash scripts.
Get some quick and dirty debugging hints for all of your Logstash-needs with Nicolas Frankel's article, Debugging Hints for Logstash.
See how to handle issues related to the "multiple" feature in Logstash with Bipin Patwardhan's article, Logstash — Quirky "Multiline."
Monitoring Logs
Learn how to set up Filebeat, Logstashs, and Elastisearch to monitor Docker Swarm logs to ensure reliable microservice architecture in Arun Sharma's two-part series on the subject. Part one and two can be found here and here, respectively.
In this article by Radu Gheorghe's, see how to replay Elastisearch Slowlogs with Logstash and JMeter.
Follow along with MVB, Comsysto Gmbh, as he explains how to combine Logstash and Graylog in order to create an enterprise-ready, flexible, scalable controlled log management system.
Creating a Plugin
Logstash for a Java developer means jumping into the world of gems, rbenv, JRuby, etc. Getting started means diving headfirst into the entire Ruby ecosystem. See how in So, You Want to Make a Logstash Plugin by Nicolas Frankel.
Kibana
Getting Started
Let's start at the very beginning: installation. Follow along with Guarav Rai Mazra, as he explains to readers how to get this powerful visualization tool onto your local machines in Installing Kibana.
Visualizing Data
Learn how to begin creating robust and powerful visualizations and dashboards with Kibana and Elastiseach data in Veronika Rovnik's article, Kibana and Beyond: How to Visualize Elastisearch Data.
In this article by Asaf Yigal, see how to get started with Kibana from basic installation to some helpful tips and tricks for data visualization.
Go in-depth on visualizations with MVB, Daniel Berman. In Creating Custom Kibana Visualizations, the author explains how to work with Vega-Lite in Kibana in order to create visualizations that better help tell a story with your data.
Kibana Queries
In his second appearance in this collection, Daniel Berman walks readers through different types of queries in Kibana to help you search for a wider variety of data in a more flexible way.
Logs
In this article by Rafal Kuc, check out basic behind logging data to Elastisearch, including log structure in Kibana, writing logs to a JSON file, and sending JSON-formatted logs to Elastisearch.
Kibana Tips and Tricks
In Kibana Hacks: Five Tips and Tricks, Daniel Berman takes a look at some workarounds he's found useful for tackling specific pain points or missing features in Kibana, including embedding images, inserting links, and adding log messages to dashboards.
The ELK Stack: Putting it All Together
Getting Started
For more on Elastisearch use cases, check out ELK Stack Overview and Use Cases by Sudip Bhandari to see just why this tool has become so popular for data analysis and visualization.
- Want to bring in the ELK stack for your AWS logging and monitoring needs? Check out this comprehensive guide by Asaf Yigal, as he walks readers through step-by-step on how to get started with this powerful set of data analysis tools.
ELK Stack Tutorials
See how the ELK stack works in real-world application with this tutorial by Asaf Yigal, as he shows readers how to use the open source, log analysis platform with OpenStack.
Follow along with developer, Ayush Jain in his two-part series on working with the ELK stack. In part one, he explains how Elastisearch, Logstash, and Kibana (plus Beats) work together. In part two, he dives into ELK's overall architecture and workflow.
In this two-part series by Guarav Rai Mazra, see how to use Elastiseach, Logstash, and Kibana to run analytics on application events and logs, and then check out how to watch and alert on real-time data within that application.
Forget about Logstash for this tutorial. In Using Telegraf Elastisearch Input Plugin by Sonia Gupta, see how to set up an InfluxDB sandbox with an Elastisearch node (populated with data using Kibana).
In this article by Joydip Kumar, learn about monitoring and logging and how to collate logs for multiple microservices in Setting Up the ELK Stack With Spring Boot Microservices.
In Kafka Logging With the ELK Stack by Daniel Berman, explore a tech combination you might not be used to — using the ELK stack to collect and analyze Kafka logs.
With this last tutorial, MVB, Rishav Rohit, shows readers how to develop a demo app for click-stream weblog ingestion, search, and visualization with the ELK stack.
Be a Part of the Conversation!
Think we missed something? Want to contribute? Let us know in the comments below... or, join the conversation by becoming a member of our community of thousands of developers eager to share their knowledge and passion for programming with others.
Further Reading
Opinions expressed by DZone contributors are their own.
{{ parent.title || parent.header.title}}
{{ parent.tldr }}
{{ parent.linkDescription }}
{{ parent.urlSource.name }}