DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Data Engineering
  3. Data
  4. Log Management With the ELK Stack on Windows Server — Part 1

Log Management With the ELK Stack on Windows Server — Part 1

Learn to how to use the ELK Stack Log Management Platform, including Elasticsearch, to process data and manage logs.

Shamil Mehdiyev user avatar by
Shamil Mehdiyev
·
Oct. 02, 18 · Tutorial
Like (3)
Save
Tweet
Share
8.05K Views

Join the DZone community and get the full member experience.

Join For Free

As the world of digital content grows, trying to read, find, or analyze any information becomes difficult and time-consuming. You need to spend a huge amount of time to process information. To solve this problem, some good guys (thanks to their passion) developed definitions like search engines, designed to help decrease the time required to find information and the amount of information which must be consulted.

In this post, we will try to understand the ELK Stack (in another word, Elastic Stack) Log Management Platform, which uses the most popular search engine, Elasticsearch. I have broken up the explanation into three parts:

  1. Introduction — What is the ELK Stack?
  2. Installation — Install ELK Stack on Windows Server 2012 R2
  3. Customization — Customize and visualize custom logs

This article is Part 1, the introduction.

What Is the ELK Stack?

The ELK Stack is a collection of three open-source products: Elasticsearch, Logstash, and Kibana. The whole process can be visualized as follows:

Image title

Elasticsearch

Elasticsearch is a distributed, RESTful search and analytics engine based on Apache Lucene, which is capable of solving a growing number of use cases. Elasticsearch is developed in Java and is released as open source under the terms of the Apache License and categorized as a NoSQL database. Which means, it stores data in an unstructured way.

According to the DB-Engines ranking, Elasticsearch is the most popular enterprise search engine, followed by Apache Solr (also based on Apache Lucene).

In the context of data analysis, Elasticsearch is used together with the other components in the ELK Stack, Logstash and Kibana, and plays the role of data indexing and storage.

Logstash

Logstash is an open-source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite "stash" (e.g. Elasticsearch).

Image title

Logstash is a log pipeline tool, which has 3 stages:

  1. Input Plugins - Accepts inputs from various sources
  2. Filter Plugins - Executes different transformations
  3. Output Plugins - Exports the data to various targets

Input Plugins: Logstash supports a variety of inputs that pull in events from a multitude of common sources, all at the same time. Input plugin examples: file, HTTP, beats, log4j, etc. Check the documentation for full list of Input Plugins.

Filter Plugins: As data travels from source to store, Logstash filters parse each event, identify named fields to build a structure, and transform them to converge on a common format for easier, accelerated analysis and business value.

A filter plugin performs intermediary processing on an event. Filters are often applied conditionally depending on the characteristics of the event. The most commonly used filter plugins, which I used during the working process, are the following (check the documentation for all possible Filter Plugins):

  1. grok - Parses unstructured event data into fields
  2. mutate - Performs mutations on fields
  3. kv - Parses key-value pairs

Output Plugins: Logstash has a variety of outputs that let you route data where you want. In our case it will be Elasticsearch, but it can be also file, email, HTTP, etc. You can check the list of all possible Output Plugins from the documentation.

Kibana

The last component of the ELK stack is Kibana. Kibana is the world's most popular open-source log analysis platform. It provides users with a tool for exploring, visualizing, and building dashboards on top of the log data stored in Elasticsearch clusters. Kibana makes it easy to understand large volumes of data.

It's simple, browser-based interface enables you to quickly create and share dynamic dashboards that display changes to Elasticsearch queries in real time. Kibana's visualization features allow users to visualize data in a variety of different ways, using charts, tables, geographical maps and other types of visualizations.

Image title

Conclusion

I hope I was able to help you to understand what the ELK Stack is. Please feel free to share with me your comments and feedback.

In Part 2, you will find the installation process of ELK Stack on Windows Server 2012 R2 platform.

References:

  1. Elasticsearch (wiki)
  2. Elasticsearch
  3. Kibana
  4. Logstash
Open source Data processing Elasticsearch Kibana

Published at DZone with permission of Shamil Mehdiyev. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Connecting Your Devs' Work to the Business
  • Too Many Tools? Streamline Your Stack With AIOps
  • The Future of Cloud Engineering Evolves
  • A Simple Union Between .NET Core and Python

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: