Learn about the components of Logstash, a data processing pipeline as part of performance monitoring, and learn how to install it.
Join the DZone community and get the full member experience.Join For Free
Logstash is a data processing pipeline which ingests data simultaneously from multiple data sources, transforms it, and sends it to different "stashes" like Elasticsearch, Redis, a database, a REST endpoint, etc. It can ingest log files, cleaning and transforming them to machine- and human-readable formats.
There are three components in Logstash: Inputs, Filters, and Outputs.
Logstash ingests data of any kind, shape, and size: logs, AWS metrics, instance health metrics, etc.
Logstash filters parse each event, build a structure, enrich the data in the event, and transform it into the desired form. For example: enriching geo-location from an IP using the GEO-IP filter, anonymizing PII information from events, transforming unstructured data to structural data, using GROK filters, etc.
This is the sink layer. There are many output plugins, like Elasticsearch, email, Slack, Datadog, database persistence, etc.
As of this writing, Logstash (6.2.3) requires Java 8 to run. To check your Java version, run the following command:
The output on my system is as follows:
java version "1.8.0_161" Java(TM) SE Runtime Environment (build 1.8.0_161-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
If Java 8 is not installed, then please download it from the Oracle website and follow the instructions for installation. Also, set the
Installing From Binaries
You can directly download the binaries here.
Installing From Package Repositories
Installation with APT:
//ADD PUBLIC SIGNING KEY wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - //add https-transports sudo apt-get install apt-transport-https //save the repository definition echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list //installation command sudo apt-get update && sudo apt-get install logstash
Installation with YUM:
// Download and install the public signing key rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
Add the following in a new .repo file in your
[logstash-6.x] name=Elastic repository for 6.x packages baseurl=https://artifacts.elastic.co/packages/6.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md
// Installation command sudo yum install logstash
You can follow this link for Docker installation.
Published at DZone with permission of Gaurav Rai Mazra, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.