Installing Logstash

Logstash is a data processing pipeline which ingests data simultaneously from multiple data sources, transforms it, and sends it to different "stashes" like Elasticsearch, Redis, a database, a REST endpoint, etc. It can ingest log files, cleaning and transforming them to machine- and human-readable formats.

Components

There are three components in Logstash: Inputs, Filters, and Outputs.

Inputs

Logstash ingests data of any kind, shape, and size: logs, AWS metrics, instance health metrics, etc.

Filters

Logstash filters parse each event, build a structure, enrich the data in the event, and transform it into the desired form. For example: enriching geo-location from an IP using the GEO-IP filter, anonymizing PII information from events, transforming unstructured data to structural data, using GROK filters, etc.

Outputs

This is the sink layer. There are many output plugins, like Elasticsearch, email, Slack, Datadog, database persistence, etc.

Installing Logstash

As of this writing, Logstash (6.2.3) requires Java 8 to run. To check your Java version, run the following command:

java -version

The output on my system is as follows:

java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

If Java 8 is not installed, then please download it from the Oracle website and follow the instructions for installation. Also, set the JAVA_HOME variable.

Installing From Binaries

You can directly download the binaries here.

Installing From Package Repositories

Installation with APT:

//ADD PUBLIC SIGNING KEY
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

//add https-transports
sudo apt-get install apt-transport-https

//save the repository definition
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list

//installation command
sudo apt-get update && sudo apt-get install logstash

Installation with YUM:

// Download and install the public signing key
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Add the following in a new .repo file in your /etc/yum.repos.d/ directory

[logstash-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

// Installation command
sudo yum install logstash

Docker Installation

You can follow this link for Docker installation.