Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Kibana Tutorial: Getting Started [Part 1]

DZone's Guide to

Kibana Tutorial: Getting Started [Part 1]

The first part of this two-part tutorial will show you the basics of installing and using Kibana, as well as some helpful tips and tricks.

· Performance Zone ·
Free Resource

Sensu is an open source monitoring event pipeline. Try it today.

Kibana is the visualization layer of the ELK Stack, the world's most popular log analysis platform which is comprised of Elasticsearch, Logstash, and Kibana.

This tutorial will guide you through some of the basic steps for getting started with Kibana—installing Kibana, defining your first index pattern, and running searches using the Lucene query syntax. Examples are provided throughout, as well as tips and best practices.

Installing Kibana

Presuming you already have Elasticsearch installed and configured, we will start with installing Kibana. If you want to find out more about installing Elasticsearch, check out this Elasticsearch tutorial.

Depending on your operating system and your environment, there are various ways of installing Kibana. We will be installing Kibana on an Ubuntu 16.04 machine running on AWS EC2 on which Elasticsearch and Logstash are already installed.

Start by downloading and installing the Elastic public signing key:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -


Add the repository definition:

echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list


All that's left to do is to update your repositories and install Kibana:

sudo apt-get update
sudo apt-get install kibana


Open up the Kibana configuration file at: /etc/kibana/kibana.yml, and make sure you have the following configurations defined:

server.port: 5601
elasticsearch.url: "http://localhost:9200"


These specific configurations tell Kibana which Elasticsearch to connect to and which port to use.

Now, start Kibana with:

sudo service kibana start


Open up Kibana in your browser with: http://<yourServerIP>:5601. You will be presented with the Kibana home page.

Defining an Index Pattern

Your next step is to define a new index pattern, or in other words, tell Kibana what Elasticsearch index to analyze. To do that you will, of course, need to have data indexed.

For the purpose of this tutorial, we've prepared some sample data containing Apache access logs that are refreshed daily. You can download the data here: sample-data

Next, we will use Logstash to collect, parse and ship this data into Elasticsearch. If you haven't installed Logstash yet, or are not familiar with how to use it, check out this Logstash tutorial.

Create a new Logstash configuration file at: /etc/logstash/conf.d/apache-01.conf

sudo vim /etc/logstash/conf.d/apache-01.conf


Enter the following Logstash configuration (change the path to the file you downloaded accordingly):

input {
  file {
    path => "/home/ubuntu/apache-daily-access.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
 
filter {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
    geoip {
      source => "clientip"
    }
}
 
output {
  elasticsearch { 
    hosts => ["localhost:9200"] 
  }
}


Start Logstash with:

sudo service logstash start


If all goes well, a new index will be created in Elasticsearch, the pattern of which can now be defined in Kibana.

In Kibana, click the Set up index patterns button, and Kibana will automatically identify the new "logstash-*" index pattern.

Define it as "logstash-*", and in the next step select @timestamp   as your Time Filter field.

Hit Create index pattern, and you are ready to analyze the data. Go to the Discover tab in Kibana to take a look at the data (look at today's data instead of the default last 15 mins).

Using Logz.io

If you're using Logz.io, simply use this curl  command to upload the sample log data. Logz.io listeners will parse the data using automatic parsing so there's no need to configure Logstash (the token can be found on the Settings page in the Logz.io UI, and the type of the file is  apache_access ):

curl -T <Full path to file> http://listener.logz.io:8021/file_upload/<Token>/apache_access


Kibana Searching

Kibana querying is an art unto itself, and there are various methods for performing searches on your data. This step will describe some of the most common search methods as well as some tips and best practices that should be memorized for optimized user experience.

Free-Text Search

Free text search works within all fields—including the _source  field, which includes all the other fields. If no specific field is indicated in the search, the search will be done on all of the fields that are being analyzed.

In the search field at the top of the Discover page, run these searches and examine the result (set the time parameter on the top right of the dashboard to the past month to capture more data):

  • category
  • Category
  • categ
  • cat*
  • categ?ry
  • “category”
  • category\/health
  • “category/health”
  • Chrome
  • chorm*

Tips and Gotchas

  1. Text searches are not case sensitive. This means that "category" and "CaTeGory" will return the same results. When you put the text within double quotes (""), you are looking for an exact match, which means that the exact string must match what is inside the double quotes. This is why [category\/health] and ["category/health"] will return different results
  2. Kibana wildcard searches—you can use the wildcard symbols [*] or [?] in searches. [*] means any number of characters, and [?] means only one character

Field-Level Searches

Another common search in Kibana is field-level queries, sued for searching for data inside specific fields. To use this type of search that, you need to use the following format:

<fieldname>:search


As before, run the following searches to see what you get (some will purposely return no results):

  • name:chrome
  • name:Chrome
  • name:Chr*
  • response:200
  • bytes:65
  • bytes:[65 TO *]
  • bytes:[65 TO 99]
  • bytes:{65 TO 99}
  • _exists_:name

Tips and Gotchas

  1. Field-level searches depend on the type of field. (Logz.io users—all fields are not analyzed by default, which means that searches are case-sensitive and cannot use wildcard searches. The reason we save all of the fields as "not analyzed" is to save space in the index because the data is also duplicated in an analyzed field called  _source )
  2. You can search a range within a field. If you use [], this means that the results are inclusive. If you use {}, this means that the results are exclusive.
  3. Using the _exists_   prefix for a field will search the documents to see if the field exists.
  4. When using a range, you need to follow a very strict format and use capital letters TO to specify the range

Logical Statements

You can use logical statements in searches in these ways:

  • USA AND Firefox
  • USA OR Firefox
  • (USA AND Firefox) OR Windows
  • -USA
  • !USA
  • +USA
  • NOT USA

Tips and gotchas

  1. You need to make sure that you use the proper format such as capital letters to define logical terms like AND or OR
  2. You can use parentheses to define complex, logical statements
  3. You can use -,! and NOT to define negative terms

Kibana Special Characters

All special characters need to be properly escaped. The following is a list of all available special characters:

+ - && || ! ( ) { } [ ] ^ " ~ * ? : \

Proximity searches

Proximity searches are an advanced feature of Kibana that takes advantage of the Lucene query language.

[categovi~2] means a search for all the terms that are within two changes from [categovi]. This means that all category will be matched.

Tips and gotchas

Proximity searches use a lot of system resources and often trigger internal circuit breakers in Elasticsearch. If you try something such as [catefujt~10], it is likely not to return any results due to the amount of memory used to perform this specific search.

Kibana Filtering Dialog

To assist users in searches, recent versions include a filtering dialog that allows easier forming of Kibana search syntax.

To use the dialog, simply click the Add a filter + button under the search box and begin experimenting with the conditionals.

Power users can also enter Elasticsearch queries using the Query DSL.

Summary

This tutorial covered the basics of using Kibana and detailed steps for setting up a test environment only. Tracking Apache access logs in production, for example, is better done using Filebeat and the supplied Apache module. The Kibana sample data provided can, of course, be replaced with other types of data, as you see fit.

If you have any suggestions on what else should be included in the first part of this Kibana tutorial, please let me know in the comments below. The next Kibana tutorial will cover visualizations and dashboards.

Sensu: workflow automation for monitoring. Learn more—download the whitepaper.

Topics:
Performance ,kibana ,installation ,basic tutorial ,elk stack ,visualization

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}