Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Analyzing Data With IP2Location and the ELK Stack

DZone's Guide to

Analyzing Data With IP2Location and the ELK Stack

Geolocation data is incredibly valuable. In this tutorial, learn how to use the IP2Location filter plugin with Elasticsearch, Logstash, and Kibana to analyze geolocation data.

· Big Data Zone ·
Free Resource

Hortonworks Sandbox for HDP and HDF is your chance to get started on learning, developing, testing and trying out new features. Each download comes preconfigured with interactive tutorials, sample data and developments from the Apache community.

Geolocation data has become increasingly important over the past few years, regardless of whether you are a marketer, business owner, or developer. We understand that geolocation provides important data about generating sales from potential leads. It helps us learn more about our online visitors and customers, such as where are they coming from, their time zone, and so on.

In this article, we will guide you on how to use IP2Location filter plugin with Elasticsearch, Logstash, and Kibana to monitor your network traffic data.

To follow this tutorial, you must have a working ELK environment. You can install the ELK stack here.

Configure Filebeat for Logging With Logstash

  1. Download the suitable package for Filebeat and install it on your machine.
  2. Configure the filebeat.yml file that is located in your Filebeat root directory. Replace the content as follows:
    filebeat.prospectors:
    - input_type: log
      paths:
        - /path/to/log/file
    output.logstash:
      hosts: ["localhost:5043"]
  3. You can get the sample dataset that was used in this tutorial here.
  4. Unpack the file and make sure the paths field in the filebeat.yml is pointing correctly to the downloaded sample dataset log file.
  5. Save the filebeat.yml file.
  6. Run the command below on your machine:
    sudo ./filebeat -e -c filebeat.yml -d "publish"

Configure Logstash to Use IP2Location Filter Plugin

  1. Install the IP2Location filter plugin with:
    bin/logstash-plugin install logstash-filter-ip2location
  2. Create a Logstash configuration pipeline named test-pipeline.conf with the following content:
    input {
      beats {
    	port => "5043"
      }
    }
    
    filter {
      grok {
    	match => { "message" => "%{COMBINEDAPACHELOG}"}
      }
      ip2location {
    	source => "clientip"
      }
    }
    
    
    output {
      elasticsearch {
    	hosts => [ "localhost:9200" ]
      }
    }
  3. Save the test-pipeline.conf file.
  4. Run the command below at your machine to start Logstash:
    bin/logstash -f test-pipeline.conf --config.reload.automatic

Visualize Data in Kibana

After you have successfully installed Kibana on your machine, you can use Kibana to view the data by opening Kibana in a web browser.

Example obtained from IP2Location filter plugin with ELK using IP2Location DB24 BIN:

Image title

And that's it!

Hortonworks Sandbox for HDP and HDF is your chance to get started on learning, developing, testing and trying out new features. Each download comes preconfigured with interactive tutorials, sample data and developments from the Apache community.

Topics:
big data ,data analytics ,logging ,filebeat ,logstash ,elk ,kibana ,ip2location ,geolocation ,tutorial

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}