Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Analyzing Data With IP2Location and the ELK Stack

DZone's Guide to

Analyzing Data With IP2Location and the ELK Stack

Geolocation data is incredibly valuable. In this tutorial, learn how to use the IP2Location filter plugin with Elasticsearch, Logstash, and Kibana to analyze geolocation data.

· Big Data Zone ·
Free Resource

The open source HPCC Systems platform is a proven, easy to use solution for managing data at scale. Visit our Easy Guide to learn more about this completely free platform, test drive some code in the online Playground, and get started today.

Geolocation data has become increasingly important over the past few years, regardless of whether you are a marketer, business owner, or developer. We understand that geolocation provides important data about generating sales from potential leads. It helps us learn more about our online visitors and customers, such as where are they coming from, their time zone, and so on.

In this article, we will guide you on how to use IP2Location filter plugin with Elasticsearch, Logstash, and Kibana to monitor your network traffic data.

To follow this tutorial, you must have a working ELK environment. You can install the ELK stack here.

Configure Filebeat for Logging With Logstash

  1. Download the suitable package for Filebeat and install it on your machine.
  2. Configure the filebeat.yml file that is located in your Filebeat root directory. Replace the content as follows:
    filebeat.prospectors:
    - input_type: log
      paths:
        - /path/to/log/file
    output.logstash:
      hosts: ["localhost:5043"]
  3. You can get the sample dataset that was used in this tutorial here.
  4. Unpack the file and make sure the paths field in the filebeat.yml is pointing correctly to the downloaded sample dataset log file.
  5. Save the filebeat.yml file.
  6. Run the command below on your machine:
    sudo ./filebeat -e -c filebeat.yml -d "publish"

Configure Logstash to Use IP2Location Filter Plugin

  1. Install the IP2Location filter plugin with:
    bin/logstash-plugin install logstash-filter-ip2location
  2. Create a Logstash configuration pipeline named test-pipeline.conf with the following content:
    input {
      beats {
    	port => "5043"
      }
    }
    
    filter {
      grok {
    	match => { "message" => "%{COMBINEDAPACHELOG}"}
      }
      ip2location {
    	source => "clientip"
      }
    }
    
    
    output {
      elasticsearch {
    	hosts => [ "localhost:9200" ]
      }
    }
  3. Save the test-pipeline.conf file.
  4. Run the command below at your machine to start Logstash:
    bin/logstash -f test-pipeline.conf --config.reload.automatic

Visualize Data in Kibana

After you have successfully installed Kibana on your machine, you can use Kibana to view the data by opening Kibana in a web browser.

Example obtained from IP2Location filter plugin with ELK using IP2Location DB24 BIN:

Image title

And that's it!

Managing data at scale doesn’t have to be hard. Find out how the completely free, open source HPCC Systems platform makes it easier to update, easier to program, easier to integrate data, and easier to manage clusters. Download and get started today.

Topics:
big data ,data analytics ,logging ,filebeat ,logstash ,elk ,kibana ,ip2location ,geolocation ,tutorial

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}