Watching/Alerting on Real-Time Data in Elasticsearch Using Kibana and SentiNL

In the previous post, we set up an ELK stack and ran data analytics on application events and logs. In this post, we will discuss how you can watch real-time application events that are being persisted in the Elasticsearch index and raise alerts if the condition for the watcher is breached using SentiNL (a Kibana plugin).

A few examples of alerting for application events (see previous posts) are:

• The same user logged in from different IP addresses.
• Different users logged in from the same IP address.
• PermissionFailures in the last 15 minutes.
• A particular kind of exception in the last 15 minutes/ hour/ day.

Watching and Alerting on Elasticsearch Index in Kibana

There are many plugins available for watching and alerting on Elasticsearch index in Kibana e.g. X-Pack, SentiNL.

X-Pack is a paid extension provided by elastic.co which provides security, alerting, monitoring, reporting, and graph capabilities.

SentiNL is free extension provided by siren.io which provides alerting and reporting functionality to monitor, notify, and report changes in an Elasticsearch index using standard queries, programmable validators, and configurable actions.

We will be using SentiNL for watching and alerting on Elasticsearch index.

Installing SentiNL

Prerequisite

For debian, we need libfontconfig and libfreetype6 libraries, if not installed already.

For centos, we need fontconfig and freetype libraries, if not installed already.

#Installing SentiNL plugin
/opt/kibana/bin/kibana-plugin --install sentinl -u https://github.com/sirensolutions/sentinl/releases/download/tag-4.6.4-4/sentinl.zip

Configuring SentiNL

SentiNL has a wide range of actions that you can configure for watchers. You can send an email, integrate with Slack channels or push apps, and send a payload to custom webhooks. Open the kibana.yml file and add the below properties for SentiNL. For our example, we will only enable notifications through email.

sentinl:
  es:
    host: 'localhost'
    port: 9200
  settings:
    email:
      active: true
      host: "smtp.gmail.com"
      user: "[EMAIL_ID]"
      password: "[PASSWORD]"
      port: 465
      domain: "gmail.com"
      ssl: true
      tls: false
      authentication: ['PLAIN', 'LOGIN', 'CRAM-MD5', 'XOAUTH2']
      timeout: 20000  # mail server connection timeout
      # cert:
      #   key: '/full/sys/path/to/key/file'
      #   cert: '/full/sys/path/to/cert/file'
      #   ca: '/full/sys/path/to/ca/file'
    slack:
      active: false
      username: 'username'
      hook: 'https://hooks.slack.com/services/' channel: '#channel' webhook: active: false host: 'localhost' port: 9200 # use_https: false # path: ':/{{payload.watcher_id}}' # body: '{{payload.watcher_id}}{payload.hits.total}}' # method: POST report: active: false executable_path: '/usr/bin/chromium' # path to Chrome v59+ or Chromium v59+ timeout: 5000 # authentication: # enabled: true # mode: # searchguard: false # xpack: false # basic: false # custom: true # custom: # username_input_selector: '#username' # password_input_selector: '#password' # login_btn_selector: '#login-btn' # file: # pdf: # format: 'A4' # landscape: true # screenshot: # width: 1280 # height: 900 pushapps: active: false api_key: '' 

That's it! Let's start Kibana to configure watchers and alerting in SentiNL.

Creating Watchers and Alerting in Kibana

We will be configuring watchers for different users logged in from the same IP address and will send e-mail alerts.

• Open Kibana dashboard on your local machine (the url for Kibana on my local machine is http://localhost:5601).
• Click on the SentiNL option in the left-hand nav pane. You will see a dashboard as below. Click on the 'New' option to create a new watcher.
  
• Click on the Watcher link highlighted as below.
  
• Enter the watcher name and schedule in the General tab.
  
• Click on the 'Input' tab and enter the below-mentioned JSON query in the body. You can also give a name to the query and save.

  {
    "search": {
      "request": {
        "index": [
          "app-events*"
        ],
        "body": {
          "query": {
            "bool": {
              "filter": [
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-30m"
                    }
                  }
                },
                {
                  "query_string": {
                    "default_field": "appEvent.eventType",
                    "query": "LOGIN_SUCCESS OR LOGIN_FAILURE"
                  }
                }
              ]
            }
          },
          "aggs": {
            "group_by_requestIP": {
              "terms": {
                "field": "appEvent.requestIP.keyword",
                "size": 5
              },
              "aggs": {
                "group_by_identifier": {
                  "terms": {
                    "field": "appEvent.identifier.keyword",
                    "size": 5
                  },
                  "aggs": {
                    "get_latest": {
                      "terms": {
                        "field": "@timestamp",
                        "size": 1,
                        "order": {
                          "_key": "desc"
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }

• Click on the 'Condition' tab and enter the below-mentioned JSON conditions in the body. You can also give a name to this condition and save.

  {
    "script": {
      "script": "var requestIPbuckets = payload.aggregations.group_by_requestIP.buckets; payload.collector = []; requestIPbuckets.filter(function(requestIP) { return requestIP.key; }).forEach(function(requestIP) { var requestIPKey = requestIP.key; var users = requestIP.group_by_identifier.buckets; if (users.length > 1) { users.filter(function(user) { return user.key; }).forEach(function(user) { payload.collector.push({ 'ip': requestIPKey, 'identifier': user.key, 'count': user.doc_count  }); }); }}); payload.collector.length > 0;"
    }
  }

• Click on the 'Action' tab and select email as an action for alerting. Give title, to, from, subject, and add below-mentioned content in the body of the email.

  Found {{payload.collector.length}} Events
  {{#payload.collector}}
  {{#.}}
  ip : {{ip}}, identifier: {{identifier}}, count: {{count}}
  {{/.}}
  {{/payload.collector}}

• Save the watcher.

This watcher will run periodically based on the schedule that you have set and if the condition for breach is met, will send an email alert. The configured email looks like below.

This is how you can watch real-time changing data in Elasticsearch index and raise alerts based on the configured conditions.