ELK Stack With Vagrant and Ansible

i had been playing with elk on a routine basis, so, for what i thought to be a quick win, i decided to add to my earlier blog post on building elasticsearch clusters with vagrant . well, it did not quite turn out that way and i had to cover a good bit of ground and publish code to other repos in order for this blog to be useful.

to recap, that post used (a) virtualbox as the means to build the vms for the cluster, and (b) a shell script to orchestrate the installation and configuration of an elasticsearch cluster on those vms. in this post, we will still use virtual box for giving us the vms, but enhance the provisioning in two ways.

1. we will build a full elk stack where application logs are shipped by beats to a logstash host for grokking and posting to an es cluster hooked to kibana for querying and dashboards. here is a schematic.
2. the provisioning (install and config) of the software for each of e (elasticsearch), l (logstash), k (kibana), and filebeat plugin is done via ansible playbooks. why? while provisioning with shell scripts is very handy, it is programmatic and can make building complex coupled software systems across a cluster of hosts too complicated. ansible hides much of that and instead presents more or a less a declarative way (playbooks!) of orchestrating the provisioning. while there are alternatives, ansible has become insanely popular lately in the devops world.

you can download the code from github to play along with the build out.

1. the inventory

we need 7 vms - 2 for applications with filebeat, 1 es master node, 2 es data nodes, and 1 each for logstash, and kibana. the names and ip addresses for these vms will be needed both by vagrant for creating these and, later, by ansible for provisioning. so we prepare a single inventory file and use it with both vagrant and ansible. further, this file rations the cpu/memory resources on my 8-core, 16gb memory laptop across these 7 vms. the file is simply yaml that is processed in ruby by vagrant and in python by ansible. our inventory.yml file looks like:

es-master-nodes:
  hosts:
    es-master-1:                    # hostname
      ansible_host: 192.168.33.25   # ip address
      ansible_user: vagrant
      memory: 2048                  # ram to be assigned in mb
      ansible_ssh_private_key_file: .vagrant/machines/es-master-1/virtualbox/private_key

es-data-nodes:
  hosts:
    es-data-1:
      ansible_host: 192.168.33.26
      ansible_user: vagrant
      memory: 2048
      ansible_ssh_private_key_file: .vagrant/machines/es-data-1/virtualbox/private_key

    es-data-2:
      ansible_host: 192.168.33.27
      ansible_user: vagrant
      memory: 2048
      ansible_ssh_private_key_file: .vagrant/machines/es-data-2/virtualbox/private_key

kibana-nodes:
  hosts:
    kibana-1:
      ansible_host: 192.168.33.28
      ansible_user: vagrant
      memory: 512
      ansible_ssh_private_key_file: .vagrant/machines/kibana-1/virtualbox/private_key

logstash-nodes:
  hosts:
    logstash-1:
      ansible_host: 192.168.33.29
      ansible_user: vagrant
      memory: 1536
      ansible_ssh_private_key_file: .vagrant/machines/logstash-1/virtualbox/private_key

filebeat-nodes:
  hosts:
    filebeat-1:
      ansible_host: 192.168.33.30
      ansible_user: vagrant
      memory: 512
      ansible_ssh_private_key_file: .vagrant/machines/filebeat-1/virtualbox/private_key

    filebeat-2:
      ansible_host: 192.168.33.31
      ansible_user: vagrant
      memory: 512
      ansible_ssh_private_key_file: .vagrant/machines/filebeat-2/virtualbox/private_key

2. the vagrantfile

the vagrantfile below builds each of the 7 vms as per the specs in the inventory.

require 'rbconfig'
require 'yaml'

default_base_box = "bento/ubuntu-16.04"
cpucap = 10                                   # limit to 10% of the cpu
inventory = yaml.load_file("inventory.yml")   # get the names & ip addresses for the guest hosts
vagrantfile_api_version = '2'

vagrant.configure(vagrantfile_api_version) do |config|
  config.vbguest.auto_update = false
  inventory.each do |group, grouphosts|
    next if (group == "justlocal")
    grouphosts['hosts'].each do |hostname, hostinfo|
      config.vm.define hostname do |node|
        node.vm.box = hostinfo['box'] ||= default_base_box
        node.vm.hostname = hostname                                       # set the hostname
        node.vm.network :private_network, ip: hostinfo['ansible_host']    # set the ip address
        ram = hostinfo['memory']                                          # set the memory
        node.vm.provider :virtualbox do |vb|
          vb.name = hostname
          vb.customize ["modifyvm", :id, "--cpuexecutioncap", cpucap, "--memory", ram.to_s]
        end
      end
    end
  end
end

the vms are created simply with the vagrant up --no-provision command and the cluster is provisioned with ansible.

3. the playbook

the main playbook will simply delegate the specific app provisioning to roles while overriding some defaults as needed. we override the port variables in the main playbook so we can see they match up as per our schematic for the cluster. some other variables are overridden in group_vars/* files to keep them from cluttering the main playbook. the cluster is provisioned with ansible-playbook -i inventory.yml elk.yml where elk.yml is the file below.

- hosts: es-master-nodes
  become: true
  roles:
    - { role: elastic.elasticsearch, cluster_http_port: 9201, cluster_transport_tcp_port: 9301}

- hosts: es-data-nodes
  become: true
  roles:
    - { role: elastic.elasticsearch, cluster_http_port: 9201, cluster_transport_tcp_port: 9301}

- hosts: kibana-nodes
  become: true
  roles:
    - { role: ashokc.logstash, kibana_server_port: 5601, cluster_http_port: 9201 }

- hosts: logstash-nodes
  become: true
  roles:
    - { role: ashokc.logstash, cluster_http_port: 9201, filebeat_2_logstash_port: 5044 }

- hosts: filebeat-nodes
  become: true
  roles:
    - {role: ashokc.filebeat, filebeat_2_logstash_port: 5044 }

the directory layout shows a glimpse of all that is under the hood.

.
├── elk.yml
├── group_vars
│ ├── all.yml
│ ├── es-data-nodes.json
│ ├── es-master-nodes.json
│ ├── filebeat-nodes.yml
│ ├── kibana-nodes.yml
│ └── logstash-nodes.yml
├── inventory.yml
├── roles
│ ├── ashokc.filebeat
│ ├── ashokc.kibana
│ ├── ashokc.logstash
│ └── elastic.elasticsearch
└── vagrantfile

common variables for all the host groups are specified in groups_vars/all.yml. the variable ' public_iface ' can vary depending on the vm provider. for vagrant here, it is "eth1." we use that to pull out the ip address of the host from ansible_facts whenever it's required in the playbook. the file groups_vars/all.yml , in our case, will be:

public_iface: eth1   # for vagrant provider
elk_version: 5.6.1
es_major_version: 5.x
es_apt_key: https://artifacts.elastic.co/gpg-key-elasticsearch
es_version: "{{ elk_version }}"
es_apt_url: deb https://artifacts.elastic.co/packages/{{ es_major_version }}/apt stable main

3.1 elasticsearch

the provisioning of elasticsearch on master and data nodes is delegated to the excellent role elastic.elasticsearch published by elastic.co. as the role allows for multiple instances of es on a host, we name the instances, "{{cluster_http_port}}_{{cluster_transport_port}}" which would be a unique identifier. the es cluster itself is taken to be defined by this pair of ports that are used by all the master/data members of the cluster. if we rerun the playbook with a separate pair, say 9202 and 9302, we will get a second cluster, '9202_9302' (in addition to '9201_9301' that we get here on the first run) on the same set of hosts, and all would work fine.

the master node configuration variables are in the file group_vars/es-master-nodes.json shown below. the key useful thing here are the lines 5, 13, and 14, where we derive the "network.host" and "discovery.zen.ping.unicast.hosts" settings for elasticsearch from the information in the inventory file.

{
  "es_java_install" : true,
  "es_api_port": "{{cluster_http_port}}",
  "es_instance_name" : "{{cluster_http_port}}_{{cluster_transport_tcp_port}}",
  "masterhosts_transport" : "{% for host in groups['es-master-nodes'] %} {{hostvars[host]['ansible_'+public_iface]['ipv4']['address'] }}:{{cluster_trans
port_tcp_port}}{%endfor %}",
  "es_config": {
    "cluster.name": "{{es_instance_name}}",
    "http.port": "{{cluster_http_port}}",
    "transport.tcp.port": "{{cluster_transport_tcp_port}}",
    "node.master": true,
    "node.data": false,
    "network.host": ["{{ hostvars[inventory_hostname]['ansible_' + public_iface]['ipv4']['address'] }}","_local_" ],
    "discovery.zen.ping.unicast.hosts" : "{{ masterhosts_transport.split() }}"
  }
}

the data node configuration variables are very similar in the file group_vars/es-data-nodes.json below . the lines 2, 12, and 13 show the only changes.

{
  "es_data_dirs" : "/opt/elasticsearch",
  "es_java_install" : true,
  "es_api_port": "{{cluster_http_port}}",
  "es_instance_name" : "{{cluster_http_port}}_{{cluster_transport_tcp_port}}",
  "masterhosts_transport" : "{% for host in groups['es-master-nodes'] %} {{hostvars[host]['ansible_'+public_iface]['ipv4']['address'] }}:{{cluster_trans
port_tcp_port}}{%endfor %}",
  "es_config": {
    "cluster.name": "{{es_instance_name}}",
    "http.port": "{{cluster_http_port}}",
    "transport.tcp.port": "{{cluster_transport_tcp_port}}",
    "node.master": false,
    "node.data": true,
    "network.host": ["{{ hostvars[inventory_hostname]['ansible_' + public_iface]['ipv4']['address'] }}","_local_" ],
    "discovery.zen.ping.unicast.hosts" : "{{ masterhosts_transport.split() }}"
  }
}

3.2 logstash

logstash is provisioned with the role ashokc.logstash . the default variables for this role are overridden with group_vars/logstash-nodes.yml. lines 4-5 specify the user and group that own this instance of logstash. lines 9 and 10 derive the elasticsearch urls from the inventory file. it will be used for configuring elasticsearch output sections.

group_vars/logstash-nodes.yml

es_java_install: true
update_java: false
logstash_version: "{{ elk_version }}"
logstash_user: logstashuser
logstash_group: logstashgroup
logstash_enabled_on_boot: yes
logstash_install_plugins:
  - logstash-input-beats
esmasterhosts: "{% for host in groups['es-master-nodes'] %} http://{{hostvars[host]['ansible_'+public_iface]['ipv4']['address'] }}:{{cluster_http_port}}
{% endfor %}"
logstash_es_urls : "{{ esmasterhosts.split() }}"

a simple elasticsearch output config and filebeat input config are enabled with the following.

roles/ashokc.logstash/templates/conf/elasticsearch-output.conf.j2

output {
  elasticsearch {
    hosts => {{ logstash_es_urls | to_json }}
  }
}

roles/ashokc.logstash/templates/conf/beats-input.conf.j2

input {
  beats {
    port => {{filebeat_2_logstash_port}}
  }
}

3.3 kibana

kibana is provisioned with the role ashokc.kibana . the default variables for this role are again overridden with group_vars/kibana-nodes.yml. unlike logstash, it is quite common to run multiple kibana servers on a single host with each instance targeting a separate es cluster. this role allows for that and identifies the kibana instance with the port it is running at (line # 7). lines 2 and 3 specify the owner/group for the instance.

group_vars/kibana-nodes.yml

kibana_version: "{{ elk_version }}"
kibana_user: kibanauser
kibana_group: kibanagroup
kibana_enabled_on_boot: yes
kibana_server_host: 0.0.0.0
kibana_elasticsearch_url : http://{{hostvars[groups['es-master-nodes'][0]]['ansible_'+public_iface]['ipv4']['address'] }}:{{cluster_http_port}}
kibana_instance: "{{kibana_server_port}}"

the template file for 'kibana.yml ' below picks up the correct elasticsearch cluster url from below

roles/ashokc.kibana/templates/kibana.yml.j2

server.port: {{ kibana_server_port }}
server.host: {{ kibana_server_host }}
elasticsearch.url: {{ kibana_elasticsearch_url }}
pid.file: {{ kibana_pid_file }}
logging.dest: {{ kibana_log_file }}

3.4 filebeat

filebeat is provisioned with the role ashokc.filebeat the default variables are overridden in groups_vars/filebeat-nodes.yml below. lines 5 and 7 figure out the logstash connection to use.

filebeat_version: "{{ elk_version }}"
filebeat_enabled_on_boot: yes
filebeat_user: filebeatuser
filebeat_group: filebeatgroup
logstashhostslist: "{% for host in groups['logstash-nodes'] %} {{hostvars[host]['ansible_'+public_iface]['ipv4']['address'] }}:{{filebeat_2_logstash_por
t}}{% endfor %}"
filebeat_logstash_hosts: "{{ logstashhostslist.split() }}"

line #14 in the template for the sample filebeat.yml below configures the output to our logstash host at the right port.

filebeat.prospectors:
- type: log
  enabled: true
  paths:
   - /tmp/custom.log
  fields:
    log_type: custom
    type: {{ansible_hostname}}
    from: beats
  multiline.pattern: '^\s[+]{2}\scontinuing .*'
  multiline.match: after
output.logstash:
  hosts:
    {{ filebeat_logstash_hosts | to_nice_yaml }}

4. logs

the last step would be to run an application on the filebeat nodes and watch the logs flow into kibana. our application would simply be a perl script that writes the log file /tmp/custom.log. we log in to each of the filebeat hosts and run the following perl script.

#!/usr/bin/perl -w
use strict ;
no warnings 'once';
my @codes = qw (fatal error warning info debug trace) ;
open(my $fh, ">>", "/tmp/custom.log") ;
$fh->autoflush(1);
my $now = time();
for my $i (1 .. 100) {
  my $message0 = "type: customlog: this is a generic message # $i for testing elk" ;
  my $ndays = int(rand(5)) ;
  my $nhrs = int(rand(24)) ;
  my $nmins = int(rand(60)) ;
  my $nsecs = int(rand(60)) ;
  my $timevalue = $now - $ndays * 86400 - $nhrs * 3600 - $nmins * 60 - $nsecs ;
  my $now1 = localtime($timevalue) ;
  my $nmulti = int(rand(10)) ;
  my $message = "$now1 $ndays:$nhrs:$nmins:$nsecs $nmulti:$codes[int(rand($#codes))] $message0" ;
  if ($nmulti > 0) {
    for my $line (1 .. $nmulti) {
       $message = $message . "\n ++ continuing the previous line for this log error..."
    }
  }
  print $fh "$message\n" ;
}
close $fh ;

the corresponding sample logstash config file for processing this log would be placed at roles/ashokc.logstash/files/custom-filter.conf

filter {
  if [fields][log_type] == "custom" {
    grok {
      match => [ "message", "(?<matched-timestamp>\w{3}\s+\w{3}\s+\d{1,2}\s+\d{1,2}:\d{1,2}:\d{1,2}\s+\d{4})\s+(?<ndays>\d{1,3}):(?<nhrs>\d{1,2}):(?<nmi
ns>\d{1,2}):(?<nsecs>\d{1,2})\s+(?<nlines>\d{1,2}):(?<code>\w+) type: (?<given-type>\w+):[^#]+# (?<messageid>\d+)\s+%{greedydata}" ]
      add_tag => ["grokked"]
      add_field => { "foo_%{ndays}" => "hello world, from %{nhrs}" }
    }
    mutate {
      gsub => ["message", "elk", "bulk"]
    }
    date {
      match => [ "timestamp" , "eee mmm d h:m:s y", "eee mmm  d h:m:s y" ]
      add_tag => ["dated"]
    }
  }
}

conclusion

by placing appropriate filter files for logstash at roles/ashokc.logstash/files and prospector config file for filebeat at roles/ashokc.filebeat/templates/filebeat.yml.j2, one can use this elk stack to analyze application logs. a variety of extensions are possible, for example enabling x-pack login/security, other distributions and versions for 'ashokc' roles, automated testing etc... but then there is always more to be done, isn't there?