Monitor Docker Swarm and Other Logs Using Filebeat, Logstash and AWS ES — Part 2

In my previous article, I described how to install Logstash and Filebeat on Ubuntu 16.04, along with the configuration of Logstash to send logs to AWS ES. The logs were sent to Logstash from Filebeat (from Docker Swarm nodes) and from other files, e.g. Jenkins logs, to Logstash as per Scenario 1.

If you have not gone through that article, I recommend you read it before reading this one because I will refer to it many times.

Scenario 2

1. Send  Docker Swarm logs directly to AWS Elasticsearch from Filebeat.

2. Send Jenkins logs to AWS Elasticsearch using Logstash.

Below diagram shows the flow of Scenario 2:

In this article, I am not going to narrate the installation part (refer to Part 1 for installation). I will just describe the configurations that need to be done to implement Scenario 2.

Logstash Config to Send Jenkins Logs to AWS ES

In this scenario, Logstash is just sending Jenkins logs to AWS ES so the configuration is very simple and straightforward.

/etc/logstash/conf.d/logstash.conf:

input {
    file{
        path => "/var/log/jenkins/jenkins.log"
        type => "jenkins"
   }
}

#filter {
#    grok {
#        match => { "source" => "%{GREEDYDATA}/%{GREEDYDATA:app}.log" }
#    }
#}

output {
    if [type] == "jenkins" {
      amazon_es{
        hosts => ["search-msdemo-logs-xfyuwloilsxmyn4ldgyzufundy.us-east-1.es.amazonaws.com"]
        region => "us-east-1"
        index => "jenkinslogs-%{+YYYY.MM.dd}"
      }
   }
}

Please refer to Part 1 to see how to install the amazon_es plugin, which will allow Logstash to connect to AWS ES.

Filebeat Config to Send Docker Swarm Logs to AWS ES

In this scenario, we are going to send Docker Swarm logs directly to AWS ES and Kibana from filebeat. Below is the filebeat.yml file configuration which will connect to AWS ES and Kibana.

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - '/var/lib/docker/containers/*/*.log'
  json.message_key: log
  json.keys_under_root: true
  processors:
   - add_docker_metadata: ~

setup.kibana:
  host: https://search-domainname-xksdfsdkjhfifwsffssuire.us-east-1.es.amazonaws.com/_plugin/kibana/

output.elasticsearch:
  hosts: ["https://search-domainname-xksdfsdkjhfifwsffssuire.us-east-1.es.amazonaws.com:443"]

You might have noticed that there is only one section which is not available, "output.logstash," in the filebeat.yml file. The only change you will notice is the "setup.kibana" and "output.elasticsearch" host configuration.

setup.kibana:
  host: https://search-domainname-xksdfsdkjhfifwsffssuire.us-east-1.es.amazonaws.com/_plugin/kibana/

output.elasticsearch:
  hosts: ["https://search-domainname-xksdfsdkjhfifwsffssuire.us-east-1.es.amazonaws.com:443"]

Both the hosts' links (Kibana and Elasticsearch) are available on the AWS ES dashboard. There is no security implemented while creating AWS ES services (Kibana and Elasticsearch) but it's highly recommended for any environment.

The links can be directly accessed from EC2 instances and logs are fed to the search see the screen below:

Once the above configurations are done, you are all set you see log messages in Kibana.