{{announcement.body}}
{{announcement.title}}

EFK Stack on Kubernetes (Part 2)

DZone 's Guide to

EFK Stack on Kubernetes (Part 2)

In this tutorial, we will learn about configuring Filebeat to run as a DaemonSet in our Kubernetes cluster in order to ship logs to the Elasticsearch backend.

· Cloud Zone ·
Free Resource

This is the final part of our Kubernetes logging series. In case you missed part 1, you can find it here. In this tutorial, we will learn about configuring Filebeat to run as a DaemonSet in our Kubernetes cluster in order to ship logs to the Elasticsearch backend. We are using Filebeat instead of FluentD or FluentBit because it is an extremely lightweight utility and has a first-class support for Kubernetes. It is best for production-level setups.

Deployment Architecture

Filebeat will run as a DaemonSet in our Kubernetes cluster. It will be:

  • Deployed in a separate namespace called Logging.
  • Pods will be scheduled on both Master nodes and Worker Nodes.
  • Master Node pods will forward api-server logs for audit and cluster administration purposes.
  • Client Node pods will forward workload related logs for application observability.

Creating Filebeat ServiceAccount and ClusterRole

Deploy the following manifest to create the required permissions for Filebeat pods.

Shell
 




x
43


 
1
apiVersion: v1
2
kind: Namespace
3
metadata:
4
  name: logging
5
---
6
apiVersion: v1
7
kind: ServiceAccount
8
metadata:
9
  name: filebeat
10
  namespace: logging
11
  labels:
12
    k8s-app: filebeat
13
---
14
apiVersion: rbac.authorization.k8s.io/v1beta1
15
kind: ClusterRole
16
metadata:
17
  name: filebeat
18
  namespace: logging
19
  labels:
20
    k8s-app: filebeat
21
rules:
22
- apiGroups: [""] # "" indicates the core API group
23
  resources:
24
  - namespaces
25
  - pods
26
  verbs:
27
  - get
28
  - watch
29
  - list
30
---
31
apiVersion: rbac.authorization.k8s.io/v1beta1
32
kind: ClusterRoleBinding
33
metadata:
34
  name: filebeat
35
  namespace: logging
36
subjects:
37
- kind: ServiceAccount
38
  name: filebeat
39
  namespace: kube-system
40
roleRef:
41
  kind: ClusterRole
42
  name: filebeat
43
  apiGroup: rbac.authorization.k8s.io


We should make sure that ClusterRole permissions are as limited as possible from a security point of view. If either of the pod associated with this service account gets compromised then the attacker would not be able to gain access entire cluster or applications running in it.

Creating Filebeat ConfigMap

Use the following manifest to create a ConfigMap which will be used by Filebeat pods.

Shell
 




xxxxxxxxxx
1
65


 
1
apiVersion: v1
2
kind: Namespace
3
metadata:
4
  name: logging
5
---
6
apiVersion: v1
7
kind: ConfigMap
8
metadata:
9
  name: filebeat-config
10
  namespace: logging
11
  labels:
12
    k8s-app: filebeat
13
    kubernetes.io/cluster-service: "true"
14
data:
15
  filebeat.yml: |-
16
    filebeat.config:
17
    #  inputs:
18
    #    path: ${path.config}/inputs.d/*.yml
19
    #    reload.enabled: true
20
      modules:
21
        path: ${path.config}/modules.d/*.yml
22
        reload.enabled: true
23
 
           
24
    filebeat.autodiscover:
25
      providers:
26
        - type: kubernetes
27
          hints.enabled: true
28
          include_annotations: ["artifact.spinnaker.io/name","ad.datadoghq.com/tags"]
29
          include_labels: ["app.kubernetes.io/name"]
30
          labels.dedot: true
31
          annotations.dedot: true
32
          templates:
33
            - condition:
34
                equals:
35
                  kubernetes.namespace: myapp   #Set the namespace in which your app is running, can add multiple conditions in case of more than 1 namespace.
36
              config:
37
                - type: docker
38
                  containers.ids:
39
                    - "${data.kubernetes.container.id}"
40
                  multiline:
41
                    pattern: '^[A-Za-z ]+[0-9]{2} (?:[01]\d|2[0123]):(?:[012345]\d):(?:[012345]\d)'.   #Timestamp regex for the app logs. Change it as per format. 
42
                    negate: true
43
                    match: after
44
            - condition:
45
                equals:
46
                  kubernetes.namespace: elasticsearch
47
              config:
48
                - type: docker
49
                  containers.ids:
50
                    - "${data.kubernetes.container.id}"
51
                  multiline:
52
                    pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}|^[0-9]{4}-[0-9]{2}-[0-9]{2}T'
53
                    negate: true
54
                    match: after
55
                    
56
    processors:
57
      - add_cloud_metadata: ~
58
      - drop_fields:
59
          when:
60
            has_fields: ['kubernetes.labels.app']
61
          fields:
62
            - 'kubernetes.labels.app'
63
 
           
64
    output.elasticsearch:
65
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']


Important concepts for the Filebeat ConfigMap:

  • hints.enabled: This activates Filebeat’s hints module for Kubernetes. By using this we can use pod annotations to pass config directly to Filebeat pod. We can specify different multiline patterns and various other types of config. More about this can be read here.
  • include_annotations: Setting this to true enables Filebeat to retain any pod annotation for a particular log entry. These annotations can be later used to filter logs in the Kibana console.
  • include_labels: Setting this to true enables Filebeat to retain any pod labels for a particular log entry. These labels can be later used to filter logs in the Kibana console.
  • We can also filter logs for a particular namespace and then can process the log entries accordingly. Here docker log processor is used. We can also use different multiline patterns for different namespaces.
  • The output is set to Elasticsearch because we are using Elasticsearch as the storage backend. Alternatively, this can also point to Redis, Logstash, Kafka or even a File. More this can be read here.
  • Cloud metadata processor includes some host-specific fields in the log entry. This is helpful when we try to filter logs specific to a particular worker node.

Deploying Filebeat DaemonSet

Use the manifest below to deploy the Filebeat DaemonSet.

Shell
 




xxxxxxxxxx
1
76


1
apiVersion: v1
2
kind: Namespace
3
metadata:
4
  name: logging
5
---
6
apiVersion: extensions/v1beta1
7
kind: DaemonSet
8
metadata:
9
  name: filebeat
10
  namespace: logging
11
  labels:
12
    k8s-app: filebeat
13
spec:
14
  template:
15
    metadata:
16
      labels:
17
        k8s-app: filebeat
18
    spec:
19
      serviceAccountName: filebeat
20
      terminationGracePeriodSeconds: 30
21
      tolerations:
22
      - effect: NoSchedule
23
        key: node-role.kubernetes.io/master
24
      containers:
25
      - name: filebeat
26
        image: elastic/filebeat:6.5.4
27
        args: [
28
          "-c", "/usr/share/filebeat/filebeat.yml",
29
          "-e",
30
        ]
31
        env:
32
        - name: ELASTICSEARCH_HOST
33
          value: elasticsearch.elasticsearch
34
        - name: ELASTICSEARCH_PORT
35
          value: "9200"
36
        securityContext:
37
          runAsUser: 0
38
          # If using Red Hat OpenShift uncomment this:
39
          #privileged: true
40
        resources:
41
          limits:
42
            memory: 200Mi
43
          requests:
44
            cpu: 100m
45
            memory: 100Mi
46
        volumeMounts:
47
        - name: config
48
          mountPath: /usr/share/filebeat/filebeat.yml
49
          readOnly: true
50
          subPath: filebeat.yml
51
        - name: inputs
52
          mountPath: /usr/share/filebeat/inputs.d
53
          readOnly: true
54
        - name: data
55
          mountPath: /usr/share/filebeat/data
56
        - name: varlibdockercontainers
57
          mountPath: /var/lib/docker/containers
58
          readOnly: true
59
      volumes:
60
      - name: config
61
        configMap:
62
          defaultMode: 0600
63
          name: filebeat-config
64
      - name: varlibdockercontainers
65
        hostPath:
66
          path: /var/lib/docker/containers
67
      - name: inputs
68
        configMap:
69
          defaultMode: 0600
70
          name: filebeat-inputs
71
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
72
      - name: data
73
        hostPath:
74
          path: /var/lib/filebeat-data
75
          type: DirectoryOrCreate
76
---


Let’s see what is going on here:

  • Logs for each pod are written to /var/log/docker/containers. We are mounting this directory from the host to the Filebeat pod and then Filebeat processes the logs according to the provided configuration.
  • We have set the env var ELASTICSEARCH_HOST to elasticsearch.elasticsearch to refer to the Elasticsearch client service which was created in part 1 of this article. In case you already have an Elasticsearch cluster running, the env var should be set to point to it.

Please note the following setting in the manifest:

Shell
 




xxxxxxxxxx
1


1
...
2
      tolerations:
3
      - effect: NoSchedule
4
        key: node-role.kubernetes.io/master
5
...


This makes sure that our Filebeat DaemonSet schedules a pod on the master node as well. Once the Filebeat DaemonSet is deployed we can check if our pods get scheduled properly.

Shell
 




xxxxxxxxxx
1
19


1
root$ kubectl -n logging get pods  -o wide
2
NAME             READY   STATUS    RESTARTS   AGE   IP            NODE                                         NOMINATED NODE   READINESS GATES
3
filebeat-4kchs   1/1     Running   0          6d    100.96.8.2    ip-10-10-30-206.us-east-2.compute.internal   <none>           <none>
4
filebeat-6nrpc   1/1     Running   0          6d    100.96.7.6    ip-10-10-29-252.us-east-2.compute.internal   <none>           <none>
5
filebeat-7qs2s   1/1     Running   0          6d    100.96.1.6    ip-10-10-30-161.us-east-2.compute.internal   <none>           <none>
6
filebeat-j5xz6   1/1     Running   0          6d    100.96.5.3    ip-10-10-28-186.us-east-2.compute.internal   <none>           <none>
7
filebeat-pskg5   1/1     Running   0          6d    100.96.64.4   ip-10-10-29-142.us-east-2.compute.internal   <none>           <none>
8
filebeat-vjdtg   1/1     Running   0          6d    100.96.65.3   ip-10-10-30-118.us-east-2.compute.internal   <none>           <none>
9
filebeat-wm24j   1/1     Running   0          6d    100.96.0.4    ip-10-10-28-162.us-east-2.compute.internal   <none>           <none>
10
 
           
11
root$ kubectl -get nodes -o wide
12
NAME                                         STATUS   ROLES    AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION   CONTAINER-RUNTIME
13
ip-10-10-28-162.us-east-2.compute.internal   Ready    master   6d    v1.14.8   10.10.28.162   <none>        Debian GNU/Linux 9 (stretch)   4.9.0-9-amd64    docker://18.6.3
14
ip-10-10-28-186.us-east-2.compute.internal   Ready    node     6d    v1.14.8   10.10.28.186   <none>        Debian GNU/Linux 9 (stretch)   4.9.0-9-amd64    docker://18.6.3
15
ip-10-10-29-142.us-east-2.compute.internal   Ready    master   6d    v1.14.8   10.10.29.142   <none>        Debian GNU/Linux 9 (stretch)   4.9.0-9-amd64    docker://18.6.3
16
ip-10-10-29-252.us-east-2.compute.internal   Ready    node     6d    v1.14.8   10.10.29.252   <none>        Debian GNU/Linux 9 (stretch)   4.9.0-9-amd64    docker://18.6.3
17
ip-10-10-30-118.us-east-2.compute.internal   Ready    master   6d    v1.14.8   10.10.30.118   <none>        Debian GNU/Linux 9 (stretch)   4.9.0-9-amd64    docker://18.6.3
18
ip-10-10-30-161.us-east-2.compute.internal   Ready    node     6d    v1.14.8   10.10.30.161   <none>        Debian GNU/Linux 9 (stretch)   4.9.0-9-amd64    docker://18.6.3
19
ip-10-10-30-206.us-east-2.compute.internal   Ready    node     6d    v1.14.8   10.10.30.206   <none>        Debian GNU/Linux 9 (stretch)   4.9.0-9-amd64    docker://18.6.3


If we tail the logs for one of the pods we can clearly see that it connected to Elasticsearch and has started harvester for the files. The snippet below shows this:

Shell
 




xxxxxxxxxx
1
13


 
1
2019-11-19T06:22:03.435Z    INFO    log/input.go:138    Configured paths: [/var/lib/docker/containers/c2b29f5e06eb8affb2cce7cf2501f6f824a2fd83418d09823faf4e74a5a51eb7/*.log]
2
2019-11-19T06:22:03.435Z    INFO    input/input.go:114  Starting input of type: docker; ID: 4134444498769889169 
3
2019-11-19T06:22:04.786Z    INFO    input/input.go:149  input ticker stopped
4
2019-11-19T06:22:04.786Z    INFO    input/input.go:167  Stopping Input: 4134444498769889169
5
2019-11-19T06:22:19.295Z    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":641680,"time":{"ms":16}},"total":{"ticks":2471920,"time":{"ms":180},"value":2471920},"user":{"ticks":1830240,"time":{"ms":164}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":20},"info":{"ephemeral_id":"007e8090-7c62-4b44-97fb-e74e8177dc54","uptime":{"ms":549390018}},"memstats":{"gc_next":47281968,"memory_alloc":29021760,"memory_total":156062982472}},"filebeat":{"events":{"added":111,"done":111},"harvester":{"closed":2,"open_files":15,"running":13}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":108,"batches":15,"total":108},"read":{"bytes":69},"write":{"bytes":123536}},"pipeline":{"clients":1847,"events":{"active":0,"filtered":3,"published":108,"total":111},"queue":{"acked":108}}},"registrar":{"states":{"current":87,"update":111},"writes":{"success":18,"total":18}},"system":{"load":{"1":0.98,"15":1.71,"5":1.59,"norm":{"1":0.0613,"15":0.1069,"5":0.0994}}}}}}
6
 
           
7
2019-11-19T06:22:49.295Z    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":641720,"time":{"ms":44}},"total":{"ticks":2472030,"time":{"ms":116},"value":2472030},"user":{"ticks":1830310,"time":{"ms":72}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":20},"info":{"ephemeral_id":"007e8090-7c62-4b44-97fb-e74e8177dc54","uptime":{"ms":549420018}},"memstats":{"gc_next":47281968,"memory_alloc":38715472,"memory_total":156072676184}},"filebeat":{"events":{"active":12,"added":218,"done":206},"harvester":{"open_files":15,"running":13}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":206,"batches":24,"total":206},"read":{"bytes":102},"write":{"bytes":269666}},"pipeline":{"clients":1847,"events":{"active":12,"published":218,"total":218},"queue":{"acked":206}}},"registrar":{"states":{"current":87,"update":206},"writes":{"success":24,"total":24}},"system":{"load":{"1":1.22,"15":1.7,"5":1.58,"norm":{"1":0.0763,"15":0.1063,"5":0.0988}}}}}}
8
 
           
9
2019-11-19T06:23:19.295Z    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":641750,"time":{"ms":28}},"total":{"ticks":2472110,"time":{"ms":72},"value":2472110},"user":{"ticks":1830360,"time":{"ms":44}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":20},"info":{"ephemeral_id":"007e8090-7c62-4b44-97fb-e74e8177dc54","uptime":{"ms":549450017}},"memstats":{"gc_next":47281968,"memory_alloc":43140256,"memory_total":156077100968}},"filebeat":{"events":{"active":-12,"added":43,"done":55},"harvester":{"open_files":15,"running":13}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":55,"batches":12,"total":55},"read":{"bytes":51},"write":{"bytes":70798}},"pipeline":{"clients":1847,"events":{"active":0,"published":43,"total":43},"queue":{"acked":55}}},"registrar":{"states":{"current":87,"update":55},"writes":{"success":12,"total":12}},"system":{"load":{"1":0.99,"15":1.67,"5":1.49,"norm":{"1":0.0619,"15":0.1044,"5":0.0931}}}}}}
10
 
           
11
2019-11-19T06:23:25.261Z    INFO    log/harvester.go:255    Harvester started for file: /var/lib/docker/containers/ccb7dc75ecc755734f6befc4965b9fdae74d59810914101eadf63daa69eb62e2/ccb7dc75ecc755734f6befc4965b9fdae74d59810914101eadf63daa69eb62e2-json.log
12
 
           
13
2019-11-19T06:23:49.295Z    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":641780,"time":{"ms":28}},"total":{"ticks":2472310,"time":{"ms":196},"value":2472310},"user":{"ticks":1830530,"time":{"ms":168}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":21},"info":{"ephemeral_id":"007e8090-7c62-4b44-97fb-e74e8177dc54","uptime":{"ms":549480018}},"memstats":{"gc_next":47789200,"memory_alloc":31372376,"memory_total":156086697176,"rss":-1064960}},"filebeat":{"events":{"active":16,"added":170,"done":154},"harvester":{"open_files":16,"running":14,"started":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":153,"batches":24,"total":153},"read":{"bytes":115},"write":{"bytes":207569}},"pipeline":{"clients":1847,"events":{"active":16,"filtered":1,"published":169,"total":170},"queue":{"acked":153}}},"registrar":{"states":{"current":87,"update":154},"writes":{"success":25,"total":25}},"system":{"load":{"1":0.87,"15":1.63,"5":1.41,"norm":{"1":0.0544,"15":0.1019,"5":0.0881}}}}}}


Once we have all our pods running, then we can create an index pattern of the type filebeat-* in Kibana. Filebeat indexes are generally timestamped. As soon as we create the index pattern, all the searchable available fields can be seen and should be imported.Lastly, we can search through our application logs and create dashboards if needed. It is highly recommended to have JSON logger in our applications because it makes log processing extremely easy and messages can be parsed easily.

Conclusion

This concludes our logging set-up. All of the provided configuration files have been tested in production environments and are readily deployable. Feel free to reach out should you have any questions around it.

This article was originally published on https://appfleet.com/blog/part-2-efk-stack-on-kubernetes/ and has been authorized by Appfleet for a republish.

Topics:
cloud, cloud native, docker, efk stack, kubernetes, tutorial

Published at DZone with permission of Sudip Sengupta . See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}