{{announcement.body}}
{{announcement.title}}

How to Set Up and Run Kafka on Kubernetes

DZone 's Guide to

How to Set Up and Run Kafka on Kubernetes

In this tutorial we are going to see an example Kafka deployment within Platform9 Free Tier Kubernetes platform, backed up by some DigitalOcean droplets.

· Cloud Zone ·
Free Resource

Apache Kafka is a leading open-source distributed streaming platform first developed at LinkedIn. It consists of several APIs such as the Producer, the Consumer, the Connector and the Streams. Together, those systems act as high-throughput, low-latency platforms for handling real-time data. This is why Kafka is preferred among several of the top-tier tech companies such as Uber, Zalando, and AirBnB.

Quite often, we would like to deploy a fully-fledged Kafka cluster in Kubernetes, just because we have a collection of microservices and we need a resilient message broker in the center. We also want to spread the Kafka instances across nodes, to minimize the impact of a failure.

In this tutorial we are going to see an example Kafka deployment within Platform9 Free Tier Kubernetes platform, backed up by some DigitalOcean droplets.

Let’s get started.

Setting Up the Platform9 Free Tier Cluster

Below are the brief instructions to get you up and running with a working Kubernetes Cluster from Platform9:

  1. Sign up with Platform9

  2. Click the Create Cluster button and inspect the instructions. We need a server to host the Cluster.

  3. Create a few Droplets with at least 3GB Ram and 2vCPU’s. Follow instructions to install the pf9cli tool and prepping the nodes.

Java
 




x


 
1
 $ bash <(curl -sL http://pf9.io/get_cli) $ pf9ctl cluster prep-node -i <nodeIP> 



  1. Switch to the Platform9 UI and click the refresh button. You should see the new nodes in the list. Assign the first node as a master and the other ones as workers.

  2. Leave the default values in the next steps. Then create the cluster.

  3. Wait until the cluster becomes healthy. It will take at least 20 minutes to finish.

  4. Click on the API Access tab and select to download the kubeconfig button:

Download kubeconfig

Download kubeconfig


  1. Once downloaded, export the config and test the cluster health:


Shell
 




x


 
1
$ export KUBECONFIG=/Users/itspare/Theo/Projects/platform9/example.yaml
2
$ kubectl cluster-info
3
 
          
4
Kubernetes master is running at https://134.122.106.235
5
CoreDNS is running at https://134.122.106.235/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
6
Metrics-server is running at https://134.122.106.235/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
7
 
          


Creating Persistent Volumes

Before we install Helm and the Kafka chart, we need to create some persistent volumes for storing Kafka replication message files.

This step is crucial to be able to enable persistence in our cluster because without that, the topics and messages would disappear after we shut down any of the servers, as they live in memory.

In our example, we are going to use a local file system, Persistent Volume (PV), and we need one persistent volume for each Kafka instance; so if we plan to deploy three instances, we need three PV’s.

Create and apply first the Kafka namespace and the PV specs:

YAML
 




x


 
1
$ cat namespace.yml
2
---
3
apiVersion: v1
4
kind: Namespace
5
metadata:
6
  name: kafka
7
 
          
8
$ kubectl apply -f namespace.yml
9
namespace/kafka created


YAML
 




xxxxxxxxxx
1
48


 
1
$ cat pv.yml
2
---
3
apiVersion: v1
4
kind: PersistentVolume
5
metadata:
6
  name: kafka-pv-volume
7
  labels:
8
    type: local
9
spec:
10
  storageClassName: manual
11
  capacity:
12
    storage: 10Gi
13
  accessModes:
14
    - ReadWriteOnce
15
  hostPath:
16
    path: "/mnt/data"
17
---
18
apiVersion: v1
19
kind: PersistentVolume
20
metadata:
21
  name: kafka-pv-volume-2
22
  labels:
23
    type: local
24
spec:
25
  storageClassName: manual
26
  capacity:
27
    storage: 10Gi
28
  accessModes:
29
    - ReadWriteOnce
30
  hostPath:
31
    path: "/mnt/data"
32
---
33
apiVersion: v1
34
kind: PersistentVolume
35
metadata:
36
  name: kafka-pv-volume-3
37
  labels:
38
    type: local
39
spec:
40
  storageClassName: manual
41
  capacity:
42
    storage: 10Gi
43
  accessModes:
44
    - ReadWriteOnce
45
  hostPath:
46
    path: "/mnt/data"
47
 
          
48
$ kubectl apply -f pv.yml


If you are using the Kubernetes UI, you should be able to see the PV volumes on standby:

Persistent volumes on standby

Persistent volumes on standby


Installing Helm

We begin by installing Helm on our computer and installing it in Kubernetes, as it’s not bundled by default.

First, we download the install script:

Shell
 




xxxxxxxxxx
1


 
1
$ curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > install-helm.sh


 

Make the script executable with chmod:

Shell
 




xxxxxxxxxx
1


 
1
$ chmod u+x install-helm.sh 



Create the tiller service account:

Shell
 




xxxxxxxxxx
1


 
1
 $ kubectl -n kube-system create serviceaccount tiller 



Next, bind the tiller serviceaccount to the cluster-admin role:

Shell
 




xxxxxxxxxx
1


 
1
$ kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller



Now we can run  helm init :

Shell
 




xxxxxxxxxx
1


 
1
 $ helm init --service-account tiller 


Now we are ready to install the Kafka chart.

Deploying the Helm Chart

In the past, trying to deploy Kafka on Kubernetes was a lengthy exercise. You had to deploy a working Zookeeper Cluster, role bindings, persistent volume claims and apply the correct configuration.

Hopefully for us, with the use of the Kafka Incubator Chart, the whole process is mostly automated (with a few quirks here and there).

We add the Helm chart:

Shell
 




xxxxxxxxxx
1


 
1
  $ helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator 



Export the chart values in a file:

Shell
 




xxxxxxxxxx
1


 
1
 $ curl https://raw.githubusercontent.com/helm/charts/master/incubator/kafka/values.yaml > config.yml


 

Carefully inspect the configuration values, particularly around the parts about persistence and about the number of Kafka stateful sets to deploy.

Then install the chart: 

Shell
 




xxxxxxxxxx
1


 
1
$ helm install --name kafka-demo --namespace kafka incubator/kafka -f values.yml --debug 


 

Check the status of the deployment

Shell
 




xxxxxxxxxx
1
67


 
1
 $ helm status kafka-demo 
2
 
          
3
LAST DEPLOYED: Sun Apr 19 14:05:15 2020
4
 
          
5
NAMESPACE: kafka
6
 
          
7
STATUS: DEPLOYED
8
 
          
9
 
          
10
 
          
11
RESOURCES:
12
 
          
13
==> v1/ConfigMap
14
 
          
15
NAME                  DATA  AGE
16
 
          
17
kafka-demo-zookeeper  3     5m29s
18
 
          
19
 
          
20
 
          
21
==> v1/Pod(related)
22
 
          
23
NAME                    READY  STATUS   RESTARTS  AGE
24
 
          
25
kafka-demo-zookeeper-0  1/1    Running  0         5m28s
26
 
          
27
kafka-demo-zookeeper-1  1/1    Running  0         4m50s
28
 
          
29
kafka-demo-zookeeper-2  1/1    Running  0         4m12s
30
 
          
31
kafka-demo-zookeeper-0  1/1    Running  0         5m28s
32
 
          
33
kafka-demo-zookeeper-1  1/1    Running  0         4m50s
34
 
          
35
kafka-demo-zookeeper-2  1/1    Running  0         4m12s
36
 
          
37
 
          
38
 
          
39
==> v1/Service
40
 
          
41
NAME                           TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)                     AGE
42
 
          
43
kafka-demo                     ClusterIP  10.21.255.214  <none>       9092/TCP                    5m29s
44
 
          
45
kafka-demo-headless            ClusterIP  None           <none>       9092/TCP                    5m29s
46
 
          
47
kafka-demo-zookeeper           ClusterIP  10.21.13.232   <none>       2181/TCP                    5m29s
48
 
          
49
kafka-demo-zookeeper-headless  ClusterIP  None           <none>       2181/TCP,3888/TCP,2888/TCP  5m29s
50
 
          
51
 
          
52
 
          
53
==> v1/StatefulSet
54
 
          
55
NAME                  READY  AGE
56
 
          
57
kafka-demo            3/3    5m28s
58
 
          
59
kafka-demo-zookeeper  3/3    5m28s
60
 
          
61
 
          
62
 
          
63
==> v1beta1/PodDisruptionBudget
64
 
          
65
NAME                  MIN AVAILABLE  MAX UNAVAILABLE  ALLOWED DISRUPTIONS  AGE
66
 
          
67
kafka-demo-zookeeper  N/A            1                1                    5m29s



During this phase, you may want to navigate to the Kubernetes UI and inspect the dashboard for any issues. Once everything is complete, then the pods and Persistent Volume Claims should be bound and green.

Now we can test the Kafka cluster.

Testing the Kafka Cluster

We are going to deploy a test client that will execute scripts against the Kafka cluster.

Create and apply the following deployment:

YAML
 




xxxxxxxxxx
1
29


 
1
$ cat testclient.yml
2
 
          
3
apiVersion: v1
4
 
          
5
kind: Pod
6
 
          
7
metadata:
8
 
          
9
  name: testclient
10
 
          
11
  namespace: kafka
12
 
          
13
spec:
14
 
          
15
  containers:
16
 
          
17
  - name: kafka
18
 
          
19
    image: solsson/kafka:0.11.0.0
20
 
          
21
    command:
22
 
          
23
      - sh
24
 
          
25
      - -c
26
 
          
27
      - "exec tail -f /dev/null"
28
 
          
29
$ kubectl apply -f testclient



Then, using the testclient, we create the first topic, which we are going to use to post messages:

Shell
 




xxxxxxxxxx
1


 
1
$ kubectl -n kafka exec -ti testclient -- ./bin/kafka-topics.sh --zookeeper kafka-demo-zookeeper:2181 --topic messages --create --partitions 1 --replication-factor 1


Created topic "messages."

Here we need to use the correct hostname for the zookeeper cluster and the topic configuration.

Next, verify that the topic exists:

Shell
 




xxxxxxxxxx
1


 
1
$ kubectl -n kafka exec -ti testclient -- ./bin/kafka-topics.sh --zookeeper kafka-demo-zookeeper:2181 --list
2
Messages



Now we can create one consumer and one producer instance so that we can send and consume messages.

First create one or two listeners, each on its own shell:

Shell
 




xxxxxxxxxx
1


 
1
$ kubectl -n kafka exec -ti testclient -- ./bin/kafka-console-consumer.sh --bootstrap-server kafka-demo:9092 --topic messages --from-beginning


Then create the producer session and type some messages. You will be able to see them propagate to the consumer sessions:

Shell
 




xxxxxxxxxx
1
21


 
1
$ kubectl -n kafka exec -ti testclient -- ./bin/kafka-console-producer.sh --broker-list kafka-demo:9092 --topic messages
2
 
          
3
>Hi
4
 
          
5
>How are you?
6
 
          
7
>Hope you're well
8
 
          
9
>
10
 
          
11
 
          
12
 
          
13
<switching on each consumer you will see>
14
 
          
15
 
          
16
 
          
17
Hi
18
 
          
19
How are you?
20
 
          
21
Hope you're well



Destroying the Helm Chart

To clean up our resources, we just destroy the Helm Chart and delete the PVs we created earlier:

Shell
 




xxxxxxxxxx
1


 
1
$ helm delete kafka-demo --purge
2
 
          
3
$ kubectl delete -f pv.yml -n kafka



Next Steps

Stay put for more tutorials showcasing common deployment scenarios within Platform9’s fully-managed Kubernetes platform.

Topics:
apache, apache kafka, cluster, digitalocean, kafka, kafka tutorial, kubernetes

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}