Ultimate Guide to Installing Kafka Docker on Kubernetes

In this ultimate guide I will give you a simple step-by-step tutorial on installing Kafka Docker on Kubernetes. This post includes a complete video walk-through.

There has been a lot of interest lately about deploying Kafka to a Kubernetes cluster. If you are wanting to take the deep dive yourself then you found the right article. Now that we have Kafka Docker, deploying a Kafka cluster to Kubernetes is a snap.

Deploy ZooKeeper to Kubernetes

Kafka relies on ZooKeeper to keep track of its configuration including what topics are available.

Before we deploy Kafka we need to deploy ZooKeeper.

Create a file called zookeeper.yml and add these contents:

---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: zookeeper-deployment-1
spec:
  template:
    metadata:
      labels:
        app: zookeeper-1
    spec:
      containers:
      - name: zoo1
        image: digitalwonderland/zookeeper
        ports:
        - containerPort: 2181
        env:
        - name: ZOOKEEPER_ID
          value: "1"
        - name: ZOOKEEPER_SERVER_1
          value: zoo1
---
apiVersion: v1
kind: Service
metadata:
  name: zoo1
  labels:
    app: zookeeper-1
spec:
  ports:
  - name: client
    port: 2181
    protocol: TCP
  - name: follower
    port: 2888
    protocol: TCP
  - name: leader
    port: 3888
    protocol: TCP
  selector:
    app: zookeeper-1

This creates a Kubernetes Deployment that will schedule zookeeper pods and a Kubernetes Service to route traffic to the pods. The service has a short name of zoo1 which we will use later when we deploy the Kafka Brokers.

Create the resource:

$ kubectl create -f zookeeper.yml

Now lets start deploying Kafka.

Deploying a Kafka Docker Service

The first thing we need to do is deploy a Kubernetes Service that will manage our Kafka Broker deployments.

Create a new file called kafka-service.yml and add the following contents:

---
apiVersion: v1
kind: Service
metadata:
  name: kafka-service
  labels:
    name: kafka
spec:
  ports:
  - port: 9092
    name: kafka-port
    protocol: TCP
  selector:
    app: kafka
    id: "0"
  type: LoadBalancer

You might notice that we have set the type to LoadBalancer. If your Kubernetes Cluster is deployed to bare-metal don't freak out. There is a new Kubernetes add-on called MetalLB that allows this. Checkout my article Kubernetes metallb bare metal loadbalancer for instructions on how to enable it. It will make your life much easier.

Create the service.

$ kubectl create -f kafka-service

Now we need to get the external IP for the service, because we will need it in order to spin up a Kafka Broker in the next section.

$ kubectl describe svc kafka-service
Name: kafka-service
Namespace: default
Labels: name=kafka
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"name":"kafka"},"name":"kafka-service","namespace":"default"},"spec":{"ports...
Selector: app=kafka,id=0
Type: LoadBalancer
IP: 10.105.148.62
LoadBalancer Ingress: 192.168.1.240
Port: kafka-port 9092/TCP
TargetPort: 9092/TCP
NodePort: kafka-port 30718/TCP
Endpoints: 10.44.0.4:9092
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>

In the example above, I would note that the LoadBalancer Ingress is set to 192.168.1.240. Now we can start our Kafka Broker.

Deploying the Kafka Broker to Kubernetes

We have the Kubernetes Service deployed but all it does is load balance our Kafka pods which are not deployed yet.

Follow these steps to deploy them.

Create a new file called kafka-broker.yml and add the following contents:

---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: kafka-broker0
spec:
  template:
    metadata:
      labels:
        app: kafka
        id: "0"
    spec:
      containers:
      - name: kafka
        image: wurstmeister/kafka
        ports:
        - containerPort: 9092
        env:
        - name: KAFKA_ADVERTISED_PORT
          value: "30718"
        - name: KAFKA_ADVERTISED_HOST_NAME
          value: 192.168.1.240
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: zoo1:2181
        - name: KAFKA_BROKER_ID
          value: "0"
        - name: KAFKA_CREATE_TOPICS
          value: admintome-test:1:1

Notice that the KAFKA_ADVERTISED_HOST_NAME is set to the IP address we noted earlier. Also note we tell the Kafka Broker to automatically create a topic admintome-test with 1 partition and 1 replica. You can create multiple topics using the same vernacular and separating them by commas (i.e. - topic1:1:1,topic2:1:1).

Save the file and create the resource.

$ create -f kafka-broker.yml

You can validate that everything is running.

$ kubectl get pod kafka-broker0

To scale your Kafka Brokers, create another file but give it a different name (i.e. kafka-broker1) and update the ID to match.

Lets test our Kafka Deployment

Testing With KafkaCat

We are going to test our Kafka deployment by using an application called KafkaCat.

To install:

$ apt-get install kafkacat

After the application is installed we will run it in consumer mode (which is the default).

kafkacat -b 192.168.1.240:9092 -t admintome-test

This should not show anything yet because we haven't sent anything to our topic yet...

To send stuff we can copy any text file into our current directory and send it to our Kafka Topic. In another window, run the following command.

$ cat README | kafkacat -b 192.168.1.240 -t admintome-test

You should see the output in the first window which has KafkaCat still running in consumer mode.

Congratulations! You have successfully deployed a Kafka Cluster to Kubernetes.