Kafka on Kubernetes, the Strimzi Way (Part 2)
Kafka on Kubernetes, the Strimzi Way (Part 2)
Learn how to run Kafka on Kubernetes using an open source operator.
Join the DZone community and get the full member experience.Join For Free
We kicked off the the first part of the series by setting up a single node Kafka cluster which was accessible to only internal clients within the same Kubernetes cluster, had no encryption, authentication or authorization and used temporary persistence. We will keep iterating/improving on this during the course of this blog series.
This part will cover these topics:
- Expose Kafka cluster to external applications
- Explore Kubernetes resources behind the scenes
- Use Kafka CLI and Go client applications to test our cluster setup
The code is available on GitHub - https://github.com/abhirockzz/kafka-kubernetes-strimzi/
What Do I Need to Try This Out?
I will be using Azure Kubernetes Service (AKS) to demonstrate the concepts, but by and large it is independent of the Kubernetes provider (e.g. feel free to use a local setup such as
minikube). If you want to use
AKS, all you need is a Microsoft Azure account which you can get for FREE if you don't have one already.
I will not be repeating some of the common sections (such as Installation/Setup for Helm, Strimzi, Azure Kubernetes Service as well as Strimzi overview) in this or subsequent part of this series and would request you to refer to part one for those details
To achieve this, we just need to tweak the Strimzi
Kafka resource a little bit. I am highlighting the key part below - here is the original manifest from part 1
To make Kafka accessible to external client applications, we added an
external listener of type
loadbalancer. Since we will exposing our application to the public Internet, we need additional layers of protection such as transport level (
TLS/SSL encryption) and application level security (authentication and authorization). In this part, we will just configure encryption and explore the other aspects in another blog. To configure end-to-end TLS encryption, we add
tls: trueconfig is actually used as a default, but I have added it explicitly for sake of clarity
To create the cluster:
The Strimzi Operator kicks into action and does all the heavy lifting for us:
- It creates a Kubernetes
- .. and seeding the appropriate Kafka server configuration in a
I will be highlighting the resources created corresponding to the external listener and TLS encryption. For a walk through of ALL the resources which are created as part of the Kafka cluster, please refer to part 1
If you look for the
Services, you will see something similar to this:
Service of the type
LoadBalancer? Since I am using Azure Kubernetes Service, this is powered by an Azure Load Balancer which has a public IP (
126.96.36.199 in this example) and exposes Kafka to external clients over port
9094. You should be able to locate it using the Azure CLI (or the Azure portal if you prefer) by using the
az network lb list command
What about the encryption part?
To figure that out, let's introspect the Kafka server configuration:
As explained in the previous blog, this is stored in a
This is what the
Common listener configuration in
Notice that in addition to inter-broker replication (over port
9091) and un-encrypted internal (within Kubernetes cluster) client access over non
9092, appropriate listener config has been added for TLS encrypted access over port
To confirm, let's try out a couple of client applications which will communicate with our freshly minted Kafka cluster on Kubernetes! We will produce and consume messages using the following:
- Kafka CLI (console) producer and consumer
- Go application (using the Confluent Kafka Go client)
Communication to our Kafka cluster has to be encrypted (non TLS client connections will be rejected).
TLS/SSL implicitly implies one way authentication, where the client validates the Kafka broker identity. In order to do this, client applications need to trust the cluster CA certificate. Remember that the cluster CA certificate is stored in a Kubernetes
Secret (refer to details in part 1). By default, these are auto-generated by Strimzi, but you can provide your own certificates as well (refer https://strimzi.io/docs/operators/master/using.html#kafka-listener-certificates-str)
Start by extracting the cluster CA certificate and password:
You should have two files:
ca.password. Feel free to check out their contents
While some Kafka clients (e.g. Confluent Go client) use the CA certificate directly, others (e.g. Java client, Kafka CLI etc.) require access to the CA certificate via a
truststore. I am using the built-in
truststore which comes in with a JDK (Java) installation - but this is just for convenience and you're free to use other options (such as creating your own)
That's it for the base setup - you are ready to try out the Kafka CLI client!
Please note that the configuration steps for the Kafka CLI as detailed below will also work for the Java clients as well - give it a try!
LoadBalancer public IP for Kafka cluster
Create a file called
client-ssl.properties with the following contents:
To use the Kafka CLI, download
Kafka if you don't have it already - https://kafka.apache.org/downloads
All you need to do is use the
kafka-console-consumer by pointing it to the
client-ssl.properties file you just created
You should see producer and consumer working in tandem. Great!
If you face SSL Handshake errors, please check whether the CA cert has been correctly imported along with its correct password. If the Kafka cluster is not reachable, ensure you are using the right value for the public IP
Now, let's try a programmatic client. Since the Java client behavior (required config properties) are same as the CLI, I am using a Go client to try something different. Don't worry, if you are not a Go programmer, it should be easy to follow along - I will not walk through the entire program, just the part where we create the connection related configuration.
Here is the snippet:
Notice that the
security.protocol are the same as ones you used in the Kafka CLI client (same for Java as well). The only difference is that
ssl.ca.location is used to point to the CA certificate directly as opposed to a
If you have
Go installed, you can try it out. Clone the Git repo...
.. and run the program:
You should see logs similar to this and confirm that messages are being produced and consumed
ctrl+cto exit the app
So we made some progress! We now have a Kafka cluster on Kubernetes which is publicly accessible but is (partially) secure thanks to TLS encryption. We also did some sanity testing using not one, but two (different) client applications. In the next part, we'll improve this further...stay tuned!
Opinions expressed by DZone contributors are their own.