Migrate Data Across Kafka Cluster Using mirrormaker2 in Strimzi
In this article, we will discuss a use case where data from one Kafka cluster has to be migrated to another Kafka Cluster. We will be using mirrormaker 2.
Join the DZone community and get the full member experience.Join For Free
In this article, we will discuss a use case where data from one kafka cluster has to be migrated to another Kafka Cluster. Here the target is strimzi and the source is a standalone Kafka cluster. Target means where data has to be copied and the source is from where we want to copy/migrate data. I have an article on how to use mirrormaker with apache kafka clusters about mirrormaker version 1. This article is about mirrormaker 2, which has more features than mirrormaker1.
At the time of writing this article, the latest version of strimzi is 0.22.1 and can be downloaded from here.
I have installed strimzi on minikube version: v1.19.0. Standalone Kafka is installed on a different laptop with RHEL 8 OS. Also, I am using a simple Kafka producer to produce messages to the source Kafka cluster.
So let's begin the proof of concept.
1. Source Kafka(Standalone) configuration: To connect from external clients and mirror-maker we have to set
[KAFKA_HOME]/config/server.properties. Start zookeeper and kafka node. Here I have only one zookeeper and kafka node.
3. Strimzi setup: Follow upstream strimzi documentation for detailed setup in minikube. I have summarized the steps below; I followed all these commands for setup.
4. MirrorMaker 2 configuration: Within the distribution itself,
strimzi-0.22.1/examples/mirror-maker, we can find mirrormaker example yaml files. We will create another copy of
kafka-mirror-maker-2-custom-replication-policy.yaml and modify that copied file.
The important configurations are:
my-source-cluster: Here we have to provide the bootstrap-server url of Source Kafka against property
my-target-cluster: Here we have to provide the bootstrap-server URL of targer Kafka against property
bootstrapServers. Target Kafka node is installed using strimzi in minikube. Thus I set
my-cluster-kafka-bootstrap listening on
replication.policy.class: We set it with
io.strimzi.kafka.connect.mirror.IdentityReplicationPolicy because we want the target kafka node to have the same topic as the source kafka node.
We have also set the readiness probe and liveliness probe with configurable seconds so that we can tune timeouts if any.
5. Apply MirrorMaker2 configuration: Apply the mirrormaker2 yaml file. Then check topics in the target kafka node.
6. Consume messages from Target Kafka node i.e. Strimzi on Kubernetes.
That's it, guys. I think you will find this article interesting and helpful!
Opinions expressed by DZone contributors are their own.