{{announcement.body}}
{{announcement.title}}

How to Use MirrorMaker With Apache Kafka Clusters

DZone 's Guide to

How to Use MirrorMaker With Apache Kafka Clusters

In this article, see how to use MirrorMaker with Apache Kafka clusters.

· Integration Zone ·
Free Resource

MirrorMaker is a process in Apache Kafka to replicate or mirror data between Kafka Clusters. Don't confuse it with the replication of data among Kafka nodes of the same cluster. One use case is to provide a replica of a complete Kafka cluster in another data center to cater to different use cases without impacting the original cluster.

You can check out my other article on Kafka, which would help to have basic idea of Apache Kafka setup and commands.

In MirrorMaker, there is a consumer connector and producer connector. The consumer will read data from topics in source Kafka cluster and the producer connector will write those events or data to target Kafka Cluster. Source cluster and target cluster are independent of each other.

Let's understand this with a simple setup where both clusters exist on the same machine. We are using two Kafka Clusters; each with two Kafka nodes and one zookeeper node. All processes run on the same host. One Kafka Cluster is the source and the other is the target. This setup is just for demonstration purposes being single zookeeper node cluster and on the same host; it is not meant for production. 

1. Create folders for zookeeper and Kafka logs.

Shell
 




xxxxxxxxxx
1


 
1
$ pwd
2
/home/chandrashekhar/kafka_2.13-2.4.1/
3
 
          
4
mkdir -p data/zookeeper1
5
mkdir -p data/zookeeper2
6
mkdir -p data/kafka-logs-1-1
7
mkdir -p data/kafka-logs-1-2
8
mkdir -p data/kafka-logs-2-1
9
mkdir -p data/kafka-logs-2-2



2. Configuration for zookeeper nodes.

Shell
 




x


1
[chandrashekhar@localhost kafka_2.13-2.4.1]$ vi config/zookeeper1.properties
2
 
          
3
dataDir=~/kafka_2.13-2.4.1/data/zookeeper1
4
 
          
5
clientPort=2181
6
 
          
7
maxClientCnxns=0
8
 
          
9
 
          
10
 
          
11
[chandrashekhar@localhost kafka_2.13-2.4.1]$ vi config/zookeeper2.properties
12
 
          
13
dataDir=~/kafka_2.13-2.4.1/data/zookeeper2
14
 
          
15
clientPort=2182
16
 
          
17
maxClientCnxns=0



3. Configuration for Kafka nodes. Total 4 Kafka nodes, 2 node connect to 2181 and other 2 to 2182.

Shell
 




x
60


 
1
[chandrashekhar@localhost kafka_2.13-2.4.1]$ cp config/server.properties config/server1-1.properties
2
 
          
3
[chandrashekhar@localhost kafka_2.13-2.4.1]$ cp config/server.properties config/server1-2.properties
4
 
          
5
[chandrashekhar@localhost kafka_2.13-2.4.1]$ cp config/server.properties config/server2-1.properties 
6
 
          
7
[chandrashekhar@localhost kafka_2.13-2.4.1]$ cp config/server.properties config/server2-2.properties
8
-----
9
 
          
10
vi ~/kafka_2.13-2.4.1/config/server1-1.properties
11
 
          
12
broker.id=0
13
 
          
14
port=9093
15
 
          
16
zookeeper.connect=localhost:2181
17
 
          
18
advertised.host.name = localhost
19
 
          
20
log.dirs=~/kafka_2.13-2.4.1/data/kafka-logs-1-1
21
-----
22
 
          
23
vi ~/kafka_2.13-2.4.1/config/server1-2.properties
24
 
          
25
broker.id=1
26
 
          
27
port=9094
28
 
          
29
zookeeper.connect=localhost:2181
30
 
          
31
advertised.host.name = localhost
32
 
          
33
log.dirs=~/kafka_2.13-2.4.1/data/kafka-logs-1-2
34
-----
35
 
          
36
vi ~/kafka_2.13-2.4.1/config/server2-1.properties
37
 
          
38
broker.id=2
39
 
          
40
port=9095
41
 
          
42
zookeeper.connect=localhost:2182
43
 
          
44
advertised.host.name = localhost
45
 
          
46
log.dirs=~/kafka_2.13-2.4.1/data/kafka-logs-2-1
47
-----
48
 
          
49
vi ~/kafka_2.13-2.4.1/config/server2-2.properties
50
 
          
51
broker.id=4
52
 
          
53
port=9096
54
 
          
55
zookeeper.connect=localhost:2182
56
 
          
57
advertised.host.name = localhost
58
 
          
59
log.dirs=~/kafka_2.13-2.4.1/data/kafka-logs-2-2
60
-----



4. Start zookeeper nodes and Kafka nodes.

Shell
 




x
10


1
chandrashekhar@chandrashekhar:~/kafka_2.13-2.4.1/bin$ ./zookeeper-server-start.sh ../config/zookeeper1.properties 
2
 
          
3
chandrashekhar@chandrashekhar:~/kafka_2.13-2.4.1/bin$ ./zookeeper-server-start.sh ../config/zookeeper2.properties 
4
 
          
5
chandrashekhar@chandrashekhar:~/kafka_2.13-2.4.1/bin$ ./kafka-server-start.sh ../config/server1-1.properties
6
 
          
7
chandrashekhar@chandrashekhar:~kafka_2.13-2.4.1/bin$ ./kafka-server-start.sh ../config/server1-2.properties
8
 
          
9
chandrashekhar@chandrashekhar:~/kafka_2.13-2.4.1/bin$ ./kafka-server-start.sh ../config/server2-1.properties
10
 
          
11
chandrashekhar@chandrashekhar:~/kafka_2.13-2.4.1/bin$ ./kafka-server-start.sh ../config/server2-2.properties



5. Create topic mirrormakerPOC on both Kafka clusters with same number of partition.

Shell
 




x


 
1
chandrashekhar@chandrashekhar:~/kafka_2.13-2.4.1/bin$ ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 2 --topic mirrormakerPOC
2
3
Created topic mirrormakerPOC.
4
5
chandrashekhar@chandrashekhar:~/kafka_2.13-2.4.1/bin$ ./kafka-topics.sh --create --zookeeper localhost:2182 --replication-factor 2 --partitions 2 --topic mirrormakerPOC
6
7
Created topic mirrormakerPOC.
8
9



6. Create consumer and producer configuration file for mirror maker.

Shell
 




xxxxxxxxxx
1
21


 
1
chandrashekhar@chandrashekhar:~$ cat sourceCluster1Consumer.config 
2
 
          
3
bootstrap.servers=localhost:9093,localhost:9094
4
 
          
5
exclude.internal.topics=true
6
 
          
7
client.id=mirror_maker_consumer
8
 
          
9
group.id=mirror_maker_consumer
10
 
          
11
 
          
12
 
          
13
chandrashekhar@chandrashekhar:~$ cat targetClusterProducer.config 
14
 
          
15
bootstrap.servers=localhost:9095,localhost:9096
16
 
          
17
acks=1
18
 
          
19
batch.size=50
20
 
          
21
client.id=mirror_maker_test_producer



7. Now run MirrorMaker process based on consumer and producer configuration defined in last step.

Shell
 




xxxxxxxxxx
1


 
1
chandrashekhar@chandrashekhar:~/kafka_2.13-2.4.1/bin$ ./kafka-mirror-maker.sh --consumer.config ../../sourceCluster1Consumer.config --num.streams 1 --producer.config ../../targetClusterProducer.config --whitelist=".*"



8. Start sending message to Kafka Cluster 1 listening on zookeeper port 2181.

Shell
 




x


1
chandrashekhar@chandrashekhar:~/kafka_2.13-2.4.1/bin$ ./kafka-console-producer.sh --broker-list localhost:9093,localhost:9094 --topic mirrormakerPOC
2
 
          
3
>2134
4
 
          
5
>111
6
 
          



9. Start consuming on Kafka nodes of both Kafka Clusters. 

- Consume for  Kafka nodes on 2nd Cluster.

Shell
 




x


1
./kafka-console-consumer.sh --bootstrap-server localhost:9095,localhost:9096 --topic mirrormakerPOC --group topic_group_2
2
2134
3
111
4
 
          



- Consume for  Kafka nodes on 1st Cluster.

Shell
 




xxxxxxxxxx
1


1
./kafka-console-consumer.sh --bootstrap-server localhost:9093,localhost:9094 --topic mirrormakerPOC --group topic_group_1
2
2134
3
111



10. Monitor list of topics,  details of topic and offset for particular consumer-group. 

Java
 




x


1
[chandrashekhar@localhost bin]$ ./kafka-topics.sh --list --zookeeper localhost:2182
2
__consumer_offsets
3
mirrormakerPOC
4
 
          
5
[chandrashekhar@localhost bin]$ ./kafka-topics.sh --list --zookeeper localhost:2181
6
__consumer_offsets
7
mirrormakerPOC
8
------------------------
9
 
          
10
[chandrashekhar@localhost bin]$ ./kafka-topics.sh --describe --zookeeper localhost:2182 --topic mirrormakerPOC
11
Topic: mirrormakerPOC   PartitionCount: 2   ReplicationFactor: 2    Configs: 
12
    Topic: mirrormakerPOC   Partition: 0    Leader: 3   Replicas: 3,2   Isr: 3,2
13
    Topic: mirrormakerPOC   Partition: 1    Leader: 2   Replicas: 2,3   Isr: 2,3
14
[chandrashekhar@localhost bin]$ 
15
[chandrashekhar@localhost bin]$ ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic mirrormakerPOC
16
Topic: mirrormakerPOC   PartitionCount: 2   ReplicationFactor: 2    Configs: 
17
    Topic: mirrormakerPOC   Partition: 0    Leader: 0   Replicas: 0,1   Isr: 0,1
18
    Topic: mirrormakerPOC   Partition: 1    Leader: 1   Replicas: 1,0   Isr: 1,0
19
------------------------
20
      
21
[chandrashekhar@localhost bin]$ ./kafka-consumer-groups.sh --bootstrap-server localhost:9095,localhost:9096 --group topic_group_2 --describe
22
 
          
23
GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                   HOST            CLIENT-ID
24
topic_group_2   mirrormakerPOC  0          4               4               0               consumer-topic_group_2-1-846dfe1f-c487-410f-961d-5df50da2ea58 /127.0.0.1      consumer-topic_group_2-1
25
topic_group_2   mirrormakerPOC  1          4               4               0               consumer-topic_group_2-1-846dfe1f-c487-410f-961d-5df50da2ea58 /127.0.0.1      consumer-topic_group_2-1
26
[chandrashekhar@localhost bin]$ 



That's it, I hope this article will help you have a basic idea of mirroring or replicating data from one Kafka cluster to another Kafka cluster.  

Topics:
apache kafka ,integration ,mirroring ,replication ,tutorial

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}