Integration Of Apache Kafka With Mule 4 to Publish/Consume Data To/From Kafka Topics

DZone 's Guide to

Integration Of Apache Kafka With Mule 4 to Publish/Consume Data To/From Kafka Topics

Get to know all about the Seamless integration of Mule 4 with a distributed streaming platform i.e. Apache Kafka to handle large volumes of data.

· Big Data Zone ·
Free Resource

In this tutorial, we will learn how to integrate Mule 4 with Apache Kafka and use its capabilities, such as Publishing and consuming of messages into/from Kafka topics respectively. First, let’s explore Apache Kafka. 

Apache Kafka is a distributed streaming platform which is capable of:

  • Publishing and subscribing to streams of records, similar to a message queue or enterprise messaging system.
  • Storing streams of records in a fault-tolerant durable way.
  • Processing streams of records as they occur.

Kafka has four core APIs:

  • The Producer API allows an application to publish a stream of records to one or more Kafka topics.
  • The Consumer API allows an application to subscribe to one or more topics and process the stream of records produced to them.
  •  The Streams API allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics.
  • The Connector API allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems.

Kafka core APIs

Basic Elements of Apache Kafka

  • Topic - a category or feed name to which records are published. A topic can have zero, one, or many consumers that subscribe to the data written to it.
  • Partition – A topic can have one or more partitions associated with it to handle large volumes of data. Each partition is an ordered, immutable sequence of records that is continually appended to—a structured commit log.
  • Partition offset - The records in the partitions are each assigned a sequential id number that uniquely identifies each record within the partition.
  • Brokers - a Kafka server that runs in a Kafka Cluster. It manages the storage of message in the topics.
  • Kafka Cluster – Kafka brokers form a Kafka cluster. The Kafka Cluster consists of many Kafka Brokers on many servers. Each Kafka broker has a unique ID.
  • Producers - publish data to topics of their choice. The producer is responsible for choosing which record to assign to which partition within the topic.
  • Consumers – read and process data from topics. 

Setting Up Apache Kafka

Download the stable release of Apache Kafka and extract it. I have downloaded ‘kafka_2.13-2.5.0’.

Kafka needs ZooKeeper to manage the cluster. ZooKeeper is used to coordinate the brokers/cluster topology. ZooKeeper is a consistent file system for configuration information. ZooKeeper is used for leadership election for Broker Topic Partition Leaders. So, we have to start Zookeeper prior to starting the Kafka server on our machine.

Apache Kafka package comes with an embedded zookeeper instance, so, we will make use of the same.

  • In the config directory, there is a file zookeeper.properties which includes all the basic properties required to start Zookeeper.
  • Open command prompt and go to the path where you have extracted Kafka (C:\kafka_2.13-2.5.0) and run the below command which will start zookeeper on port 2181

     .\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties

Running ZooKeeper

Running ZooKeeper

  • Zookeeper is now started on Port 2181. We can change the port number if required in the zookeeper.properties file. Now, open server.properties, present in the config directory. Update log.dirs= /tmp/kafka-logs to log.dirs= C:\kafka_2.13-2.5.0\kafka-logs. 
  • Open command prompt and go to C:\kafka_2.13-2.5.0. Run the below command to start the Kafka server – 

 .\bin\windows\kafka-server-start.bat .\config\server.properties

Running Kafka server

Running Kafka server

Kafka sever is now started. We will create a Kafka topic. Open a new command prompt and go to C:\kafka_2.13-2.5.0\bin\windows and run the below command:

kafka-topics.bat –create –zookeeper localhost:2181 –replication-factor 1 
–partitions 1 –topic DemoTopic<topic name>

Creating Kafka Topic

Apache Kafka Connector in Mule 4

Check if your Mule Palette has already installed the Kafka connector. If not, you can install it by connecting to AnyPoint Exchange via Studio.

AnyPoint Exchange via Studio

Kafka as Message Producer

We will now create a Mule flow that publishes message to the Kafka topic created above. Drag an HTTP listener and configure it. In the Responses Tab, update the Response Body with #['Message delivered successfully to DemoTopic']

HTTP Listener config

Configuring listener

Checking responses

Drag the Apache Kafka connector to use it as Message Producer and configure its properties like Bootstrap servers. You will find it mentioned in the producer.properties file, present in the config directory of Apache Kafka installed package. 

Note – Bootstrap servers is a comma-separated list of host and port pairs that are the addresses of the Kafka brokers in a "bootstrap" Kafka cluster that a Kafka client connects to initially to bootstrap itself. A host and port pair uses : as the separator.

Apache Kafka Producer configConfigure Publish Message operation and give topic name and key (a unique key published with message)

Configuring publish message operation

The entire flow looks like:

Kafka configuration

Start the Kafka Producer. Open Command Prompt and go to C:\kafka_2.13-2.5.0\bin\windows and run the below command:

kafka-console-producer.bat –broker-list localhost:9092 –topic DemoTopic.

Starting the Kafka Producer

Start the Kafka Consumer by running:

kafka-console-consumer.bat –bootstrap-server localhost:9092 –topic DemoTopic.

Starting Kafka Consumer

Test the application. Open Postman and try hitting the URL – http://localhost:8089/publish.

Hitting publish URL

We have successfully received the response that we set in the HTTP Listener configuration. Now, let’s check at the Kafka side, as it will start consuming the message whenever it is posted via Postman. (We have already started the Kafka consumer)

Response from Kafka server

Kafka as Message Consumer

We will now use the Kafka Consume Operation to consume the message published to the Kafka topic.

Drag the Apache Kafka Message Consumer into the source section of a new Mule Flow. Configure the Kafka connector with properties like Bootstrap servers, Group id etc.

You can find Group id mentioned in the consumer.properties file present in the config directory of the installed Apache Kafka Package.

Note – Group id is a unique string that identifies the consumer group this consumer belongs to.

Consumer configuration

Message consumer

Drag a Logger component to log the message consumed from the Kafka topic

Logger component

• The entire flow looks like:


•    Test the Application.

We can use Postman to publish the message to the Kafka topic and then check the whether the message gets logged at the Mule side through the KafkaConsumerFlow.

Logged message

Basic Kafka Commands

•    Start Zookeeper: .\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties

•    Start Kafka Server: .\bin\windows\kafka-server-start.bat .\config\server.properties

•    Create a Topic: kafka-topics.bat –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic TopicName<topic name>

•    Describe a Topic: kafka-topics.bat –describe –zookeeper localhost:2181 –topic TopicName<topic name>

•    List Kafka Topics: .\bin\windows\kafka-topics.bat --list --zookeeper localhost:2181

•    Delete a topic: kafka-run-class.bat kafka.admin.TopicCommand –delete –topic TopicName<topic name> –zookeeper localhost:2181

•    Start Kafka Producer: kafka-console-producer.bat –broker-list localhost:9092 –topic TopicName<topic name>

•    Start Kafka Consumer: kafka-console-consumer.bat –bootstrap-server localhost:9092 –topic TopicName<topic name>

Happy Learning!!

apache kafka, data management, integration, kafka consumer, mule 4, zookeeper

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}