{{announcement.body}}
{{announcement.title}}

Mule 4 Integration With Kafka

DZone 's Guide to

Mule 4 Integration With Kafka

In this article, we discuss how to go about integrating Apache Kafka with a sample Mule 4 application.

· Big Data Zone ·
Free Resource

Hello Muleys,

Nowadays, there is a lot of buzz around the Apache Kafka messaging platform. Enterprises are showing their interest in moving towards a more reliable, stable, and open-source solution for their messaging needs. This article talks about integrating Apache Kafka with a sample Mule4 application. 

So let's start with Kafka set up first!

  1. Download the latest binaries from Apache Kafka and extract it to local drive C:\
  2. Download the latest binaries for Apache ZooKeeper and extract it to local drive C:\
  3. Configure Environment variables:
    1. JAVA_HOME: point this env variable to a suitable, installed version of JAVA e.g, "C:\Program Files\Java\jre1.8.0_212". Use command prompt to identify the installed Java version using java -version, as shown below:
      Getting Java version
      Update the Path variable with %JAVA_HOME%\bin.
    2. ZOOKEEPER_HOME: point this env variable to C:\zookeeper-3.5.5 and update the PATH variable with %ZOOKEEPER_HOME%\bin.
  4. Start the ZooKeeper server. Open the command prompt, run as administrator, and run "zkserver"       inside C:\zookeeper-3.5.5\bin. Possible output:ZooKeeper running output
  5. Start Kafka server:
    1. Edit server.properties with line 30. Go to C:\kafka_2.12-2.2.0\config\server.properties and uncomment line 30: listeners = PLAINTEXT://localhost:9092.
    2. Open command prompt and trace to C:\kafka_2.12-2.2.0\bin\windows and execute following:

      C:\kafka_2.12-2.2.0\bin\windows>kafka-server-start.bat C:\kafka_2.12-2.2.0\config\server.properties
  6. Set Kafka producer: Creating a topic with name “test” and publish some messages after “>” prompts. Open command prompt and trace to C:\kafka_2.12-2.2.0\bin\windows and execute following: C:\kafka_2.12-2.2.0\bin\windows>kafka-console-producer.bat --broker-list 192.168.0.184:9092 --topic testSetting Kafka producer
  7. Set Kafka consumer: Open command prompt and trace to C:\kafka_2.12-2.2.0\bin\windows and execute following: C:\kafka_2.12-2.2.0\bin\windows>kafka-console-consumer.bat --bootstrap-server 192.168.0.184:9092 --topic test --from-beginning

Setting Kafka consumer


So, we have covered the basic Kafka set up on local a Windows machine and tried the message publish consume mechanism on a sample topic "test".

Logs for command prompt triggered message:

Plain Text
 




xxxxxxxxxx
1
16


 
1
C:\WINDOWS\system32>cd C:\kafka_2.13-2.4.0\bin\windows
2
 
          
3
C:\kafka_2.13-2.4.0\bin\windows>kafka-console-producer.bat --broker-list 192.168.0.184:9092 --topic test
4
>hello
5
 
          
6
 
          
7
 
          
8
 
          
9
C:\WINDOWS\system32>cd C:\kafka_2.13-2.4.0\bin\windows
10
 
          
11
C:\kafka_2.13-2.4.0\bin\windows>kafka-console-consumer.bat --bootstrap-server 192.168.0.184:9092 --topic test --from-beginning
12
[2020-02-14 15:55:20,671] WARN [Consumer clientId=consumer-console-consumer-38663-1, groupId=console-consumer-38663] Error while fetching metadata with correlation id 2 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
13
[2020-02-14 15:55:20,858] WARN [Consumer clientId=consumer-console-consumer-38663-1, groupId=console-consumer-38663] Error while fetching metadata with correlation id 4 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
14
[2020-02-14 15:55:21,004] WARN [Consumer clientId=consumer-console-consumer-38663-1, groupId=console-consumer-38663] Error while fetching metadata with correlation id 5 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
15
[2020-02-14 15:55:21,358] WARN [Consumer clientId=consumer-console-consumer-38663-1, groupId=console-consumer-38663] Error while fetching metadata with correlation id 7 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
16
hello



Let's start with Anypoint Studio (7.4.2) and Mule RunTime (4.2.2) setup:

  1. Open Anypoint studio and create a sample project named as "KafkaDemo".
  2. The default flow can be renamed to "Kafka Publisher", which will listen on an HTTP listener, logs the POST request on the 8099 port and action "/kafkademo", and publishes the message on Kafka topic "test".
  3. Create a "Consumer" flow starting with "Kafka Consumer" and followed by the logger to log the message is being received.Publisher workflow
  4. Kafka Publisher configuration: under the global element, you may need to create a "KafkaPublisher" connection as shown:
    Getting Kafka producer configuration
  5. Similarly for Kafka Consumer:
    Kafka Consumer configuration
  6. Code snippet:
XML
 




x
27


1
<?xml version="1.0" encoding="UTF-8"?>
2
 
          
3
<mule xmlns:kafka="http://www.mulesoft.org/schema/mule/kafka" xmlns:http="http://www.mulesoft.org/schema/mule/http"
4
    xmlns="http://www.mulesoft.org/schema/mule/core"
5
    xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
6
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
7
http://www.mulesoft.org/schema/mule/kafka http://www.mulesoft.org/schema/mule/kafka/current/mule-kafka.xsd">
8
    <http:listener-config name="HTTP_Listener_config" doc:name="HTTP Listener config" doc:id="03316fb4-b7aa-4b59-8fc1-be7e67cb81e7" >
9
        <http:listener-connection host="0.0.0.0" port="8099" />
10
    </http:listener-config>
11
    <kafka:kafka-producer-config name="Apache_Kafka_Producer_configuration" doc:name="Apache Kafka Producer configuration" doc:id="34378022-e968-4207-84e4-695a51681d74" >
12
        <kafka:basic-kafka-producer-connection bootstrapServers="localhost:9092" />
13
    </kafka:kafka-producer-config>
14
    <kafka:kafka-consumer-config name="Apache_Kafka_Consumer_configuration" doc:name="Apache Kafka Consumer configuration" doc:id="561bce8d-0ca4-4c37-be5c-6c1e928e6147" >
15
        <kafka:basic-kafka-consumer-connection groupId="test-consumer" bootstrapServers="localhost:9092" />
16
    </kafka:kafka-consumer-config>
17
    <flow name="KafkaPublisher" doc:id="95036497-ffb3-45db-9860-5a6883cdaa57">
18
        <http:listener doc:name="Listener" doc:id="bb867536-18a4-4d2c-bc69-3e4bd39bf277" config-ref="HTTP_Listener_config" path="/kafkademo" allowedMethods="POST" />
19
        <logger level="INFO" doc:name="Logger" doc:id="b3524fdb-0b55-4adb-a659-493254a4ec09" message="Payload received: #[payload]" />
20
        <kafka:producer doc:name="Publish Message" doc:id="1143950e-1758-4c83-9871-e33bcba76599" config-ref="Apache_Kafka_Producer_configuration" topic="test" key="2" />
21
        <logger level="INFO" doc:name="Logger" doc:id="c1ce9762-7ac2-4d80-bd36-e159af85c9ff" message="Message published to Kafka topic -test" />
22
    </flow>
23
    <flow name="Consumer" doc:id="877cced3-516d-4b45-a98e-6dcf37f3b89b" >
24
        <kafka:consumer doc:name="Message Consumer" doc:id="33a0aee2-d33e-4284-a217-9537946cb631" config-ref="Apache_Kafka_Consumer_configuration" topic="test"/>
25
        <logger level="INFO" doc:name="Logger" doc:id="42f3a992-3e72-4e2e-b96d-20ae62ba0ea2" message="Message consumed: #[payload]"/>
26
    </flow>
27
</mule>



Let's test it!

  1. You can use any REST API tool like POSTMAN or SOAP UI or Advanced Rest Client and point to POST operation on the "http://localhost:8089/kafkademo" with a sample string message.
  2. Run the application in the Mule Anypoint studio and trigger the request from POSTMAN.
  3. Studio console will show message published and consumed:
Plain Text


Hence, we were able to publish and consume the message over a Kafka Topic.

I hope this tutorial helps as a first step to use Kafka in Mule 4 application for anyone new to these concepts. Do reach out to me in case of queries. Happy learning !!

Topics:
apache ,big data ,kafka ,mule ,mule 4 ,tutorial

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}