Over a million developers have joined DZone.

Using Jakarta EE/MicroProfile to Connect to Apache Kafka: Part 1 — Hello World

DZone's Guide to

Using Jakarta EE/MicroProfile to Connect to Apache Kafka: Part 1 — Hello World

Learn how to securely integrate Apache Kafka with Eclipse MicroProfile.

· Big Data Zone ·
Free Resource

The open source HPCC Systems platform is a proven, easy to use solution for managing data at scale. Visit our Easy Guide to learn more about this completely free platform, test drive some code in the online Playground, and get started today.

Apache Kafka is a community-distributed streaming platform that has three key capabilities: publish and subscribe to streams of records, store streams of records in a fault-tolerant durable way, and then process streams as they occur. Apache Kafka has several success cases in the Java world. This post will cover how to benefit from this powerful tool in the Jakarta EE/MicroProfile universe.

Apache Kafka Core Concepts

Kafka is run as a cluster on one or more servers that can span multiple data centers; Kafka cluster stores a stream of records in categories called topics, and each record consists of a key, a value, and a timestamp.

From the documentation, Kafka has four core APIs:

  • The Producer API allows an application to publish a stream of records to one or more Kafka topics.

  • The Consumer API allows an application to subscribe to one or more topics and process the stream of records produced to them.

  • The Streams API allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams.

  • The Connector API allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems. 

APIs Apache Kafka

Install Apache Kafka

The official documentation has a nice getting started with Apache Kafka post that teaches you how to install it with Zookeeper. Briefly, Kafka uses Zookeeper to Cluster membership, topic configuration, and so on.

Download the 2.1.0 release and un-tar it.

tar -xzf kafka_2.11-2.1.0.tgz
cd kafka_2.11-2.1.0

First, start a Zookeeper instance

bin/zookeeper-server-start.sh config/zookeeper.properties

And finally, start with Apache Kafka:

 bin/kafka-server-start.sh config/server.properties

Using Docker

Docker Logo

There is also the possibility of using Docker. As it requires two images, one to Zookeeper and one to Apache Kafka, this tutorial will use docker-compose. Follow these instructions:

version:  '3.2'


    image: "confluent/zookeeper"
      - microprofile
      - 2181:2181

    image: "confluent/kafka"
      - microprofile
      - 9092:9092
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - zookeeper


Then, run the command:

docker-compose -f docker-compose.yml up -d

To connect as localhost, also define Kafka as the localhost within Linux, append the value below at the /etc/hosts:       localhost kafka

Application With Eclipse MicroProfile

The Eclipse MicroProfile is an open-source project whose goal is to define a microservices application platform that is portable across multiple runtimes. Eclipse MicroProfile has several plans to make it easier to create enterprise applications at the Microservices and cloud age. To connect with Apache Kafka, there is the Eclipse MicroProfile Reactive that has the Reactive Message.

The sample code will smoothly create a sender and receiver message from Apache Kafka using CDI 2.0 and Java SE; this project will use the Maven structure. Therefore, the first step of this demo is to define the dependencies. The project needs a MicroProfile configuration implementation, CDI 2.0 implementation, and MicroProfile reactive dependencies.



                    <!-- this avoid having the logging working, to exclude it -->



Then, the next step is to create the configuration files at src/main/resources/META-INF.

The first one is to enable the CDI, the bean.xml file.

<beans xmlns="http://xmlns.jcp.org/xml/ns/javaee"

The second and last one is microprofile-config.properties that has the setup configurations to Eclipse MicroProfile that connects to Apache Kafka, such as implementation to serialize/deserialize the records, which are composed of a key and the respective value.

# Kafka Sink
## Kafka Source

Consequently, with all the configuration set up, the last step is to create the class that will send/receive messages from Apache Kafka. The API has fluent information; therefore, there are Incoming and Outgoing annotations. At the Sender class, there is a BlockingQueue, which just sends a message when there is a text in the queue.

public class Receiver {

    private static final Logger LOGGER = Logger.getLogger(Receiver.class.getName());

    public CompletionStage<Void> consume(KafkaMessage<String, String> message) {
        String payload = message.getPayload();
        String key = message.getKey();
        MessageHeaders headers = message.getHeaders();
        Integer partition = message.getPartition();
        Long timestamp = message.getTimestamp();
        LOGGER.info("received: " + payload + " from topic " + message.getTopic());
        return message.ack();


public class Sender {

    private static final Logger LOGGER = Logger.getLogger(Sender.class.getName());

    private BlockingQueue<String> messages = new LinkedBlockingQueue<>();

    public void add(String message) {

    public CompletionStage<KafkaMessage<String, String>> send() {
        return CompletableFuture.supplyAsync(() -> {
            try {
                String message = messages.take();
                LOGGER.info("Sending message to kafka with the message: " + message);
                return KafkaMessage.of("kafka", "key", message);
            } catch (InterruptedException e) {
                throw new RuntimeException(e);


Finally, the last class is to start the CDI container, and then, send some message to the Kafka. Thus, you can see the result log.

public class App {
    public static void main(String[] args) {
        SeContainer container = SeContainerInitializer.newInstance().initialize();
        Sender sender = container.select(Sender.class).get();
        sender.add("Hello world");

To conclude, we see the potential of Apache Kafka and why this project became so accessible to Big-Data players. This is a simple example of how secure it is to integrate with Eclipse MicroProfile. A special thanks to Clement Escoffier because he helped me out at this sample code, which can also be found on GitHub.

Managing data at scale doesn’t have to be hard. Find out how the completely free, open source HPCC Systems platform makes it easier to update, easier to program, easier to integrate data, and easier to manage clusters. Download and get started today.

microprofile ,jakarta ee ,java ,reactive ,eclipse ,eclipse microprofile ,tutorial ,apache kafka ,big data

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}