Setting Up Local Kafka Container for Spring Boot Application

In today's microservices and event-driven architecture, Apache Kafka is the de facto for streaming applications. However, setting up Kafka for local development in conjunction with your Spring Boot application can be tricky, especially when configuring it to run locally. 

Spring Boot application provides support for Kafka integration through the spring-kafka maven package. To work with spring-kafka, we need to connect to the Kafka instance. Typically, during development, we would just run a local Kafka instance and build against it. But with Docker Desktop and containers, things are much easier to set up than running a local Kafka instance. This article guides us through the steps for setting up the local Kafka container with the Spring Boot application.  

Prerequisites 

1. We need to set up Docker Desktop. To do so, we can refer to this article. 
2. Spring Boot application with spring-kafka package configured. 

Running Kafka Container

For running the Kafka container, we will first use the following docker-compose file:

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
    ports:
      - "2181:2181"

  kafka:
    image: confluentinc/cp-kafka:latest
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    ports:
      - "9092:9092"
    depends_on:
      - zookeeper

This docker-compose file contains the configurations to pull the Kafka container and its dependency, the Zookeeper container. Zookeeper manages Kafka broker nodes in the cluster; we can find further details about this in this article.  

For registering the containers with Docker Desktop, we will use the following command:

docker-compose up -d

This will pull the required images and launch the containers, and once the containers are launched, you can see the containers in Docker Desktop like below:

Now that the Kafka container is up, we can then create the required topics using Docker Desktop console using the following command: 

kafka-topics --create --topic user-notification --partitions 1 --replication-factor 1 --bootstrap-server localhost:9092

Now that the container is up and the required prerequisites have been performed, we can launch the Spring Boot application.  For the Spring Boot application, configure the Kafka bootstrap address as below:

kafka.bootstrapAddress=localhost:9092

When we launch the Spring Boot application, we should see the logs for connection with Kafka, depending on the type of Spring Boot application, whether it is a producer or consumer.

Following the steps outlined in the article, we set up a local development environment using a Kafka container and a Spring Boot application.