Getting Kafka Ready for a Camel Ride

DZone 's Guide to

Getting Kafka Ready for a Camel Ride

Extend Kafka Connect integrations with Apache Camel Kafka Connectors.

· Integration Zone ·
Free Resource

Apache Kafka has become one of the most used pieces of software in modern application development because of its distributed nature, high throughput, and horizontal scalability. Every day more and more organizations are adopting it as the central event bus for their systems architecture. As a result, more and more data is flowing through the cluster making the connectivity requirements rise in priority in any backlog. For this reason, the Apache Camel community has released the first wave of Kafka Connect connectors for the purpose of easing the burden on the development teams.

What is Apache Camel?

The Camel community has built one of the busiest open source integration frameworks in the Apache Foundation. The Camel framework enables quick and easy integration of data consumer and producer systems. It implements the most used Enterprise Integration Patterns (EIPs) and, interfaces and protocols used everywhere, as they emerge. Everything under the same component base configuration, creating the required building blocks to solve almost any integration requirement. 

What is the Camel Kafka Connector Project?

After maturing for almost a decade, the Camel community spun a couple of subprojects to foster innovation in areas like runtime support and container readiness. In particular, the Camel Kafka Connector subproject focuses on the use of Camel components as Kafka Connect connectors. With this in mind, they built a tiny layer between Camel and Kafka frameworks to generate a new connector for each camel component. As a result, the expanded Kafka Connect compatibility can be used effortlessly in the Kafka ecosystem.   

This first release of the project allows the community to try and share feedback on the autogenerated connectors. Despite being the first release of the subproject, most of the Camel components are battle-tested and used all around the world in production scenarios.

Getting Started With The Camel Kafka Connectors

There are more than 340+ Camel connectors to get started, ranging from AWS S3 integration to telegram or slack. So it should be fairly easy to find a use case to implement a small project. 

  • In order to simplify the Kafka Connect cluster deployment you can use the Kubernetes operator from the Strimzi project to run on Minikube or Kind on your laptop. 

  • Then search and download the connector package zip version from the Maven central repository because the zip file should contain all the library jars required to run the connector in your plugins' path. 

  • Next, you can create a container image with your plugins on top of the strimzi base image so you can use the image as the Kafka Connect base. 

  • Finally, configure the connector using the Kafka Connect REST API or the new Strimzi Kafka Connector custom resource definition (CRD).

But even so, you can also try the connectors locally on your laptop without the need of Kubernetes. You just need a locally running Kafka instance and follow the instructions.

Want to take a peek at how it looks when running? Take a look at my video on integrating Slack to Kafka:


Apache Kafka is getting used as an event backbone in new organizations everyday. Communities like Apache Camel are working on how to speed up development on key areas of the modern application, like integration. The Camel Kafka Connect project from the Apache Foundation has enabled their vastly set of connectors to interact with Kafka Connect natively so developers can start sending and receiving data from kafka on their preferred systems.

Image: “Getting Kafka ready for the Camel Ride” is a derivative work of “Kafka” by “g p” and “Camel” by “Ziad Fhema”, used under CC BY 2.0

apache camel ,apache kafka ,integration ,kafka connectors ,kubernetes ,strimzi

Published at DZone with permission of Hugo Guerrero . See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}