Confluent Operator to Establish a Cloud-Native Apache Kafka Platform

DZone 's Guide to

Confluent Operator to Establish a Cloud-Native Apache Kafka Platform

Take a look at how the Confluent Operator can help solve many of the issue that come with using a Kafka-Kubernetes combo.

· Cloud Zone ·
Free Resource

Confluent Operator is now GA for production deployments. This is a Kafka Operator for Kubernetes which provides automated provisioning and operations of an Apache Kafka cluster and its whole ecosystem (Kafka Connect, Schema Registry, KSQL, etc.) on any Kubernetes infrastructure.

Confluent Operator Kafka Operator for Kubernetes Download

I want to share a slide deck which explains:

  • Why Kubernetes is getting more and more traction to build a cloud-native infrastructure.
  • Why this is relevant for Apache Kafka and Confluent Platform.
  • The challenges running Kafka on Kubernetes.
  • How Confluent Operator solves these problems providing a powerful Kafka Operator for Kubernetes.

Cloud-Native vs. SaaS/Serverless

Software-as-a-Service (SaaS) and serverless platforms provide software and services in the public cloud as managed service. Cloud-native infrastructures allow you to leverage the features of SaaS/Serverless in your own self-managed infrastructure (either on-premise or in public cloud, and without vendor lock-in).

What is cloud-native? Many different definitions exist on the web. Two definitions which I like are "The Twelve-Factor App" and "10 key characteristics of The New Stack."

Some of the key benefits of cloud-native infrastructures:

  • Scalable
  • Flexible
  • Agile
  • Elastic
  • Automated

This is very different from traditional bare metal or VM infrastructures. Even if you use containers like Docker, you don't automatically get these benefits. Providing cloud-native infrastructure is a key requirement to building a DevOps infrastructure and culture. Note that technology is just one part of a fully successful DevOps mentality, of course.

Kubernetes Won the Container War

In the beginning, many cloud-native container platforms built their own cloud-native technology and infrastructure. Many of these solutions were open-source, but only one took over. Just take a look at these Google Trends of last five years:

Google Trends for Kubernetes (Mesosphere, Cloud Foundry, OpenShift)

In the meantime, most cloud-native infrastructure providers (such as Red Hat OpenShift, Mesosphere, Pivotal Cloud Foundry) moved their whole strategy around supporting Kubernetes. These vendors enhance the user experience and add additional features to differentiate themselves from vanilla Kubernetes. OpenShift made this decision a few years earlier than most others; take a look at how the above trends reflect this decision. Furthermore, Kubernetes is also available as a managed service on all major cloud providers (AWS, Azure, GCP) in the meantime.

Stateful Kubernetes Deployments Using Operator Pattern

Kubernetes was mainly used for stateless deployments in the early phases (for instance, to deploy REST microservices). Today, people deploy everything on Kubernetes because it adds a lot of value, and this includes the Kafka backend and clients.

Stateful deployments of backend services leverage the Kubernetes Operator pattern for many infrastructure components, like databases, messaging, search engines, etc. The implementation of the Operator Pattern includes standard Kubernetes objects like StatefulSets, ConfigMaps, Secrets and Persistent Volumes. However, the secret sauce is the custom Kubernetes Controller and Custom Resource Definitions (CRDs) which implement unique application functionality for the specific stateful deployment.

Challenges Running Kafka on Kubernetes

Apache Kafka became the de facto standard for event streaming platforms. Apache Kafka and its ecosystem provide a powerful option to build reliable, scalable, mission-critical distributed systems. Therefore, as you can imagine, it is harder to operate than a traditional messaging system or database which does not scale elastically without downtime and just uses active/passive for high availability.

Kubernetes environments are similar: very powerful but not easy to operate. Hence, the combination of both Kafka and Kubernetes, does not make it easier. Here are some of the common challenges with running the Apache Kafka ecosystem on Kubernetes:

  • Translating an existing architecture to Kubernetes
  • Failover handling
  • Data rebalancing
  • Communication between ZooKeeper, Kafka Brokers, Clients (Java, REST, Connect, KSQL), Schema Registry, etc.
  • External access from / to outside Kubernetes cluster
  • Persistent storage options on premise and in the cloud
  • Security configuration
  • Rolling upgrades

This is what a Kubernetes Operator has to implement and automate. Consequently, a Kafka Operator sounds like a very good and valuable component.

Confluent Operator as Kafka Operator to Establish a Cloud-Native Kafka Platform

Confluent has extensive experience running Kafka on Kubernetes:

Confluent Experience for Kafka on Kubernetes

Confluent Cloud runs on Kubernetes using a Kafka Operator to offer "Serverless Kafka": Confluent Cloud provides mission-critical SLAs on all three major cloud providers (Google GCP, Microsoft Azure, Amazon AWS), consumption-based pricing and throughput of several GB / sec using a single Kafka cluster. Seems like running Kafka on Kubernetes using a Kafka Operator is not a bad idea.

Slide Deck: Confluent Operator for Kafka Ecosystem on Kubernetes

My slide deck describes the journey and the features of Confluent Operator to deploy and operate Kafka in a cloud-native way similar to how Kafka and its ecosystem (like Kafka Connect, Schema Registry, KSQL) is deployed in Confluent Cloud.

Confluent Operator - A Kafka Operator for Kubernetes

Here is the agenda of the slide deck:

  • Cloud-native vs. SaaS/Serverless Kafka
  • The Emergence of Kubernetes
  • Kafka on K8s Deployment Challenges
  • Confluent Operator as Kafka Operator

Also check out the documentation for Confluent Operator.

Please let me know if you have any comments or feedback.

cloud ,cloud native ,confluent ,kafka ,kafka connectors ,kubernetes ,kubernetes operators ,messaging ,openshift ,stream processing

Published at DZone with permission of Kai Wähner , DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}