{{announcement.body}}
{{announcement.title}}

Manage Azure Event Hubs With Azure Service Operator on Kubernetes

DZone 's Guide to

Manage Azure Event Hubs With Azure Service Operator on Kubernetes

How to use Kubernetes to provision Azure services

· Cloud Zone ·
Free Resource

Azure Service Operator is an open source project to help you provision and manage Azure services using Kubernetes. Developers can use it to provision Azure services from any environment, be it Azure, any other cloud provider, or on-premises — Kubernetes is the only common denominator!

It can also be included as a part of CI/CD pipelines to create, use and tear down Azure resources on-demand. Behind the scenes, all the heavy lifting is taken care of by a combination of Custom Resource Definitions which define Azure resources and the corresponding Kubernetes Operator(s) which ensure that the state defined by the Custom Resource Definition is reflected in Azure as well.

In this blog post:

  • You will get a high level overview of Azure Service Operator (sometimes referred to as ASO in this blog).
  • How to set it up and use it to provision Azure Event Hubs.
  • Deploy apps to Kubernetes which use the Azure Event Hubs cluster.

All the artifacts are available on this GitHub repo https://github.com/abhirockzz/eventhubs-using-aso-on-k8s

Getting started....

Azure Service Operator supports many Azure services, including databases (Azure Cosmos DB, PostgreSQL, MySQL, Azure SQL etc.), core infrastructure components (Virtual Machines, VM Scale sets, Virtual Networks, etc.), and others as well.

It also supports Azure Event Hubs, which is a fully managed data streaming platform and event ingestion service with support for Apache Kafka and other tools in the Kafka ecosystem. With Azure Service Operator you can provision and manage Azure Event Hubs namespaces, Event Hub and Consumer Groups.

So, let's dive in without further ado! Before we do that, please note that you will need the following in order to try out this tutorial:

Prerequisites

Start by getting an Azure account if you don't have one already — you can get for FREE! Please make sure you've kubectl and Helm 3 installed as well.

Although the steps outlined in this blog should work with any Kubernetes cluster (including minikube etc.), I used Azure Kubernetes Service (AKS). You can set up a cluster using Azure CLI, Azure portal, or even an ARM template. Once that's done, simply configure kubectl to point to it

Shell


Ok, you're now ready to...

... Install Azure Service Operator

Nothing too fancy about it... just follow the steps to install it using Helm

Start by installing cert-manager.

Setu Up cert-manager

Shell


Authentication

Since the operator will create resources on Azure, we need to authorize it to do so by providing the appropriate credentials. Currently, you can use Managed Identity or Service Principal.

I will be using a Service Principal, so let's start by creating one (with Azure CLI) using the az ad sp create-for-rbac command

Shell


Install

Setup required environment variables:

Shell


Add the repo, and create a namespace.

Shell


Use helm upgrade to initiate setup:

Shell


Before you proceed, wait for the Azure Service Operator Pod to startup

Shell


Setup Azure Event Hubs Components

Start by cloning the repo:

Shell


Create an Azure Resource Group

I have used the southeastasia location. Please update eh-resource-group.yaml if you need to use a different one

Shell


Create Event Hubs namespace

I have used the southeastasia location. Please update eh-namespace.yaml if you need to use a different one

Shell


Once done, you should see this:

Plain Text


You can get details with kubectl describe eventhubnamespaces and also double-check using az eventhubs namespace show.

The namespace is ready, we can now create an Event Hub

Shell


You can get details with kubectl describe eventhub and also double-check using az eventhubs eventhub show.

As a final step, create the consumer group

This is addition to the default consumer group (appropriately named $Default)

Shell


You can get details with kubectl describe consumergroup and also double-check using eazventhubs eventhub consumer-group show.

What's Next?

Let's make use of what we just setup! We'll deploy a pair of producer and consumer apps to Kubernetes that will send and receive messages from Event Hubs respectively. Both these client apps are written in Go and use the Sarama library for Kafka. I am not going to dive into the details since they are relatively straightforward

Deploy the consumer app:

Shell


Keep a track of the logs for the consumer app:

Shell


You should see something similar to:

Plain Text


Using another terminal, deploy the producer app:

Shell


Once the producer app is up and running, the consumer should kick in, start consumer the messages and print them to the console. So you'll see logs similar to this:

Plain Text

In case you want to check producer logs as well: kubectl logs -f $(kubectl get pods -l=app=eh-producer --output=jsonpath={.items..metadata.name})

Alright, it worked!

  • We created an Event Hubs namespace, Event Hub along and a consumer group.. all using kubectl (and YAMLs of course)
  • Deployed a simple producer and consumer for testing

But, What Just Happened ...?

... how did the consumer and producer apps connect to Event Hubs without connection info, credentials etc.?

Notice this part of Event Hub manifest (eh-hub.yaml file):

YAML


secretName: eh-secret ensured that a Kubernetes Secret was created with the required connectivity details including connection strings (primary, secondary), keys (primary, secondary), along with the basic info such as Event Hubs namespace and hub name.

The producer and consumer Deployments were simply able to refer to this. Take a look at this snippet from the consumer app Deployment

YAML


The app uses env vars EVENTHUBS_CONNECTION_STRING, EVENTHUBS_NAMESPACE and EVENTHUBS_TOPIC whose values were sourced from the Secret (eh-secret). The value for EVENTHUBS_CONSUMER_GROUPID is hardcoded to eh-aso-cg which was the name of the consumer group specified in eh-consumer-group.yaml.

Clean up

To remove all the resources including Event Hubs and the client apps, simply use kubectl delete -f deploy

Conclusion

Azure Service Operator provides a layer of abstraction on top Azure specific primitives. It allows you to manage Azure resources and also provide ways to connect to them using other applications deployed in the same Kubernetes cluster.

I covered Azure Event Hubs as an example, but as I mentioned earlier, Azure Service Operator also supports other services too. Head over to the GitHub repo and give them a try!

Topics:
azure, distributed systems, go, kubernetes, messaging, open souce, tutorial

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}