Private DNS Zone With Azure HDInsight Kafka

What Is HDInsight Kafka?

Azure HDInsight Kafka is a cloud-based, managed Apache Kafka service offered by Microsoft Azure. Apache Kafka is an open-source, distributed streaming platform that allows for the processing and analyzing of high-volume, real-time data streams.

Azure HDInsight Kafka provides a fully managed and scalable environment for deploying and running Apache Kafka clusters, allowing users to easily create, configure, and manage Kafka topics, producers, and consumers. It also provides integration with other Azure services, such as Azure Data Lake Storage, Azure Event Hubs, and Azure Stream Analytics, enabling users to ingest, process, and analyze large volumes of data in real-time.

With Azure HDInsight Kafka, users can leverage the benefits of Apache Kafka without the hassle of managing the underlying infrastructure, allowing them to focus on building real-time data pipelines and applications to support their business needs.

HDInsight Architecture

The following diagram shows a typical Kafka configuration that uses consumer groups, partitioning, and replication to offer a parallel reading of events with fault tolerance:

Challenge

The Azure DNS Private Zone auto registration feature provides DNS record management for virtual machines deployed in a virtual network. When you link a virtual network with a private DNS zone and enable auto registration for all the virtual machines, the DNS records for the virtual machines deployed in the virtual network are automatically created in the private DNS zone.

However, the Azure HDInsght Kafka cluster created under your Virtual New broker doesn't get auto-linked to the Private DNS attached to the virtual network. So, if you would like to call the Kafka broker, you can not do it. Kafka Broker information is very important — clients need to pass to Kafka broker names to connect with the Kafka cluster.

Most of the time, customers use the default broker names provided by Azure (provided by Ambari UI)  and connect with the cluster. This works perfectly fine for most customers. However, enterprise customers want to use Private DNS records for calling every object they have in their network. These allow enterprise customers to call resources among multiple networks with ease.

Solution

Follow the below steps to integrate the private DNS zone with HDInsight Kafka:

1. Enable the auto registration of the foundation level DNS e.g local.company.net for the HDInsight VNET — Click here for more info.
2. Using Ambari UI or API — Update Kafka-env template to set the Kafka listener property with DNS local.company.net
   
   #Configure Kafka to advertise host name based on Private DNS Zone
HOST_NAME=$(hostname) + '.local.company.net' 
echo advertised.listeners=$HOST_NAME sed -i.bak -e '/advertised/{/advertised@/!d;}' /usr/hdp/current/kafka-broker/conf/server.properties 
echo "advertised.listeners=PLAINTEXT://$HOST_NAME:9092" >> /usr/hdp/current/kafka-broker/conf/server.properties 

Benefits

Azure Private DNS integration with HDInsight Kafka provides the following benefits:

• Removes the need for custom DNS solutions. Many customers created custom DNS solutions to manage DNS zones in their virtual network. You can now manage DNS zones using the native Azure infrastructure, which removes the burden of creating and managing custom DNS solutions.
• Automatic hostname record management. Along with hosting your custom DNS records, Azure automatically maintains hostname records for the Kafka brokers in the specified virtual networks. In this scenario, you can optimize the Broker names you use without needing to create custom DNS solutions or modify applications.
• Hostname resolution between virtual networks. Unlike Azure-provided default Kafka Broker names, private DNS zones can be shared between virtual networks. This capability simplifies cross-network and service-discovery scenarios, such as virtual network peering. Kafka clusters can easily discover among virtual networks.
• Split-horizon DNS support. With Azure DNS, you can create zones with the same name that resolve to different answers from within a virtual network and the public internet. A typical scenario for split-horizon DNS is to provide a dedicated version of a service for use inside your virtual network.