Setting up Apache Pinot on Kubernetes in Under 15 Minutes

As a seasoned professional, I have gained extensive experience as a maintainer of Apache Druid k8s operator and have co-authored and committed to various k8s operators/libraries. Through my experience, I have recognized the need to establish application-specific control planes to facilitate the smooth running of big data on Kubernetes.

In my past roles, I have built Kubernetes control planes for Druid at RillData, Pinot at Startree, and Clickhouse at Chistadata. Having worked on all three OLAP stores, I am now fully dedicated to my latest venture, DataInfra; a company focused on building highly efficient control planes that enable the seamless running of real-time analytics on Kubernetes.

Why Control Plane for Pinot?

We believe that Kubernetes can serve as a control plane for any application, including those with data and stateful sets. While Helm charts are useful for configuration management, they don't maintain the state of the application. That's why we've built a control plane based on the Kubernetes operator pattern that acts as a bridge between your application's requirements and Kubernetes infrastructure.

Pinot control plane for k8s is specifically designed to improve the user experience of running Apache Pinot clusters. As a distributed database, Pinot can be challenging to run on Kubernetes without the right tools. Our project is based on the DSOI Spec and built using the Operator-Runtime library to provide a more user-friendly and Kubernetes-friendly experience.

This project is based on the Kubernetes operator pattern, but it is not exclusively limited to this pattern. Given the complexity of Pinot, relying solely on Kubernetes operators may or may not be sufficient to effectively manage its operations. Our vision for the project is to create a comprehensive set of tools and utilities that enable seamless deployment and operation of Pinot on Kubernetes.

• In this tutorial, we are going to deploy an Apache Pinot cluster on Kind.
• This tutorial can easily run on your local machine.
• Apache Pinot needs Zookeeper for metadata management and is a dependency to run Pinot.
• In this tutorial, we shall install Zookeeper using zookeeper-operator.

Prerequisites

To follow this tutorial, you will need the following:

• The Kind CLI installed.
• The KUBECTL CLI installed.
• Docker up and running.

Install Kind Cluster

Create a Kind cluster on your machine.

kind create cluster --name pinot

Installing Pinot Control Plane

Pinot control plane supports Helm chart-based deployment. To install the chart run the following commands. We will use the Makefile commands to bootstrap the control plane.

• Clone the Repo

git clone https://github.com/datainfrahq/pinot-control-plane-k8s.git
cd pinot-control-plane-k8s/

• Deploy Pinot Control Plane — The following commands deploys the control plane using helm in the pinot-control-plane namespace.

helm repo add datainfra https://charts.datainfra.io
helm upgrade --install \
--namespace pinot-control-plane \
--create-namespace \
pinot-control-plane datainfra/pinot-control-plane

• Deploy Zookeeper Operator and Zookeeper — The following command will take a few minutes to complete. It will deploy the Zookeeper operator in the Zookeeper-operator namespace and zk statefulset in the pinot namespace.

make helm-install-zk-operator

Installing Pinot Cluster

• Install Pinot Cluster

kubectl apply -f tutorials/00-pinot-kind/pinot-basic.yaml -n pinot

Installing Kafka Cluster and Create Topics

• Add Helm Repo and install Kafka cluster

# Add Kafka
helm repo add kafka https://charts.bitnami.com/bitnami
# Deploy kafka
helm install -n pinot kafka kafka/kafka --set replicas=1,zookeeper.image.tag=latest

•  Once Kafka Pods are up and running, create topics

# Add Kafka
helm repo add kafka https://charts.bitnami.com/bitnami
# Deploy kafka
helm install -n pinot kafka kafka/kafka --set replicas=1,zookeeper.image.tag=latest

Create Pinot Schema

• Pinot control plane supports the creation of schema.

kubectl apply -f tutorials/00-pinot-kind/pinotschema-basic.yaml -n pinot

Create Pinot Table

• Pinot control plane supports the creation of tables.

kubectl apply -f tutorials/00-pinot-kind/pinottable-basic.yaml -n pinot

Ingest Data Into Kafka

• Ingest Data into Kafka topics.

kubectl apply -f tutorials/00-pinot-kind/pinot-realtime-kafka.yaml

Access Pinot Console

• Access the Pinot console by port-forward controller pod or svc.

kubectl port-forward pinot-controller-controller-0 -n pinot 9000