Pulsar on KubeSphere: Installing Distributed Messaging and Streaming Platform
Apache Pulsar and KubeSphere are powerful tools for cloud-native applications. In this post, we'll demonstrate how to deploy Pulsar on KubeSphere from scratch.
Join the DZone community and get the full member experience.Join For Free
KubeSphere, an open-source container platform running on Kubernetes, provides users with an app-centric experience. In this connection, it features a comprehensive set of tools for developers to manage apps across their entire lifecycle. In this article, I will demonstrate how to install Apache Pulsar on a KubeSphere cluster as an example. Apache Pulsar, a cloud-native, distributed messaging and streaming tool, represents a go-to platform to meet the real-time event-streaming needs of enterprises.
Before You Begin
To install Pulsar on KubeSphere, you need to do the following beforehand:
- Create a Kubernetes cluster with KubeSphere installed. You can deploy both Kubernetes and KubeSphere using KubeKey on public cloud instances or in an air-gapped environment. KubeKey is a lightweight, brand-new Kubernetes installer. It installs Helm 3.0+ on your Kubernetes cluster by default, which is also a must for Apache Pulsar. Alternatively, use a managed Kubernetes service such as Google Kubernetes Engine (GKE) or Azure Kubernetes Servce (AKS) to create a Kubernetes cluster, and then install KubeSphere with ks-installer. In production, it is recommended that you choose the second option because managed services can free you from additional settings (for example, persistent storage and load balancers) as you deploy workloads.
- Enable the KubeSphere App Store. As you can notice from the Pulsar documentation, it can be installed on a Kubernetes cluster through its Helm chart. The KubeSphere App Store allows you to add a Helm repository and install apps in it on KubeSphere as app templates on the KubeSphere console. The KubeSphere App Store is powered by OpenPitrix, an open-source web-based system to package, deploy, and manage different types of apps across their entire lifecycle.
In my environment, I created a Kubernetes cluster through Google Kubernetes Engine. The following table lists the machines in my cluster for your reference. Feel free to configure your cluster based on your own needs.
Installing Pulsar as an App Template on KubeSphere
KubeSphere boasts a highly interactive and responsive user interface where tenants with different roles can perform corresponding actions directly. That said, admin users can run commands with
kubectl in its web-based terminal as always. For demonstration purposes, I used the built-in account
admin throughout the entire steps.
Note: In fact, with the multi-tenant system of KubeSphere, you can easily set accounts and assign roles to them without manually creating RBAC-related resources like
RoleBinding objects. With them in place, it is the responsibility of each user to do their own job in a cluster, such as adding an app repository and deploying an app. Check out this quickstart article in the KubeSphere documentation to learn more.
Now, let’s get started with our journey.
Log in to the KubeSphere console with the default account and password (
admin/P@88w0rd). Keep in mind that the console Pod, controlled by a Deployment (
kubesphere-systemnamespace), is exposed with a Kubernetes Service called
NodePortby default. That means you can access it by using any of the node IP addresses in your cluster with the port number (port
30880). If you are using a Kubernetes cluster hosted on GKE like me, it is recommended that you change the Service type to
LoadBalancer. In this way, Google will assign a public IP address to the Service automatically, helping route traffic to the Pod in a more secure and reliable way.
- In the KubeSphere console, go to Access Control in Platform. This is the place where you manage all workspaces, accounts, and platform roles. To manage Pulsar resources in a unified way, create a workspace first. As you can see from the image below, there is a workspace called
pulsarin my cluster.
Click the workspace and navigate to App Repositories under App Management. The App Repositories page lists all repositories added in the selected workspace. Add the Pulsar repository as below:
Note that adding the Pulsar repository is only one of the multiple ways to install it on KubeSphere. You can also upload its Helm chart file as an independent app template and release it to the KubeSphere App Store.
In addition to the repository, we also need Secrets for Pulsar. The username (
pulsar) and password (
pulsar) stored in them can be used to log in to the Grafana dashboard and Pulsar Manager later. You can create these Secrets one by one, but I highly advise you to use a script provided in the Pulsar documentation. Clone a GitHub repository and run the script to set these Secrets automatically.Shell
git clone https://github.com/apache/pulsar-helm-chart cd pulsar-helm-chart ./scripts/pulsar/prepare_helm_release.sh \ -n pulsar \ -k pulsar-mini \ -c
- The above commands create a namespace called
pulsarin your Kubernetes cluster. To add it to the
pulsarworkspace, go to Cluster Management in Platform and manually assign it to the workspace.
For more information, see Add Existing Kubernetes Namespaces to a KubeSphere Workspace.
- Now let’s get back to the
pulsarworkspace. We can see that the
pulsarnamespace has been added. In KubeSphere, a Kubernetes namespace is known as a KubeSphere project. Click it to see details.
values.yamlfile in this step. When you are done, click Install. Keep in mind that you need to set the value of
truewhen installing Pulsar the first time. This command installs and starts Pulsar.
- Wait for Pulsar to be up and running. Switch between Services, Workloads, Jobs, and Pods to check all resources are ready.
pulsar-admin to Create Pulsar Tenants, Namespaces, and Topics
pulsar-admin is the command-line interface (CLI) tool for Pulsar. You can perform the following steps to create resources with
Navigate to the
toolsetPod and click the terminal icon on its detail page to access its container.
A terminal will appear, where we can run commands in the container of the Pod directly. Create a tenant named
apacheand list all the tenants.Shell
bin/pulsar-admin tenants create apache bin/pulsar-admin tenants list
toolsetcontainer, create a namespace named
pulsarfor the tenant
apacheand list the namespaces of the tenant
apacheto see if the namespace is created successfully.Shell
bin/pulsar-admin namespaces create apache/pulsar bin/pulsar-admin namespaces list apache
Accessing Pulsar Manager
Pulsar Manager is a web-based visualized tool for managing and monitoring Pulsar. We can follow the steps below to access it.
- Navigate to the
pulsar-managerService and go to its detail page. By default, the Pulsar Manager Pod is controlled by a Deployment (
pulsar-mini-pulsar-managerin my example), exposed by a Service via
LoadBalanceras shown below.
- If you are using a managed Kubernetes service, the provider will assign a public IP address to it. Access Pulsar Manager using the IP address (
188.8.131.52) and the Service port (
If no available public IP address is assigned for the
LoadBalancerService (this may happen if your cluster is created on cloud instances directly without load balancers or in a private environment), access Pulsar Manager using a node IP address and the node port. However, you may need to configure relevant firewall rules or port forwarding rules depending on your environment.
Use the username (
pulsar) and password (
pulsar) to log in to Pulsar Manager.
Create an environment as needed.
Both Apache Pulsar and KubeSphere have much to offer for enterprise users and open-source enthusiasts. What I have covered in this article only serves as a quickstart for someone who wants to get a glimpse of them. Check out their latest documentation to understand more.
Opinions expressed by DZone contributors are their own.