Securing Hazelcast With Cert-Manager

DZone 's Guide to

Securing Hazelcast With Cert-Manager

Cert-Manager became a standard way for issuing and rotating certificates in Kubernetes and OpenShift environments. Simple to install. Simple to use. Well int...

· Security Zone ·
Free Resource

Cert-Manager became a standard way of issuing and rotating certificates in Kubernetes and OpenShift environments. Simple to install. Simple to use. Well integrated with Vault and other secret managers. No surprise it's the way to go if you want to set up secure communication between your applications!

In this blog post, I show how to secure Hazelcast communication using keys provisioned with cert-manager. I focus on all necessary steps, from installing cert-manager and issuing certificates, to using them for the Hazelcast member-to-member and client-to-member communication.

All you need to follow this tutorial is:

  • Kubernetes (or OpenShift) cluster.
  • Hazelcast Enterprise license key (get a free trial here).

Step 0: Installing Cert-Manager

For the latest information on how to install cert-manager, please check its installation guide. In general, it's enough to execute two simple commands. Note that you need to have admin access to the Kubernetes cluster (or log in as admin in case of OpenShift).


At this point, you should have cert-manager installed into your cluster.

Step 1: Creating Issuer

Issuer is a resource that represents the Certificate Authority (CA). It uses a private key stored as a Secret in order to issue other keys securing the TLS communication.

You can generate a private key by yourself and put it in the secret ca-key-pair. However, for the sake of simplicity in this tutorial, let's use cert-manager Self-Signed Issuer to generate the private key for us.


We created Self-Signed Issuer and Certificate, which in turn generates a secret ca-key-pair with the private key. Note, that instead of the configuration above, you could directly create the ca-key-pair secret.

Now, when the private key is ready, we can create an Issuer, which will sign keys encrypting Hazelcast communication.


Issuer ca-issuer uses a private key stored in ca-key-pair to sign all keys we need for securing Hazelcast communication. With that in place, we are ready to configure the Hazelcast cluster.

Step 2: Starting Secure Hazelcast Cluster

Let's now create a Certificate to secure the communication between Hazelcast members.


A few words about the Certificate we have just defined:

  • It stores keys in the secret named hazelcast-server-tls
  • Keys are signed by ca-issuer, which acts as Certificate Authority
  • Hostname verification is disabled for Hazelcast communication, so the dnsNames property is ignored
  • The same keys are shared among all Hazelcast members

Now, we need to define one more secret with the Hazelcast license key.


As mentioned at the beginning, if you don't have a valid Hazelcast license key, please request a trial license via the Hazelcast website.

Let's also define the Hazelcast configuration in hazelcast.yaml.


A few comments:

  • We use kubernetes configuration to enable automatic discovery between Hazelcast members
  • Placeholders ${serviceName} and ${namespace} are filled with JVM parameters during runtime
  • We use advanced-network with 3 different ports for accessing Hazelcast cluster:
    • 5701 for member to member communication (secured with SSL)
    • 5702 for client to member communication (secured with SSL)
    • 5703 for health checks used in livenessProbe and readinessProbe (not secured because it's not possible to use Kubernetes probes with SSL Mutual Authentication)
  • Certificates (ca.crt, tls.key, tls.crt) are automatically injected into the secret hazelcast-server-tls

Note, that if we used network instead of advanced-network, then Hazelcast health checks would also be secured with the SSL Mutual Authentication. This would, in turn, prevent us from using livenessProbe and readinessProbe.

As the last preparation step, let's put the Hazelcast configuration into a ConfigMap.


We are all set up. Now, we can start a Hazelcast cluster. There are different ways to do it, let's examine three exclusive options: Helm Chart, Hazelcast Operator, and raw Kubernetes configuration.

Step 2.1 (Option 1) Hazelcast Cluster With Helm Chart

Helm is the simplest way to deploy a Hazelcast cluster. Assuming you have the Helm 3 command installed, you can create values.yaml with the following content.


This configuration:

  • Mounts Certificate keys hazelcast-server-tls into Hazelcast pods
  • Sets livenessProbe/readinessProbe port to 5703 as defined earlier in the Hazelcast configuration
  • Uses Secret hz-license-key as Hazelcast license key and ConfigMap hazelcast as Hazelcast configuration
  • Disables securityContext.readOnlyRootFilesystem since OpenSSL library needs write access to the filesystem
  • Disables Hazelcast Management Center

With such configuration in place, we can add Hazelcast Helm Chart repository and start Hazelcast Enterprise deployment.


We successfully installed the Hazelcast cluster using Helm Chart. Let's see how to achieve exactly the same using Hazelcast Operator.

Step 2.2 (Option 2) Hazelcast Cluster with Hazelcast Operator

Operators are more and more popular among Kubernetes and OpenShift users. To install and use Hazelcast Operator, please follow the instructions described in hazelcast/hazelcast-operator. Use the following listing as the resource file hazelcast.yaml.


Parameters are exactly the same as described in step 2.1 related to the Helm Chart method.

Step 2.3 (Option 3) Hazelcast Cluster with Kubernetes configuration

I strongly suggest using the Hazelcast Helm Chart or Hazelcast Operator methods because they incorporate a lot of features that would simply be too long to describe here in the raw Kubernetes configuration. However, if you are forced to use pure Kubernetes or you are simply interested in details, then here's the minimal configuration to start a Hazelcast cluster.


Explanation of all the configuration above is out of the scope for this blog post. However, if you're interested in details, please check the Hazelcast Kubernetes plugin and Hazelcast Kubernetes Code Samples.

Step 3: Verifying Hazelcast Cluster

No matter which installation method you chose, you should now have a running Hazelcast cluster secured with keys from cert-manager. Let's verify that everything is set up correctly.

You should have three running PODs.


They should use SSL for communication.


They should all form one Hazelcast cluster.


When we are sure our cluster started successfully, we can now configure the Hazelcast client application.

Step 4: Starting Hazelcast Client Application

We will start a sample client application in a few steps:

  • Create client Certificate.
  • Create Hazelcast client configuration.
  • Create client application.
  • Deploy client application.

Step 4.1: Creating a Client Certificate

Technically, we could re-use the Certificate that we already created for the Hazelcast cluster. However, it's better to issue separate keys for the client part. The Certificate resource looks very similar to what we saw before.


As the next step, let's create a Hazelcast client configuration.

Step 4.2: Creating Hazelcast Client Configuration

We can define hazelcast-client.yaml with the following content.


A few words of explanation:

  • We use Hazelcast Kubernetes plugin configuration to automatically discover the Hazelcast cluster defined with the service name hazelcast-hazelcast-enterprise in the default namespace
  • We use the SSL configuration which is very similar to the server.

Let's create a ConfigMap with the specified configuration.


With that configuration in place, we can finally create our client application.

Step 4.3: Creating a Client Application

To present the client application, we can use any supported programming language. Here, let's use Java and create a sample application, which connects to the Hazelcast cluster. The application does nothing more than inserting a value to a Hazelcast map and reading it periodically.

That is the code of Main.java.


And here's the related Dockerfile.


The Dockerfile above downloads the necessary dependencies and starts the client application. Let's build and push it into the Docker Hub registry.


Replace leszko with your Docker Hub username. Please also make sure your repository is public (or define necessary Docker credentials in the Kubernetes configuration) before proceeding with the next step.

Step 4.4: Deploying Client Application

As the last step, let's apply the following configuration.


In the logs, we should see that the application successfully connected to the cluster.



What I presented above is the correct way to configure Hazelcast security in the Kubernetes/OpenShift environment. As always with the security aspects, some steps may look superfluous, for example, defining separate Certificates for server and client. Nevertheless, in the long run, that's what you should do to provide the high-security level for your services!

cert-manager, cloud (add topic), hazelcast, helm, kubernetes, openshift, operator, security

Published at DZone with permission of Rafał Leszko . See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}