Use KubeKey To Set Up a Kubernetes and KubeSphere Cluster With NFS Storage
How to install KubeSphere, a container platform running on top of Kubernetes with streamlined DevOps workflows, unified multi-cluster management, and more.
Join the DZone community and get the full member experience.Join For Free
In my previous articles, I talked about how to use KubeKey to set up and scale a Kubernetes cluster. As you may already know, KubeKey can do way more than that. You can also use KubeKey to install KubeSphere, a container platform running on top of Kubernetes with streamlined DevOps workflows, unified multi-cluster management, and more. Besides, KubeKey is able to install cloud-native add-ons by Chart or YAML files.
Among other things, storage represents one of, if not the most important element as you set up a Kubernetes cluster. Kubernetes itself supports multiple storage solutions, such as NFS-client, Ceph CSI, and GlusterFS (in-tree). In this article, I am going to show you how to use KubeKey to create a Kubernetes and KubeSphere cluster with NFS-client Provisioner providing external storage.
What Is NFS?
Network File System (NFS) provides you with remote access to files and directories on a server machine. It allows multiple clients to access shared resources over a network.
You can install the NFS-client Provisioner as a storage plugin for your Kubernetes cluster while you must configure an NFS server beforehand. Here is how it works: an NFS client mounts a directory on the server machine so that files residing on the NFS server are accessible to the NFS client. The directory, which you create on the server and share with your client, is also called an exported directory. To ensure that your client can access the directory smoothly, you must grant access to them with several commands which I will show you later.
At the same time, a component called
nfs-common (for Debian-based Linux distributions) or
nfs-utils (for CentOS-based Linux distributions) needs to be installed on all the client machines to provide essential NFS client libraries and utilities.
How Does KubeKey Use NFS-client Configurations?
As I said above, you can use KubeKey to install different add-ons by Chart or YAML files. The configuration of these add-ons must be offered in the configuration file (
config-sample.yaml by default) created by KubeKey. To let KubeKey apply these configurations during installation, there are generally two ways:
- Input necessary parameters under the
addonsfield directly in
- Create a separate configuration file for your add-on to list all the necessary parameters and provide the path of the file in
config-sample.yamlso that KubeKey can reference it.
In this article, I will use the second way for demonstration. The general steps are:
- Set up an NFS server.
nfs-utilson all the client machines in your cluster and create a separate configuration file for NFS-client Provisioner on one of the client machines that servers as the taskbox for installation.
- Download KubeKey on the taskbox.
- Use KubeKey to create your cluster configuration file (
config-sample.yaml) and edit it.
- Install Kubernetes, KubeSphere, and NFS-client Provisioner including the storage class.
Here is my node information for your reference:
|192.168.0.2||server||NFS server||Ubuntu 16.04, 4 Cores, 4 G Memory, 100 G Disk|
|192.168.0.3||client1||master, etcd||Ubuntu 16.04, 8 Cores, 8 G Memory, 50 G Disk|
|192.168.0.4||client2||worker||Ubuntu 16.04, 8 Cores, 8 G Memory, 50 G Disk|
|192.168.0.5||client3||worker||Ubuntu 16.04, 8 Cores, 8 G Memory, 50 G Disk|
server machine, where the NFS server will be installed, provides external storage services for three client machines in the cluster.
For more information about requirements for nodes, networks, and dependencies, see one of my previous posts.
Install and Configure an NFS Server
Step 1: Install the NFS Kernel Server
To set up your server machine, you must install the NFS kernel server on it.
Run the following command so that you will be using the latest package on Ubuntu for installation.
Install the NFS kernel server.
Step 2: Create an Exported Directory
Your NFS client will mount a directory on the server machine which has been exported by the NFS server.
Run the following command as root to specify a mount folder name (for example,
/mnt/demo). This is also the directory that will be shared with your client machines.
To ensure that all the clients can access the directory, remove permissions of the folder.
Step 3: Grant Your Client Machine Access To the NFS Server
Run the following command to edit the
/etc/exports file with
Add your client information to the file. Note that if you have multiple directories that you want to share with your clients, you need to add them all to the file. Here is the syntax:
If you have multiple client machines, add each line for all of them. Alternatively, specify a subnet in the file so that all the clients within it can access the NFS server. For example:
rw: Read and write operations. The client machine will have both read and write access to the volume.
sync: Changes will be written to disk and memory.
no_subtree_check: Prevent subtree checking. It disables the security verification required for a client to mount permitted subdirectories.
- Save the file.
Step 4: Apply the Configuration
Run the following command to export your shared directory.
To make your configuration effective, restart the NFS kernel server.
Configure the Client Machine
Now that we have our server machine ready, we need to install
nfs-common on all of our clients. It provides necessary NFS functions while you do not need to install any server components.
Likewise, execute the following command to make sure you are using the latest package.
nfs-common on all the clients.
Go to one of the client machines (taskbox) where you want to download KubeKey later (for example,
client1). Create a configuration file that contains all the necessary parameters of your NFS server which will be referenced by KubeKey during installation.
Here is my configuration for your reference:
- If you want to configure more values, see chart configurations for NFS-client.
storageClass.defaultClassfield controls whether you want to set the storage class of NFS-client Provisioner as the default one. If you input
falsefor it, KubeKey will install OpenEBS to provide local volumes, while they are not provisioned dynamically as you create workloads on your cluster. After you install KubeSphere, you can change the default storage class on the console directly.
- Save the file. Now you can use KubeKey to install both Kubernetes and KubeSphere, together with NFS-client Provisioner.
Download KubeKey and Install a Cluster
Download KubeKey from its GitHub Release Page or use the following command to download KubeKey version 1.0.1. You only need to download KubeKey to one of your machines that serves as the taskbox for installation.
The above command downloads KubeKey and unzips the file. Your folder now contains a file called
kk. Make it executable.
Specify a Kubernetes version and a KubeSphere version that you want to install. For more information about supported Kubernetes versions, see this list.
A default file
config-sample.yaml will be created if you do not customize the name. Edit the file.
Pay special attention to the field of
addons. For more information about each parameter in this file, you can see one of my previous posts, or have a look at this example file and explanations for the
addons field. Note that you can also enable pluggable components of KubeSphere in this file, such as DevOps, service mesh, and App Store.
Save the file and execute the following command to install Kubernetes and KubeSphere:
When the installation finishes, you can inspect installation logs with the following command:
You can see the following message if the installation is successful:
You can verify that NFS-client has been successfully installed either from the command line or from the KubeSphere web console.
Run the following command to check your storage class.
I did not set
nfs-client as the default storage class so KubeKey installed OpenEBS for me as well. Expected output:
Run the following command to check the statuses of Pods.
nfs-client is installed in the namespace
kube-system. Expected output (exclude irrelevant Pods):
ks-console Service is being exposed through a NodePort. Log in to the console at
<node IP>:30880 with the default account and password (
admin/P@88w0rd). You may need to open the port in your security groups and configure relevant port forwarding rules depending on your environment.
Click Platform in the top left corner and go to Cluster Management. In Storage Classes under Storage, you can see two storage classes:
Go to Pods under Application Workloads, the Pod of
nfs-client is also functioning well in the
You can create a PVC in a project and verify the persistent volume bound to it can be mounted to a workload successfully.
For more information about how to create a PVC on the KubeSphere console, see Volumes.
Opinions expressed by DZone contributors are their own.