{{announcement.body}}
{{announcement.title}}

Kubeflow in IBM Kubernetes Services (IKS)

DZone 's Guide to

Kubeflow in IBM Kubernetes Services (IKS)

Kubeflow makes life simpler by allowing deployment of Machine Learning workflows on Kubernetes. Learn more in this tutorial!

· AI Zone ·
Free Resource

Introduction

Kubeflow makes life simpler by allowing deployment of Machine Learning workflows on Kubernetes. This brings in the flexibility of scale and portability.

In this tutorial you will learn: 

  1. Provisioning of a Kubernetes cluster in IBM Cloud and installing the required tools
  2. Installing Kubeflow
  3. Launching the Kubeflow dashboard

Prerequisites

To run this tutorial, you need:

  1. An Ubuntu 18 machine with a minimum 8 cores, 16 GB RAM, and 250 GB storage
  2. Root privilege on the ubuntu machine
  3. IBM Cloud account

Deployment Setup

IBM Cloud UI

Log in to IBM Cloud ( https://cloud.ibm.com ). Go to Catalog, under services you can find Kubernetes service.

In this demo, I have used the Kubernetes 1.16.14 (Stable) version in a classic infrastructure in a single zone region. Once you choose the version, select Geography ( Asia Pacific), Availability (Single Zone), Worker Zone (Chennai 01) 

The worker node size used in this setup was 4 vCPUs and 16  GB Memory with 1 worker node per zone.

The master service endpoint is set to Public endpoint only.

Choose the cluster name wisely, you can leave it default but recommend to use something of your choice.

Create the cluster 

Ubuntu Machine

Till the cluster setup is up, log in to the Ubuntu Machine 

Download the necessary tools for IBM Cloud.

Shell
 




x


1
curl -sL https://ibm.biz/idt-installer | bash



Login in to your IBM Cloud Account, the region, and resource group might change based on your selection.

Shell
 




xxxxxxxxxx
1


1
ibmcloud login -a cloud.ibm.com -r jp-tok -g default



Set the Kubernetes context to your cluster for this terminal session.

Shell
 




x


 
1
ibmcloud ks cluster config --cluster "YOUR CLUSTER ID"



Verify that you can connect to your cluster.

Shell
 




xxxxxxxxxx
1


 
1
kubectl config current-context



Note: In case you forget your cluster setting, you can retrieve this information by visiting https://cloud.ibm.com/kubernetes/clusters and clicking Access under your cluster.

To verify your setting, run the following code.

Shell
 




xxxxxxxxxx
1


 
1
kubectl version --short



The following should be your output.

Shell
 




xxxxxxxxxx
1


1
Client Version: v1.17.11
2
Server Version: v1.16.14+IKS



IBM Cloud Block Storage Setup 

When using the classic worker nodes provider of IBM Cloud Kubernetes cluster, by default, it uses IBM Cloud File Storage based on NFS as the default storage class. File Storage is designed to run RWX (read-write multiple nodes) workloads with proper security built around it. Therefore, File Storage does not allow group securityContext which is needed for DEX and Kubeflow Jupyter Server.

IBM Cloud Block Storage provides a fast way to store data and satisfy many of the Kubeflow persistent volume requirements such as fsGroup out of the box and optimized RWO (read-write single node) which is used on all Kubeflow’s persistent volume claim.

Check if there is a default storage class setup to Block Storage.

Shell
 




xxxxxxxxxx
1


1
kubectl get storageclass | grep \(default\)



If not, we need to set up IBM Cloud storage.

Follow the instructions to install the Helm version 3 client on your local machine.

Add the IBM Cloud Helm chart repository to the cluster where you want to use the IBM Cloud Block Storage plug-in.

Shell
 




xxxxxxxxxx
1


 
1
helm repo add iks-charts https://icr.io/helm/iks-charts
2
 helm repo update
3
 helm install 1.6.0 iks-charts/ibmcloud-block-storage-plugin -n kube-system



The following code shows the sample output.

Shell
 




xxxxxxxxxx
1


1
NAME: 1.6.0
2
 LAST DEPLOYED: Fri Mar 20 11:38:21 2020
3
 NAMESPACE: kube-system
4
 STATUS: deployed
5
 REVISION: 1
6
 NOTES:
7
 Thank you for installing: ibmcloud-block-storage-plugin.   Your release is named: 1.6.0



Set the block storage as default, ensure that there is only one default storage

Shell
 




xxxxxxxxxx
1


 
1
kubectl patch storageclass ibmc-block-gold -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
2
 
          
3
# Check the default storageclass is block storage
4
kubectl get storageclass | grep \(default\)



Installing Kubeflow v.1.1.0

You can create your directory, I have created a kubeflow directory under root. 

Shell
 




xxxxxxxxxx
1
15


 
1
mkdir -p /root/kubeflow/v1.1.0
2
cd /root/kubeflow/v1.1.0
3
wget https://github.com/kubeflow/kfctl/releases/download/v1.1.0/kfctl_v1.1.0-0-g9a3621e_linux.tar.gz
4
tar -xvf kfctl_v1.0.1-0-gf3edb9b_linux.tar.gz
5
export PATH=$PATH:/root/kubeflow/v1.1.0
6
export KF_NAME=my-kubeflow
7
export BASE_DIR=/root/kubeflow/v1.1.0
8
export KF_DIR=${BASE_DIR}/${KF_NAME}
9
# config file is IBM specific
10
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_ibm.v1.1.0.yaml"
11
 
          
12
mkdir -p ${KF_DIR}
13
cd ${KF_DIR}
14
kfctl apply -V -f ${CONFIG_URI}



When the installation is complete which might take a few mins, please ensure that all pods are running. Only then proceed to the next steps. If not you might have to recreate this exercise.

Verify if all pods are running:

Shell
 




xxxxxxxxxx
1


 
1
kubectl get pod -n kubeflow



The sample output below:

output

Launching the Kubeflow Dashboard

Kubeflow Dashboard can be accessed through the istio-ingressgateway service. To see your setting of istio-ingressgateway service, execute the following code.

Shell
 




xxxxxxxxxx
1


 
1
kubectl get -n istio-system svc/istio-ingressgateway



You might get the following output:

Shell
 




x


 
1
NAME                   TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                                                                                                      AGE
2
istio-ingressgateway   NodePort   111.11.11.111   <none>        TCP



If your output has a NodePort TYPE (not LoadBalancer) and EXTERNAL-IP is blank like above, run the following command to change the type from NodePort to LoadBalancer. 

Shell
 




xxxxxxxxxx
1


 
1
kubectl edit -n istio-system svc/istio-ingressgateway



loadbalancer

Run the following command again to get the EXTERNAL-IP.

Shell
 




xxxxxxxxxx
1


 
1
kubectl get -n istio-system svc/istio-ingressgateway



Now you can access the Kubeflow dashboard in a web browser: http://<EXTERNAL-IP>:<PORT>

kubeflow

Summary

That's all folks! In this tutorial, I have explained to you how to set up Kubeflow on IBM Kubernetes Service (IKS). I hope you liked it.

Topics:
ai, ibm cloud, iks, kubeflow, kubernetes, tutorial

Published at DZone with permission of Deepak Rai . See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}