Implementing EKS Multi-Tenancy Using Capsule (Part 2)

In the first part, we learned what multi-tenancy is, different types of tenant isolation models, challenges with Kubernetes native services, and how to install the capsule framework on AWS EKS.

We will now dive further into creating tenants using the Capsule framework.

Capsule Tenant creation requires an owner (s) assignment. Though we can create local Kubernetes users as tenant owners, we will be using IAM users as tenant owners. This process involves following steps:

How To Set Up Capsule Tenants With AWS EKS

Step 1: Create IAM Policy

The first step in creating a read-only user in EKS is to create an IAM policy that grants read-only access to the EKS cluster. We will create an IAM policy “t-CapsuleDescribeCluster' which only describes permissions on the cluster.

• Sign in to the AWS Management Console and navigate to the IAM dashboard
• In the left-hand menu, select “Policies”
• Click the “Create policy” button
• Choose “JSON” tab for editing policies
• Apply the below IAM policy that allows read-only access to an EKS cluster

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Action": [
                "eks:DescribeCluster"
            ],
            "Resource": "*"
        }
    ]
}

• Click Next. Enter the policy name as “t-CapsuleDescribePolicy” and click ‘Create’

Step 2: IAM User Group

We will create an IAM user group that is associated with the IAM policy created in step 1

• Log in to the AWS Management Console and go to the IAM dashboard
• Click “User groups” in the left-hand menu and then click “Create group”
• Enter name as “capsule.clastix.io”. Click Next
• In the “Attach Permission policies” select, search for “t-CapsuleDescribePolicy” and select the IAM policy created in step 1

• Review the user group details and click “Create user group.”

Step 3: Create IAM Users and Add to the IAM User Group

Follow the below steps to create two IAM users (shiva and ganesha).

• Log in to the AWS Management Console and go to the IAM dashboard
• Click “Users” in the left-hand menu and then click “Add user”
• Enter a name for the user
• Select “Add Users to the group” and select “capsule.clastix.io” group. Click Next

• Review the user details and click “Create user.”

Step 4: Create a ClusterRole and ClusterRoleBinding

We will create a new “k8s-read.yaml” file with below code:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: k8s-read
rules:
- apiGroups:
  - '*'
  resources:
  - deployments
  - pods
  - pods/log
  - configmaps
  - secrets
  - services
  - virtualservices
  - horizontalpodautoscalers
  - gateways
  - namespaces
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: k8s-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: k8s-read
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: k8s-read

Apply the above manifest file using kubectl.

kubectl apply -f k8s-read.yaml

Step 5: Map User in aws-auth Config Map

We will need to add these users in aws-auth configmap to grant read access to EKS cluster.

Open the aws-auth configmap for editing by running the below command.

kubectl -n kube-system edit configmap aws-auth

Go to the IAM Dashboard. Navigate to the users section and copy the userarn for shiva and ganesha. These will be used to update the mapUsers section in the config map.

userarn: arn:aws:iam::<account-number>>:user/shiva
userarn: arn:aws:iam::<<account-number>>:user/ganesha

Add user in mapUsers section. Here is a sample code:

    - groups:
      - k8s-read
      - capsule.clastix.io
      userarn: arn:aws:iam::<account-number>:user/<user-name>
      username: shiva

Step 6: Create Access Keys for the IAM Users (shiva and ganesha)

Repeat the below steps to create access keys for IAM users (shiva and ganesha)

• Go to IAM Dashboard and navigate to the users section
• Select the user. Click the “security credentials” tab
• From the Access Keys section, click create Access Keys
• Select the use case ”Command Line Interface (CLI)” option and confirm

• Copy the access key and secret access key generated for both the users.

Step 7: Configure AWS CLI for IAM Users (shiva and ganesha)

Configure AWS CLI for the shiva profile. Enter the shiva access key and secret access key from the above table.

aws configure --profile shiva

Configure AWS CLI for the ganesha profile. Enter the ganesha access key and secret access key from the above table.

aws configure --profile ganesha

Capsule Tenant’s Creation and Verification

Create Single Tenant With Single Tenant Owner

As a cluster administrator, in the "eks-cluster1" cluster, create a new tenant oil with IAM user ‘shiva’ as the owner — as shown in the below yaml file:

aws eks --region us-east-1 update-kubeconfig --name eks-cluster1
kubectl apply oil.yaml

Oil.yaml file is as follows:

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: oil
spec:
  owners:
  - kind: User
    name: shiva

As a cluster admin, get the available tenants in the cluster.

kubectl get tenants

The result will have a namespace count as '0'.

Login as tenant owner, shiva, and verify if shiva can access tenants.

aws eks --region us-east-1 update-kubeconfig --name eks-cluster1 --profile shiva
kubectl get tenants

Shiva should get the below error:

As a tenant owner, shiva, create a new namespace within the oil tenant by running the below command.

kubectl create ns oil-production

Now, as a cluster admin, verify that the created namespace is mapped to the oil tenant.

aws eks --region us-east-1 update-kubeconfig --name eks-cluster1
kubectl get tenants

The results show the namespace count is increased to '1'.

As a cluster admin, verify the tenant ‘oil’ description.

kubectl get tenant oil -o yaml

The results show that the namespace ‘oil-production’ is mapped to the tenant ‘oil’.

Create Single Tenant With Multiple Tenant Owners

As a cluster administrator, in the "eks-cluster1" cluster, create a new tenant oil with IAM user ‘shiva’ as the owner as shown in the below yaml file:

aws eks --region us-east-1 update-kubeconfig --name eks-cluster1
kubectl apply oil_multi_owners.yaml

oil_multi_owners.yaml file is as follows:

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: oil
spec:
  owners:
  - name: shiva
    kind: User
  - name: ganesha
    kind: User

Now, log in as a tenant owner ‘ganesha’, and create a new namespace within the oil tenant by running the below command.

kubectl create ns oil-development

Now, as a cluster admin, verify that the created namespace is mapped to the oil tenant.

aws eks --region us-east-1 update-kubeconfig --name eks-cluster1
kubectl get tenants

The results show the namespace count is increased to '2'.

As a cluster admin, verify the tenant ‘oil’ description.

kubectl get tenant oil -o yaml

The results show that the namespace ‘oil-development’ is mapped to the tenant ‘oil’ in the namespaces section shown below.

Create Multiple Tenants With Single Tenant Owner

As a cluster administrator, in the "eks-cluster1" cluster, create two new tenants — oil and gas — with the same IAM user ‘shiva’ as the owner as shown in the below yaml file:

kubectl apply -f oil.yaml
kubectl apply -f gas.yaml

The oil and gas yaml files would look like the following:

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: oil
spec:
  owners:
  - name: shiva
    kind: User

# gas.yaml
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: gas
spec:
  owners:
  - name: shiva
    kind: User

As tenant owner ‘shiva’, create two new namespaces — ‘oil-production' and 'gas-production’.

Note: As shiva has access to all namespaces belonging to both the oil and gas tenants, namespace will not be created by running the command “kubectl create ns oil-production” or “kubectl create ns gas-production.“

Running these commands, Capsule will deny with the following message: Unable to assign namespace to tenant. Please use capsule.clastix.io/tenant label when creating a namespace. 

The tenant owner has to specify the “tenant name” as a label — capsule.clastix.io/tenant=<desired_tenant> — in the namespace manifest and has to be applied using kubectl command. 

The sample manifest for gas-production looks like the below:

# gas-production.yaml
kind: Namespace
apiVersion: v1
metadata:
  name: gas-production
  labels:
    capsule.clastix.io/tenant: gas

The tenant owner has to apply the manifest file to create the namespaces.

# Create namespace gas-production 
kubectl apply -f gas-production.yaml 

# Create namespace oil-production 
kubectl apply -f oil-production.yaml

Create Multiple Tenants With Different Tenant Owners

As a cluster administrator, in the "eks-cluster1" cluster, create a new tenant oil with IAM user ‘shiva’ as the owner, as shown in the below yaml file:

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: oil
spec:
  owners:
  - kind: User
    name: shiva

As a cluster administrator, create a new tenant gas with IAM user ‘ganesha’ as the owner, as shown in the below yaml file:

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: gas
spec:
  owners:
  - kind: User
    name: ganesha

Expose Service and Verify “Block” Network Access Between Tenant namespaces

As the oil tenant owner, run the following commands to create a namespace and resources in the given tenant.

aws eks --region us-east-1 update-kubeconfig --name eks-cluster1 --profile shiva
kubectl create ns oil-production
kubectl -n oil-production run webserver --image nginx:latest
kubectl -n oil-production expose pod webserver --port 80

As the gas tenant owner, run the following commands to create a namespace and resources in the given tenant.

aws eks --region us-east-1 update-kubeconfig --name eks-cluster1 --profile ganesha
kubectl create ns gas-production
kubectl -n gas-production run webserver --image nginx:latest
kubectl -n gas-production expose pod webserver --port 80

As a cluster admin, get the services in the namespaces created. One would note that the services are exposed as type “ClusterIP” by default. ClusterIP is used for pod-to-pod communication within the same Kubernetes cluster.

kubectl -n oil-production get svc
kubectl -n gas-production get svc

As the first tenant owner, verify you can access the service within the tenant namespace but not the second tenant namespace.

kubectl -n oil-production exec webserver -- curl http://webserver.oil-production.svc.cluster.local
kubectl -n gas-production exec webserver -- curl http://webserver.gas-production.svc.cluster.local

As the second tenant owner, verify you can access the service within the tenant namespace but not the first tenant namespace.

kubectl -n oil-production exec webserver -- curl http://webserver.oil-production.svc.cluster.local
kubectl -n gas-production exec webserver -- curl http://webserver.gas-production.svc.cluster.local

Summary

In this part, we have learned how to create single or multiple tenants with single or multiple AWS IAM users as tenant owners. In the next part of the series, we will further dive into how to configure namespace options, resource quotas, and limit ranges.