Istio Service Mesh: Implementing “Zero Risk” Deployment Using Mirroring

In this hands-on exercise, we will build a Kubernetes cluster, install Istio, build a simple dockerized microservice using Spring Boot, deploy it to the cluster and configure mirroring using Virtual Service and Destination Rules of the service mesh.

The exercise assumes a basic knowledge of Kubernetes, Istio, Spring Boot, and Docker. We will use Google Cloud Engine for building the Kubernetes cluster.

All the source code used in this exercise is available here: https://github.com/pmusale/Istio-Tutorials

Docker images are available here: https://cloud.docker.com/repository/list

Pre-Requisites

• You have a Google Cloud Platform account.
• You have installed GCP SDK and configured it to your GCP account.
• You have installed and configured Kubectl using GCP SDK.
• You have installed Docker Quick Start. If you are using the Windows 10 Home Edition, please follow the instructions here.

What Is Mirroring?

As developers, we often desire a way to test our services in a production setup, with real users, but without negatively impacting users. This is generally not easy. We could achieve this goal, to an extent, by implementing a canary release.  However, in case of canary releases, some percentage of users will be directly exposed to the newer version of the service by design.

What if you can replay all the HTTP traffic going to your main service and, at the same time, direct it towards the newer implementation without impacting end users.

Enter Istio mirroring. Istio mirroring can help implement this type of setup. Mirroring is sometimes also referred to as shadowing. With mirroring you get the ability to replay all the HTTP traffic going to your main service and route it to the newer version with no response going back to the end user. This is basically “fire and forget” mode and is very useful for achieving no risk deployments and testing.

All traffic routed to main service

Traffic mirroring to version v2 in “fire and forget” mode

Microservice Application

Let’s get started. We will build a very simple microservice application to test the mirroring concept. The microservice is named "Hello Mirroring Microservice." You can build this service using Spring. Please make sure you choose web as one of the dependencies while creating the app.

Building and Dockerizing Microservices Using Maven

 Let’s look at our Hello Mirroring microservice's REST Controller:

@RestController
public class HelloMirroring {

 @RequestMapping("/mirroring")
    public String welcome() {
 String message ="Hello Mirroring v1";
 System.out.println("Hello Mirroring");

   return message;     
    }
}

This simple RestController is returning a “Hello Mirroring v1” message. The request mapping for this service is /mirroring. 

Let's make sure that we have enabled HTTP access logs in Spring Boot Tomcat by adding the following properties to the application.properties file. This will help us validate mirroring implementations.

server.tomcat.accesslog.buffered=true
server.tomcat.accesslog.directory=logs
server.tomcat.accesslog.enabled=true
server.tomcat.accesslog.file-date-format=.yyyy-MM-dd
server.tomcat.accesslog.pattern=common
server.tomcat.accesslog.prefix=access_log
server.tomcat.accesslog.rename-on-rotate=false
server.tomcat.accesslog.request-attributes-enabled=false
server.tomcat.accesslog.rotate=true
server.tomcat.accesslog.suffix=.log

For building the microservice, you can either pull the code from the git repository or use pre-built Docker images uploaded to Docker Hub.

If you decide to build it from the code, you can go to the root folder of the project folder named istiomirroring and run following command:

mvnw install dockerfile:build 

This will build the image for the microservice and push it to local Docker repository. Please make sure you are running Docker on the machine before you begin building the application.

You can list Docker images in the local repository using the docker images command. Please see the tag information. This tag will be used for building containers with the right versioned image.

Push the images to Docker Hub using the docker push command as shown below:

docker push <Docker repository name>/<image name>

#e.g. docker push pmusale/istiomirroring.

Please go to Docker Hub and make sure your image is successfully uploaded.

Building the Kubernetes Cluster

You can follow the instructions for building a Kubernetes cluster on GCP and installing Istio in my previous post about canary releases here.

Deploying Our Microservice Application to the Kubernetes Cluster

Now let’s deploy our microservice into a Kubernetes cluster and inject Istio controls for the microservice. To achieve this, we need to run the  istioctl kube-inject command on your Kubernetes workload deployment file.

I have named the workload file workloadv2.yaml. This file defines Kubernetes services and deployment for your microservice. The file is available under the configFiles folder in the git repository.

##################################################################################################
# Main service receiving response from ingress 
##################################################################################################
apiVersion: v1
kind: Service
metadata:
  name: istio-mirroring-app
  labels:
    app: istio-mirroring-app
spec:
  ports:
  - port: 8080
    name: http
  selector:
    app: istio-mirroring-app
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: istio-mirroring-app-deployment-v1
spec:
  replicas: 1
  template: # template for the pods
    metadata:
      labels:
        app: istio-mirroring-app
        version: v1
    spec:
      containers:
      - name: istio-mirroring-app
        image: pmusale/istiomirroring:1.0
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: istio-mirroring-app-deployment-v2
spec:
  replicas: 1
  template: # template for the pods
    metadata:
      labels:
        app: istio-mirroring-app
        version: v2
    spec:
      containers:
      - name: istio-mirroring-app
        image: pmusale/istiomirroring:1.0
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
---

Please pay attention to the Deployment section. We have created two deployments for the same service: version v1 and version v2. We will use version v1 as the main service and v2 for mirroring. Now run istioctl kube-inject on the workload file

istioctl kube-inject -f  workloadv2.yaml > workloadv2-inject.yaml

This will create the workloadv2-inject.yaml file with all the configurations needed by Istio.

Execute the following command on the file to deploy your application to Kubernetes with Istio enabled:

kubectl apply -f workloadv2-inject.yaml 

Validate your deployment using the following command:

kubectl get services 

Please note that there is no External IP for the service. The only way to access the service is to configure the ingress gateway to point to the service.

Now let’s configure the ingress gateway, virtual service, and destination rule. First, let’s configure the virtual service in such a way that all the traffic is routed to version v1 of the service.

kubectl apply -f destination-rule.yaml 

Here is the destination-rule.yaml file:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-mirroring-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: istio-mirroring-app
spec:
  hosts:
  - "*"
  gateways:
  - istio-mirroring-gateway
  http:
  - route:
    - destination:
        host: istio-mirroring-app
        subset: v1
        port:
          number: 8080          
      weight: 100
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: istio-mirroring-app
spec:
  host: istio-mirroring-app
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
--- 

If you take a closer look at this file, you will see that the ingress controller is routing all the traffic into the service mesh. We have used Istio Virtual Service to direct all the traffic to subset v1 for our service. This is achieved by assigning a weight of 100 to subset v1. Subsets are defined using the destination rule. Subset v1 is pointing to version v1 and subset v2 is pointing to version v2 of the service.

Check your ingress controller's externaIPip usinthe g following command:

Now go to the browser and generate some traffic by hitting the URL.

Now let’s check the access logs on Tomcat by directly logging in to the pod for each service.

You can use the following command to log in to the pod.

kubectl -it exec <pod name> sh

Let’s get the list of pods by executing the following command:

Kubectl get pods 

Let’s log in to each pod and go to access the logs folder of Tomcat on each pod and open the log file.

kubectl -it exec <pod name> sh

Spring Boot Tomcat's log file is located under the tmp directory. Change into the tmp directory and run ls  to list the folders under tmp.

You will see two Tomcat directories. One with docbase in the name and the other without it. Change to the directory without docbase in the name. The log folder is located under this directory. You can run vi to open the log file. Look at the access log file for both versions. You will see that there are no access log entries for version v2 as all the traffic is currently being routed to version v1.

Let’s look at the log file for version 1:

Now let’s log in to v2 pod and look at the log file. The access log file has no entries in it.

Let’s update the destination rule to implement mirroring now.

Let’s first delete existing Virtual Service and Destination Rule as below:

Kubectl delete -f destination-rule.yaml 

Now apply the updated destination rule and Virtual Service as below:

Kubectl apply -f destination-ruleMirror.yaml 

Let’s look at the destination-ruleMirror.yaml file:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-mirroring-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: istio-mirroring-app
spec:
  hosts:
  - "*"
  gateways:
  - istio-mirroring-gateway
  http:
  - route:
    - destination:
        host: istio-mirroring-app
        subset: v1
        port:
          number: 8080          
      weight: 100
    mirror:
      host: istio-mirroring-app
      subset: v2
      port:
          number: 8080
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: istio-mirroring-app
spec:
  host: istio-mirroring-app
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
---

Please look at Virtual Service section and destinations under it. You can now see that we have added a mirror attribute and made it point to subset v2.

Let’s generate some traffic by hitting the service URL in the browser. Keep refreshing the browser to generate enough access logs. Let’s log in to the mirrored pod, version v2, and view the Tomcat access log file. You can see that it now has access log entries exactly same as version v1 of the service.

What’s happening here?

All the HTTP traffic is still being directed to version v1. However, as we have enabled mirroring on version v2 of the service, version v2 is also being sent the same HTTP traffic in fire and forget mode. Although this is a simplistic application for demonstration purposes, the mirroring feature is very powerful to achieve no risk deployment and testing of your service.