Developing A Spring Boot Application for Kubernetes Cluster: A Tutorial [Part 4]

This is the final installment of this four part series. Check out part 1, part 2, and part 3 here. 

Service Deployment Into Kubernetes

So far we have created a Kubernetes cluster in Amazon EC2 environment. We have also developed our sample application consisting of service and web layers and deployed its components into a private repository in Docker Hub. In the remainder of this article, we will discuss how to retrieve the application from Docker Hub and deploy it into the cluster. Throughout this section, all commands should be executed as root in Kubernetes master node.

Secret Key

The first step is to create a secret key to access Docker Hub. Execute

kubectl create secret docker-registry regcred --docker-server=https://index.docker.io/v1/ --docker-username=<username> --docker-password=<password> --docker-email=<email>

where username, password and email are the username, password and email associated with the Docker Hub private repository user. Note that value of --docker-server parameter is the repository server for Docker Hub. Now, you can execute

kubectl get secret regcred --output=yaml

to describe in YAML format the newly created key. The output will be similar to below.

apiVersion: v1
data:
  .dockerconfigjson: eyJhdXRocyI6eyJodHRwczov...
metadata:
  creationTimestamp: 2018-08-01T19:27:46Z
  name: regcred
  namespace: default
  resourceVersion: "4648"
  selfLink: /api/v1/namespaces/default/secrets/regcred
  uid: f7e846b0-95c0-11e8-8ade-066280576724
type: kubernetes.io/dockerconfigjson

Node Labels

Referring to the output of kubectl get nodes executed previously, we consider worker nodes ip-172-31-16-16 and ip-172-31-42-220 as Web-1 and Web-2, and ip-172-31-33-22 and ip-172-31-35-232 as Service-1 and Service-2, respectively. We will create a label named servicetype and set its value to webservice in Web-1, Web-2 and zipcodeservice in Service-1, Service-2.

Execute those four commands in sequence:

kubectl label nodes ip-172-31-33-22 servicetype=zipcodeservice

kubectl label nodes ip-172-31-35-232 servicetype=zipcodeservice

kubectl label nodes ip-172-31-16-16 servicetype=webservice

kubectl label nodes ip-172-31-42-220 servicetype=webservice

To verify that labels have been assigned execute

kubectl get nodes --show-labels

In the response, you should see the newly assigned labels under LABELS column as shown below.

NAME               STATUS    ROLES     AGE       VERSION   LABELS
ip-172-31-16-16    Ready     <none>    23h       v1.11.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=ip-172-31-16-16,servicetype=webservice
ip-172-31-22-14    Ready     master    1d        v1.11.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=ip-172-31-22-14,node-role.kubernetes.io/master=
ip-172-31-33-22    Ready     <none>    23h       v1.11.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=ip-172-31-33-22,servicetype=zipcodeservice
ip-172-31-35-232   Ready     <none>    23h       v1.11.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=ip-172-31-35-232,servicetype=zipcodeservice
ip-172-31-42-220   Ready     <none>    23h       v1.11.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=ip-172-31-42-220,servicetype=webservice

Deploying the Service Layer

Create a file named zip-service-deployment.yaml with the following content.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: zip-service-deployment
  namespace: default
  labels:
    app: zip-service
spec:
  replicas: 2
  selector:
    matchLabels:
      app: zip-service
  template:
    metadata:
      labels:
        app: zip-service
    spec:
      containers:
      - name: zip-service
        image: konuratdocker/spark-examples:zipcode-service
        command: ["java"]
        args: ["-jar","/app.jar","8085"]
        ports:
        - containerPort: 8085
      nodeSelector:
        servicetype: zipcodeservice
      imagePullSecrets:
      - name: regcred 
      dnsConfig:
        nameservers:
          - 8.8.8.8

Highlights:

• Name of the service layer application is zip-service, value of app parameter.
• The "konuratdocker/spark-examples:zipcode-service", value of the image parameter, is constructed from the values of repository, tag , and imageTag elements in pom.xml for the service layer.
• The value of name parameter under imagePullSecrets is regcred , the secret key we had created to access the private repository in Docker Hub.
• In Kubernetes environment, we would like to run the service application at port 8085 and therefore we override the port number 2223 in Dockerfile. (Of course, we could have kept the same port — we just wanted to illustrate how to override the value previously set in Dockerfile.)
• The value 8.8.8.8 is IP of the DNS server for our application to get the IP address associated with zipcodeapi.com.
• Observe the replicas: 2 and servicetype: zipcodeservice for nodeSelector. We create two replicas of the service and those will be deployed in the two nodes with label zipcodeservice.

Execute

kubectl create -f ./zip-service-deployment.yaml

You should see a response similar to this.

deployment.apps/zip-service-deployment created

After waiting a few minutes, execute

kubectl get pods -o wide

In response, you should see something like:

NAME                                      READY     STATUS    RESTARTS   AGE       IP           NODE
zip-service-deployment-6546457848-crwdp   1/1       Running   0          20s       10.244.3.2   ip-172-31-33-22
zip-service-deployment-6546457848-tx7tv   1/1       Running   0          20s       10.244.4.2   ip-172-31-35-232

In each of ip-172-31-33-22 (Service-1) and ip-172-31-35-232 (Service-2), a pod has been created in which an instance of the service layer application is running inside a container.

Let us inspect the application log. Pass name of pod to kubectl logs. For example,

kubectl logs zip-service-deployment-6546457848-crwdp

would display:

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.0.3.RELEASE)
1: INFO  ZipcodeServer - No active profile set, falling back to default profiles: default
1: INFO  Http11NioProtocol - Initializing ProtocolHandler ["http-nio-8085"]
1: INFO  StandardService - Starting service [Tomcat]
1: INFO  StandardEngine - Starting Servlet Engine: Apache Tomcat/8.5.31
1: INFO  AprLifecycleListener - The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64:/usr/lib/jvm/java-1.8-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib]
1: INFO  [/] - Initializing Spring embedded WebApplicationContext
1: INFO  ServiceController - ServiceController initiated
1: INFO  ServiceConfiguration$$EnhancerBySpringCGLIB$$f5e038b8 - ServiceConfiguration initialized
1: INFO  Http11NioProtocol - Starting ProtocolHandler ["http-nio-8085"]
1: INFO  NioSelectorPool - Using a shared selector for servlet write/read
1: INFO  ZipcodeServer - Started ZipcodeServer in 8.225 seconds (JVM running for 9.674)
1: INFO  [/] - Initializing Spring FrameworkServlet 'dispatcherServlet'

Observe the line printed by Tomcat's HTTP NIO connector, ["http-nio-8085"], the port number coincides with the port specified in zip-service-deployment.yaml.)

The next step is to create a service to access the newly created deployment. Create file named zipcode-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: zipcode-service
  namespace: default
spec:
  ports:
  - port: 8085
    targetPort: 8085
    name: http
    protocol: TCP
  selector:
    app: zip-service
  type: ClusterIP

Highlights:

• Name of the service is zipcode-service, in default namespace.
• The service port number is 8085, which coincides with the port in zip-service-deployment.yaml file.
• Value of app under selector is zip-service, name of the service layer application in zip-service-deployment.yaml.
• The type of service is ClusterIP (https://kubernetes.io/docs/concepts/services-networking/service/), which will expose the service via a cluster-internal IP. The service IP address will be accessible only from inside the cluster. The IP could be resolved by the nodes inside the cluster using the kube-dns service we had discussed earlier.

Now execute

kubectl create -f ./zipcode-service.yaml

You should see a response like this:

service/zipcode-service created

The domain name of the newly created service will be zipcode-service.default.svc.cluster.local where zipcode-service.default is constructed from metadata section of zipcode-service.yaml. If you execute

kubectl get services -o wide

response should look like below:

NAME              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE       SELECTOR
kubernetes        ClusterIP   10.96.0.1      <none>        443/TCP    23h       <none>
zipcode-service   ClusterIP   10.97.71.233   <none>        8085/TCP   30s       app=zip-service

The newly created service has been assigned a ClusterIP of 10.97.71.233. In addition, if we execute

dig zipcode-service.default.svc.cluster.local @10.96.0.10

the answer section in response will include

...
zipcode-service.default.svc.cluster.local. 5 INA 10.97.71.233
...

where IP corresponds to the ClusterIP and zipcode-service.default.svc.cluster.local is domain name of the service. Note that 10.96.0.10 is IP of the kube-dns service we had previously talked about.

We can test the service layer as follows. In master node, edit /etc/resolv.conf temporarily to insert 10.96.0.10 as the first line. Hence, the file should read:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
10.96.0.10
...

Then, execute

curl http://zipcode-service.default.svc.cluster.local:8085/zipcodeservice/info/33301

The response will be as follows:

{"zip_code":"33301","lat":26.121317,"lng":-80.128146,"city":"Fort Lauderdale","state":"FL","timezone":{"timezone_identifier":"America\/New_York","timezone_abbr":"EDT","utc_offset_sec":-14400,"is_dst":"T"},"acceptable_city_names":[{"city":"Ft Lauderdale","state":"FL"}]}

Similarly,

curl http://zipcode-service.default.svc.cluster.local:8085/zipcodeservice/nearby/33301/5

should display:

{"zip_codes":[{"zip_code":"33004","distance":4.428,"city":"Dania","state":"FL"},{"zip_code":"33315","distance":2.821,"city":"Fort Lauderdale","state":"FL"},{"zip_code":"33312","distance":4.052,"city":"Fort Lauderdale","state":"FL"},...]}

High Availability

The service named zipcode-service is an access point to the service layer running in two distinct nodes. In EC2 console, stop one of the instances corresponding to those nodes, e.g. Service-1, as shown below.

With only Service-2 node running, if you execute

curl http://zipcode-service.default.svc.cluster.local:8085/zipcodeservice/info/33301

and

curl http://zipcode-service.default.svc.cluster.local:8085/zipcodeservice/nearby/33301/5

you would still get the same responses from the service as before. Then, start instance Service-1 and stop Service-2. Execute curl, you will continue getting the same response.

Deploying the Web Layer

Create a file named zip-web-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: zip-web-deployment
  namespace: default
  labels:
    app: zip-web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: zip-web
  template:
    metadata:
      labels:
        app: zip-web
    spec:
      containers:
      - name: zip-web
        image: konuratdocker/spark-examples:web-service
        command: ["java"]
        args: ["-jar","/app.jar","zipcode-service.default.svc.cluster.local:8085","3334"]
        ports:
        - containerPort: 3334
      nodeSelector:
        servicetype: webservice
      imagePullSecrets:
      - name: regcred 
      dnsConfig:
        nameservers:
          - 10.96.0.10

Highlights:

• Name of the web layer application is zip-web, value of app parameter.
• The "konuratdocker/spark-examples:web-service", the value of image parameter, is constructed from the values of repository, tag and imageTag elements in pom.xml for the web layer.
• The value of name parameter under imagePullSecrets is regcred, the secret key we had created to access the private repository in Docker Hub.
• In Kubernetes environment, we would like to run the service application at port 3334 and therefore we override the port number in Dockerfile, 3333. The service layer will be accessed via domain name zipcode-service.default.svc.cluster.local at port 8085.
• The value 10.96.0.10 is IP of the DNS server for our application to get the IP address associated with the service. That IP corresponds to kube-dns service.
• Observe the replicas: 2 and servicetype: webservice for nodeSelector. We create two replicas of the service and those will be deployed in the two nodes with label webservice.

Now executing

kubectl create -f ./zip-web-deployment.yaml

would yield

deployment.apps/zip-web-deployment created

To verify that the web layer has been deployed, we execute

kubectl get pods -o wide

The response should read like this:

NAME                                      READY     STATUS    RESTARTS   AGE       IP           NODE
...
zip-web-deployment-85cd564498-9c86s       1/1       Running   0          2m        10.244.2.2   ip-172-31-42-220
zip-web-deployment-85cd564498-wzfqx       1/1       Running   0          2m        10.244.1.2   ip-172-31-16-16

In each of ip-172-31-16-16 (Web-1) and ip-172-31-42-220 (Web-2), a pod has been created in which an instance of the web layer application is running inside a container. Let us inspect the log file generated when the application inside one of those pods is started. For example, if we execute

kubectl logs zip-web-deployment-85cd564498-9c86s

the following will be displayed:

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.0.1.RELEASE)
1: INFO  WebServer - No active profile set, falling back to default profiles: default
1: INFO  Http11NioProtocol - Initializing ProtocolHandler ["http-nio-3334"]
1: INFO  StandardService - Starting service [Tomcat]
1: INFO  StandardEngine - Starting Servlet Engine: Apache Tomcat/8.5.29
1: INFO  AprLifecycleListener - The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64:/usr/lib/jvm/java-1.8-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib]
1: INFO  [/] - Initializing Spring embedded WebApplicationContext
1: INFO  WebController - WebController initiated
1: INFO  Http11NioProtocol - Starting ProtocolHandler ["http-nio-3334"]
1: INFO  NioSelectorPool - Using a shared selector for servlet write/read
1: INFO  WebServer - Started WebServer in 8.574 seconds (JVM running for 10.092)
1: INFO  [/] - Initializing Spring FrameworkServlet 'dispatcherServlet'

(Observe the line printed by Tomcat's HTTP NIO connector, ["http-nio-3334"], the port number specified in zip-web-deployment.yaml.)

Next, create a file named zip-web-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: zip-web-service
  namespace: default
spec:
  ports:
  - port: 3334
    targetPort: 3334
    nodePort: 30000
    name: http
    protocol: TCP
  selector:
    app: zip-web
  type: NodePort

Highlights:

• Name of the service is zip-web-service, in default namespace.
• Port number 3334 coincides with the port in zip-web-deployment.yaml.
• Type of service is NodePort (https://kubernetes.io/docs/concepts/services-networking/service/), which will allow a node outside the cluster to access web service at port 30000 (nodePort) using individual IP addresses of the nodes where the service is deployed. The advantage of NodePort is that it allows access from outside the cluster. (With service type being NodePort, the service still gets a ClusterIP and a request to node-specific IP address to access the service from outside the cluster is routed to the particular ClusterIP.)
• Finally, value of app under selector is zip-web, name of deployment in zip-web-deployment.yaml. Execute

kubectl create -f ./zip-web-service.yaml

You should see a response similar to below.

service/zip-web-service created

At this point, we have created the two services needed. Execute "kubectl get services". You should see:

NAME              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
...
zip-web-service   NodePort    10.110.145.20   <none>        3334:30000/TCP   14s
zipcode-service   ClusterIP   10.97.71.233    <none>        8085/TCP         25m

Let us test the web layer with internal IP addresses of the individual nodes for Web-1 and Web-2. Execute

kubectl get nodes -o wide

The response should look like this:

NAME               STATUS    ROLES     AGE       VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
ip-172-31-16-16    Ready     <none>    1h        v1.11.1   172.31.16.16    <none>        Ubuntu 16.04.4 LTS   4.4.0-1061-aws   docker://17.3.2
ip-172-31-22-14    Ready     master    1d        v1.11.1   172.31.22.14    <none>        Ubuntu 16.04.5 LTS   4.4.0-1061-aws   docker://17.3.2
ip-172-31-33-22    Ready     <none>    1h        v1.11.1   172.31.33.22    <none>        Ubuntu 16.04.4 LTS   4.4.0-1061-aws   docker://17.3.2
ip-172-31-35-232   Ready     <none>    1h        v1.11.1   172.31.35.232   <none>        Ubuntu 16.04.4 LTS   4.4.0-1061-aws   docker://17.3.2
ip-172-31-42-220   Ready     <none>    1h        v1.11.1   172.31.42.220   <none>        Ubuntu 16.04.4 LTS   4.4.0-1061-aws   docker://17.3.2

Observe that the internal IPs for ip-172-31-16-16 (Web-1) and ip-172-31-42-220 (Web-2) are 172.31.16.16 and 172.31.42.220, respectively.

Execute

curl http://172.31.16.16:30000/zip/getZipcodeInfo/33301

In response, you should see:

<html><body><p>Zipcode Information:<p>zip: 33301, latitude: 26.121317, longitude: -80.128146, city: Fort Lauderdale, state: FL<p>Timezone: America/New_York (EDT)<p>Acceptable City Names:<p>Ft Lauderdale, FL</body></html>

Similarly,

curl http://172.31.16.16:30000/zip/getNearbyZipcodes/33301/5

should yield:

<html><body><p>Zip codes:<br><p>zip_code=33004, distance=4.428 miles, city=Dania, state=FL<p>zip_code=33315, distance=2.821 miles, city=Fort Lauderdale, state=FL...</body></html>

If you replace '172.31.16.16' with '172.31.42.220', you would get the same responses.

Connect to the outside node, i.e. the test node outside the cluster. Execute

curl http://172.31.16.16:30000/zip/getZipcodeInfo/33301

and

curl http://172.31.42.220:30000/zip/getZipcodeInfo/33301

Those should provide the same responses as we obtained in the master node. The same applies for

curl http://172.31.16.16:30000/zip/getNearbyZipcodes/33301/5

and

curl http://172.31.42.220:30000/getNearbyZipcodes/33301/5

Conclusions

In this tutorial, we discussed deploying a Spring Boot application into a Kubernetes cluster in Amazon EC2 environment. The Spring application consisted of weakly coupled web and service layers each running in their own Docker container. We described how to locally test web and service layers and then how to push them to a private repository in Docker Hub using com.spotify Maven plug-in as part of Maven build. (The files related to the application can be obtained from Git Hub.)

We provided detailed steps to create a single master Kubernetes cluster in Amazon EC2 environment using kubeadm. Then, we discussed how to deploy the individual web and service layers of the application into the cluster. We illustrated how to override during deployment the ENTRYPOINT instruction originally set in the Dockerfile, e.g. to change the listening port.

We deployed the web and service layers into separate nodes. For that purpose, we first assigned labels to individual nodes to give them either web or service layer designation. Of course, it is also possible to deploy each of the web and service layers into every node. However, typically service layer applications will need to access additional sources, e.g. databases and it may be wise to separately deploy web layer, directly accessed by end users, to avoid potential security risks. That is why we chose the segregated approach described in the tutorial.

Because service layer is accessed only within the cluster (by the web layer), its service type was set as ClusterIP, which ensures that a distinct IP address is assigned to the service, resolvable inside the cluster via a domain name. A request to the service is routed to one of the back-end nodes where service is running to ensure a level of high availability. The NodePort service type used for the web layer did not have the same type built-in high availability because the consumers have to access a web layer node using its node-specific IP address. In a real production deployment this limitation could be addressed by employing a load balancer in front of the web layer.

Here, a minimum set of required ports are listed while installing a Kubernetes cluster via kubeadm. Depending on the particular network plug-in and the hardware/network infrastructure additional ports may need to be opened. For simplicity, we allowed all TCP and UDP ports in every node. However, in a real application a more restrictive policy should be adhered to.

Flannel, as we used in this tutorial, does not provide network policy. In a real cluster environment you will probably need an additional network policy plug-in, such as Calico. Also see the discussion here.

The single master Kubernetes cluster discussed in this article has the obvious drawback of having only one master, i.e. a single point of failure. For creating highly available clusters with multiple masters see here.