Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Deploying Talend ESB Using Azure Container Service and Kubernetes

DZone's Guide to

Deploying Talend ESB Using Azure Container Service and Kubernetes

This tutorial will help you learn how to create an Azure Container Service and use it to deploy the Talend Open Source ESB application.

· Integration Zone ·
Free Resource

SnapLogic is the leading self-service enterprise-grade integration platform. Download the 2018 GartnerMagic Quadrant for Enterprise iPaaS or play around on the platform, risk free, for 30 days.

You’ve probably heard that Kubernetes on the Microsoft Azure Container Service (ACS) is now generally available. This is great news for the developer community. With a production-ready ACS for K8S, you’ll now be able to use it in real applications. In this tutorial, we’ll show you how to create an ACS and deploy the Talend Open Source enterprise service bus (ESB) application.

Before You Get Started

You’ll first need to fulfill a few prerequisites. First, create a client_id and client_secret_id. Refer to Microsoft’s instructions on creating an Active Directory application and service principal for information on creating your ids. You will also need RSA public and private keys, and you will need to add the private part of the key to the ssh client. Next, obtain azure CLI 2.0 and kubectl. Finally, you must have docker on your workstation.

Deploying

First, log into http://portal.azure.com, and click + in the upper right. Enter “Azure Container Service,” click “Azure Container Service,” and click “Create.”

Select “Orchestrator — Kubernetes.” Enter the Resource group and Location (for example: group1, East US), and click “ok.”

Complete the required fields, using your own SSH public key. Enter client_id and client_secret_id as the values for Service principal client ID and Service principal client secret. Enter 1 as the value for the Master count. (Note that one master is enough for demonstration purposes, but is not enough for production use.) Click “ok”.

Set the Agent count to 2, select the size of VM, and select the OS (2, Standard DS2, Linux). Review the summary, and then click “ok.” Your deployment will now start.

Validating the Deployment

To validate the deployment, go to the Dashboard and select “Resource groups” from the left menu. Then, click on “group1.”

Ensure that you have a couple of different types of resources, as shown in the following example screen.

Image title

Connecting to the Management Console

Open the command line client with SSH support, type az login, and follow the instructions.

Type az account list and find your subscription id (this is the id you used to create the “rgroup1” resource group).

Enter the following information, replacing the content in {} with your unique information:

az account set — subscription={your-subscription-id}

az acs kubernetes get-credentials — resource-group={your-resource-group} — name={name-of-container-service} — ssh-key-file={path-to-your-private-key}

(You can find the name of the container service under the resource group in the portal. You must specify the password of the key.)

You should now have a “config” file in the /home/user/.kube  folder.

Ensure you have a connection to the cluster by typing kubectl get nodes. You should see a display like the following:

NAME STATUS AGE 
  k8s-agent-e2fcdf77–0 Ready 49m 
  k8s-agent-e2fcdf77–1 Ready 50m 
  k8s-master-e2fcdf77–0 Ready,SchedulingDisabled 50m 

Next, run kubectl proxy, and return to the dashboard. You can now start using your cluster.

Building the Docker Image

Before you can begin building your docker image, you’ll need to download the Talend ESB, and you will also need an account on docker hub. Next, you must modify the build docker image file, Dockerfile, to ensure you have the correct paths.

In the following example, Runtime_ESBSE_light.zip is an archive which contains Talend ESB.

FROM openjdk:7-jdk 
LABEL maintainer aocheretnoy@gmail.com

ENV 
TALEND_DOWNLOAD_URL=https://sourceforge.net/projects/talendesb/files 
/Talend%20Open%20Studio%20for%20ESB/5.6.2/TOS_ESB-20150508_1414-
V5.6.2.zip/download
ENV TALEND_LOCAL_STORAGE=talend-release
ENV JAVA_HOME /usr/lib/jvm/java-7-openjdk-amd64

ENV TALEND_DIR=/opt/talend

# Update ubuntu and install the required packages
RUN apt-get update && apt-get install -y make ant

# Download Talend
#
# Talend official packages are hosted on Sourceforge and it is slow
to download in general
#RUN wget — no-check-certificate -O $TALEND_DIR.zip
$TALEND_DOWNLOAD_URL && unzip $TALEND_DIR.zip -d $TALEND_DIR

# Copy Talend from local storage
#
# This is here until a fast server is available to download Talend
packages
COPY $TALEND_LOCAL_STORAGE/Runtime_ESBSE_light.zip /opt
RUN mkdir $TALEND_DIR && unzip /opt/Runtime_ESBSE_light.zip -d
/opt/talend; \
   rm /opt/Runtime_ESBSE_light.zip

VOLUME [“/opt/talend/Runtime_ESBSE_light/container/deploy”]

EXPOSE 8040 9001 1099 44444 8101 8000 8001 8888 61616 2181 1527 8082
8088 8090

CMD [“/opt/talend/Runtime_ESBSE_light/container/bin/trun”]

Enter “docker build -t tesb .”, where tesb is the name of a new image. Then, enter “docker tag tesb arkadiy/tesb:latest”, where arkadiy is the name of a public docker repository on docker hub. Replace arkadiy with the name of your own public docker repository.

To complete the build, enter “docker login” and then enter “docker push arkadiy/tesb:latest.”

Deploying an Application

To deploy an application, take a moment first to ensure you’ve completed the steps in “Deploying,” “Validating the Deployment,” “Connecting to the Management Console,” and “Building the Docker Image.”

Then, go to the dashboard and select “Deployment” in the left menu. Click “+Create” in the upper right, and select the “Upload a YAML or JSON file” option.

Image title

Modify tesb-deployment-acs.yaml to use your image name. Replace “image: arkadiy/tesb:0.2” with your image name from “Building the Docker Image.”

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
 name: tesb
spec:
 replicas: 2
 template:
  metadata:
   labels:
    name: tesb
  spec:
   containers:
   - name: tesb
     image: arkadiy/tesb:0.2
     ports:
     - containerPort: 8040
     - containerPort: 9001
     - containerPort: 1099
     - containerPort: 44444
     - containerPort: 8000
     - containerPort: 8001
     - containerPort: 8101
     - containerPort: 8888
     - containerPort: 61616
     - containerPort: 2181
     - containerPort: 1527
     - containerPort: 8082
     - containerPort: 8088
     - containerPort: 8090
     volumeMounts:
     - name: deploy
       mountPath: /opt/talend/Runtime_ESBSE_light/container/deploy
     volumes:
     - name: deploy
       hostPath:
        path: /home/azureuser/deploy

Select tesb-deployment-acs.yaml, and click “Upload.” Next, go to “Pods” in the left menu. Wait until “tesb-….” is ready. This should take approximately 5 minutes because the image is 2GB.

Image title

Go to “Services,” click “create,” and upload the two services.

First service:  tesb-service-acs.yaml

apiVersion: v1
kind: Service
metadata:
 name: tesb
spec:
 type: LoadBalancer
 ports:
   - name: api
     port: 8088
     targetPort: 8088
   - name: apiint
     port: 8090
     targetPort: 8090
 selector:
  name: tesb

Second service:  tesb-service-acs-nodeport.yaml 

apiVersion: v1
kind: Service
metadata:
 name: tesb-int
 labels:
   name: tesb-int
spec:
 type: NodePort
 ports:
  - name: http
    port: 8040
    targetPort: 8040
  - name: https
    port: 9001
    targetPort: 9001
  - name: jmx
    port: 1099
    targetPort: 1099
  - name: jmxrmi
    port: 44444
    targetPort: 44444
  - name: jobserver
    port: 8000
    targetPort: 8000
  - name: jobserverfile
    port: 8001
    targetPort: 8001
  - name: karafssh
    port: 8101
    targetPort: 8101
  - name: jobservermonitor
    port: 8888
    targetPort: 8888
  - name: activemq
    port: 61616
    targetPort: 61616
  - name: zookeeper
    port: 2181
    targetPort: 2181
  - name: esbsam
    port: 1527
    targetPort: 1527
  - name: esbinstaller
    port: 8082
    targetPort: 8082
  selector:
    name: tesb

Go to “Services” and wait until service tesb displays an IP address. This will take a few minutes as the load balancer and rules are created.

Image title

Deploying the Integration Scenario

To deploy the integration scenario, ensure that you have completed the steps in the previous sections and that you have a Talend OpenESB compatible application (download here).

TalendESB is based on the Apache Karaf OSGI container, so you only need to copy archives to “deploy” deploy. Refer to tesb-deployment-acs.yaml for the path. In this example, we use /home/azureuser/deploy as the mount path for the container.

To test your integration scenario, use Postman to ensure the application works.

Image title

That’s it! You’ve created an ACS and deployed the Talend Open Source enterprise service bus (ESB) application.

Share your thoughts and questions in the comments section below.

Download A Buyer's Guide to Application and Data Integration, your one-stop-shop for research, checklists, and explanations for an application and data integration solution.

Topics:
kubernetes ,azure ,docker ,talend esb ,integration ,containers ,acs ,deployment

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}