Talend Integration Cloud 101: SDLC and Code Promotion Pipeline

Talend Integration Cloud puts powerful graphical tools, prebuilt integration templates, and 900+ connectors and components at your fingertips to connect databases, big data sources, on-premises, and cloud applications. This enables the faster design of cloud-to-cloud and hybrid integration workflows in Talend Studio and the ability to publish them to a fully managed cloud.

Once a connection is established, any update in your scenario from Talend Studio can be pushed to the Cloud right away. In this blog, I would like to talk more about Software Development Life Cycle development and the process of moving code from one environment to another in Talend Integration Cloud.

SDLC With Talend Integration Cloud

Talend Integration Cloud enables you to manage your Software Development Life Cycle within a single Cloud application. In a typical project lifecycle with your Talend Integration Cloud environment, the software development phases would typically work like this:

1. The development environment is a dedicated environment where the flows are designed and tested. Development starts in Talend Studio where developers build actions, connections, and plan for data integration job execution. The actions/connections/flows are then stored in Git for source code management. Git can be configured using the Talend Management Console while creating projects. To download and install specific third-party Java libraries or database drivers that are needed by Talend Studio, you need to connect to the User Libraries repository (integrated within the Nexus artifact repository) and its embedded repository Talend-custom-libs-release.

2. Once the job design is completed in Talend Studio, the developer then pushes the code to cloud development via the 'Publish to Cloud' option. As an alternative, we could automate this using command line also.

3. Developers publish these Jobs into the Cloud environment dedicated to QA through a promotion pipeline. QA allows developers to validate the processes created in the previous development environment without shutting down development work. Once tested, the QA lets the DevOps team know that the integration flow is ready to be deployed to production.

4. The DevOps team then triggers the flows to the production environment. The production environment is a live environment where the product or feature is deployed and used in production.

Note: Talend Studio still holds your job and when you publish it to cloud, the job binaries (and not the original job) are pushed to the cloud.

Talend Integration Cloud empowers you to create separate environments, each with its own set of artifacts to support the objectives of each phase in the SDLC process. One very important point to note is that Talend Integration Cloud needs a minimum of one instance of an engine (this can either be in the cloud or remote) per environment. So here in this SDLC, as we have 3 environments, the recommendation is to have minimum 3 cloud engine and/or remote engine.

Code Promotion

A "promotion pipeline" is how you promote your Cloud artifacts from a source to a target environment. In practice, this can be from Dev to Test, Test to QA, QA to UAT, UAT to Production.

Talend Integration Cloud offers two ways to promote code:

1. Using Talend Studio

One way to promote jobs from Talend Studio to the cloud is via the 'Publish to Cloud' option available when you right-click a job as shown in the screenshot below. From the Repository tree view, simply right-click the Job and select Publish to Cloud.

From the Workspace list in the open dialogue box, select the workspace and click Finish. The job is then published to the cloud.

One important thing to note before we move on: in Talend Studio, you will need to manually right click and publish the jobs to the target environment. However, using the promotion pipeline in Talend Integration Cloud offers you flexible options to move the code between environments. Let's talk about that method next.

2. Using Talend Integration Cloud

Another way to promote jobs from one environment to another is via the 'Promotion Pipeline' option available in Talend Integration Cloud. The promotion pipeline can be accessed via Administration->Promotion Pipelines as shown below.

As a prerequisite, you'll need to have your pipelines created before you start the migration. There are then two options for your promotion pipeline: the first is the promotion of the full environment. With this option, all actions, flows, connections, execution plans - the whole workspace - is migrated between environments. This option can be accessed using Operations-> promotions and the pipelines created as a prerequisite.

Select the pipelines created and then select if you want to override the resources or skip the resources. Talend Integration Cloud gives a very detailed promotion analysis report that contains important information about cloud artifact status, which might have one of the following:

• Created- the Artifact does exist in the Target Environment.
• Reused- the Artifact does exist in the Target Environment and in the same version.
• Replaced- the Artifact does exist in the Target Environment and will be replaced by the Source Environment version.
• Failed- the Artifact cannot be promoted because of a conflict.

Dependencies between artifacts are highlighted. If you agree with the promotion analysis, click Promote. Once the code promotion is completed, TIC displays the promotion status. This status is also saved as history and can be accessed later as well.

The second method is the promotion of selected objects. This is done using the Advance Promotion option and allows a user to choose the flows, actions, and executions.

Using this option, you could promote either Full workspace, Action, flow, or Execution Plan. The rest of the process is similar to using full promotion pipeline, you would be prompted for the options to overwrite or skip resources and to confirm the promotion analysis report.

Conflicts During Promotion

There can be conflicts when promoting artifacts. Conflict details are listed in the promotion analysis report. Any of the following situations prevent you from promoting artifacts to the target environment:

• Several workspaces share the same name in the Source environment.
• Several workspaces from the target environment and one from the Source environment, in the same workspace, share the same name.
• Several Flows in the Source environment have the same name and path.
• Several Flows in the Target environment and one from the Source environment, in the same workspace, share the same name.
• A Flow has Resources or Connections in Personal workspaces of the Source environment.