Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Hybrid CI/CD: Hosting Dockerized Jenkins Worker Nodes On Alibaba Cloud

DZone 's Guide to

Hybrid CI/CD: Hosting Dockerized Jenkins Worker Nodes On Alibaba Cloud

This article demonstrates implementing a Dockerized node pool with Jenkins and Alibaba Cloud.

· Cloud Zone ·
Free Resource

Image title


This article shares how to implement a distributed and Dockerized Jenkins Node Pool on Alibaba Cloud which enables a dynamic and parallel CI/CD pipeline provisioning to accomplish best resource utilization, on-demand scalability and cost-effectiveness.

What Is Hybrid CI/CD

Hybrid CI/CD is a distributed platform that uses a mix of on-premises and public cloud elastic compute resources for CI/CD purpose. Hybrid CI/CD basically controls the workloads between private and public clouds to balance needs and costs.

At a high level, the architecture is shown in the below diagram.


The Benefits Of Hybrid CI/CD

Scalability and Flexibility

On-premises does offer a certain level of scalability, which is still based on each data center and highly depends on their budget. Frequently, when one system consumes resources than expected, all other systems are affected, so setting the boundary for budgeting becomes tricky. Public cloud services will offer greater scalability with larger cloud infrastructure. Considering CI/CD jobs are usually isolated, short-lived and generally stateless, by moving CI/CD jobs to the public cloud, it allows an organization to reserve scalability for the core services.

Cost-Effectiveness

Public clouds are more likely to offer more economic cost-effectiveness as the cost of centralized management is shared by every user.

Security

Compared to a pure public cloud solution, Hybrid CI/CD reserves the sensitiveness for security demand as you can always restrict data to be accessible only to applicable CI/CD jobs.

Using Docker Host As a Build Worker

Resource Utilization

Docker will dynamically assign resources so you can have multiple Jenkins worker nodes running in parallel. It will tremendously increase the throughput of the CI/CD pipeline.

Isolation

Docker ensures your applications and resources are isolated and segregated. You can simply destroy the target container without worrying about if Jenkins worker nodes really clean up thoroughly.

Environment Standardization

You can simply build a CI/CD environment inherited from the standard images.

Jenkins Worker Container Implementation

There are two types of communication protocols from Jenkins Master to Jenkins Worker being used. They are Java Web Start (a.k.a. JNLP) and SSH. Although SSH is encrypted under Blocking I/O, which limits the scalability, it is still the most common method because it is less dependent on the load balancer. So in this article, we will only discuss SSH.

Jenkins Worker entrypoint.sh

    #!/bin/bash -ex

    write_key() {
        mkdir -p "${JENKINS_AGENT_HOME}/.ssh"
        echo "$1" > "${JENKINS_AGENT_HOME}/.ssh/authorized_keys"
        chown -Rf jenkins:jenkins "${JENKINS_AGENT_HOME}/.ssh"
        chmod 0700 -R "${JENKINS_AGENT_HOME}/.ssh"
    }

    if [[ $JENKINS_SLAVE_SSH_PUBKEY == ssh-* ]]; then
    write_key "${JENKINS_SLAVE_SSH_PUBKEY}"
    fi
    if [[ $# -gt 0 ]]; then
    if [[ $1 == ssh-* ]]; then
        write_key "$1"
        shift 1
    else
        exec "$@"
    fi
    fi

    # ensure variables passed to docker container are also exposed to ssh sessions
    env | grep _ >> /etc/environment

    ssh-keygen -A
    exec /usr/sbin/sshd -D -e "${@}"

Jenkins Master Configuration

We assume you have already set up your Jenkins Master before. You may also refer to this trilogy of articles to build a new one on Alibaba Cloud:

Then go to Jenkins Manage Plugins, search for "Docker Plugin" to install it.

Next, go to "Configure System," and under "Docker," fill in the Docker URL with your ECS public IP address and the port of the Docker remote API, which by default is 2375. There is a "test connection" button so you can try and see if the connection is successful.

After that choose "Add Docker Template" and click "Docker Template," and fill in the details:

  • Docker Image: jenkinsci/ssh-slave

  • Remote Filing System Root: /home/jenkins
  • Labels: ssh-slave
  • Credentials: the public key you have injected for ssh-slave container

When everything is set, you can click the "Save" button.

Test Configuration

Go to "New item" and create a freestyle project named "ssh-slave-test".

Under Restrict, type the label name you have given in the slave template. In this case, it is "ssh-slave."

Under Build select the execute shell option and type an echo statement as shown below.

echo "hello {JOB_NAME}"

Save this job and click build now, you should get the following output.

hello ssh-slave-test

What Is Next

Once you finished everything above, you can integrate with your code repository and create real-world CI/CD jobs. You may also customize the Docker image based on your actual requirements.

Topics:
alibaba cloud ,jenkins ,ci/cd ,docker ,cloud ,hybrid cloud

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}