Over a million developers have joined DZone.
Refcard #218

Continuous Delivery With Jenkins Workflow

The Jenkins Plugin for Managing Complex DevOps Pipelines

Written by

Andy Pemberton Senior Director, Solution Architecture, CloudBees

Provides an introduction to the Jenkins Workflow plugin, a tool that extends the popular CD application to manage even the most complex software pipelines and help you continuously deliver more efficiently.

Free PDF
DOWNLOAD
Brought to you by CloudBees
Section 1

Introduction

Jenkins is an open-source automation tool with a powerful plugin architecture that helps development teams automate their software lifecycle. Jenkins is used to drive many industry-leading companies’ continuous integration and continuous delivery pipelines.

Jenkins Workflow is a first-class feature for managing complex, multi-step pipelines. This open-source plugin for Jenkins brings the power of the Jenkins domain and plugin ecosystem into a scriptable Domain-Specific Language (DSL). Best of all, like Jenkins core, Workflow is extensible by third-party developers, supporting custom extensions to the Workflow DSL and various options for plugin integration.

Image title

Workflow Stage View UI

This Refcard provides an overview and introduction to Jenkins Workflow, as well as a full Workflow syntax reference. It also provides a real-world delivery pipeline example, building on the more basic Workflow snippets from earlier examples.

Section 2

Installing Jenkins Workflow

It is assumed you have already installed Jenkins—either via the CloudBees Jenkins Platform or jenkins-ci.org. For Jenkins Workflow, Jenkins version 1.580+ is required; version 1.609.1+ is recommended. To install Jenkins Workflow:

  •  Open Jenkins in your web browser
  •  Navigate to Manage Jenkins > Manage Plugins
  •  Navigate to the Available tab, filter by Workflow
  •  Select the Workflow Aggregator and install
  • Restart Jenkins

This Refcard was written using Workflow version 1.10. Installing the Workflow Aggregator installs all necessary Workflow dependencies and a new job type called Workflow.

Section 3

Creating a Workflow

Now that you have Jenkins running and have installed the Workflow plugin, you are ready to create your first pipeline. Create a new workflow by selecting New Item from the Jenkins home page.

First, give your workflow a name (e.g., “hello-world- flow”). Workflows are simple Groovy scripts, so let’s add the obligatory Hello World. Add a workflow to the Workflow script text area: echo 'Hello world' Now save your workflow, ensuring the Use Groovy Sandbox option is checked (more details to follow on this setting).

echo 'Hello world'

Click Build Now to run your workflow.

Editing Your Workflow

Because workflows are simple text scripts, they are easy to edit. As you’ve seen, workflows can be edited directly in the Jenkins UI when configuring your workflow.

Image title

Using the Snippet Generator

To make editing your workflows easier, use the Snippet Generator. The Snippet Generator is dynamically populated with the latest Workflow steps. Depending on the plugins installed in your environment, you may see more available steps.

Image title

Loading External Workflow Scripts

Because workflows are text assets, they are ideal to store in a source control system. Workflows can be edited in your external IDE then loaded into Jenkins using the Workflow script from SCM option.

Building Your Workflow

Now that you’ve created a workflow, let’s continue to build on it. For a complex flow, you should leverage Jenkins’ job scheduling queue:

node{ sh 'uname' } 

The concept of a node should be familiar to Jenkins users: node is a special step that schedules the contained steps to run by adding them to Jenkins’ build queue. Even better, requesting a node leverages Jenkins’ distributed build system. Of course, to select the right kind of node for your build, the node element takes a label expression:

node('unix && 64-bit'){ echo 'Hello world' } 

The node step also creates a workspace: a directory specific to this job where you can check out sources, run commands, and do other work. Resource-intensive work in your pipeline should occur on a node. You can also use the ws step to explicitly ask for another workspace on the current slave, without grabbing a new executor slot. Inside its body all commands run in the second workspace.

Checking out Code

Usually, your workflows will retrieve source code from your source control server. Jenkins Workflow has a simple syntax for retrieving source code, leveraging the many existing SCM plugins for Jenkins.

checkout([$class: 'GitSCM', branches: [[name: '*/ master']], userRemoteConfigs: [[url: 'http://github.com/ cloudbees/todo-api.git']]])

If you happen to use a Git-based SCM like GitHub, there’s an even further simplified syntax:

git 'https://github.com/cloudbees/todo-api.git' 

Running Your Workflow

Because workflows are built as Jenkins jobs, they can be built like other jobs. You can use the Build Now feature to manually trigger your build on demand or set up triggers to execute your pipeline based on certain events.

Adding Stages And Steps

Stages are usually the topmost element of Workflow syntax. Stages allow you to group your build step into its component parts. By default, multiple builds of the same workflow can run concurrently. The stage element also allows you to control this concurrency:

stage 'build'
  node{ ... }
stage name: 'test', concurrency: 3
  node{ ... }
stage name: 'deploy', concurrency: 1
  node{ ... }

In this example, we have set a limit of three concurrent executions of the test stage and only one execution of the deploy stage. You will likely want to control concurrency to prevent collisions (for example, deployments).

Newer builds are always given priority when entering a throttled stage; older builds will simply exit early if they are preempted.

General Build Steps

Within your stages, you will add build steps. Just like with free-style Jenkins jobs, build steps make up the core logic of your pipeline. Jenkins Workflow supports any compatible Build Step and populates the Snippet Generator with all available Build Steps in your Jenkins environment.

step([$class: 'JavadocArchiver', javadocDir: 'target/ resources/javadoc', keepAll: false])
step([$class: 'Fingerprinter', targets: 'target/api. war']) 

Scripting

Jenkins Workflow supports executing shell (*nix) or batch scripts (Windows) just like free-style jobs:

sh 'sleep 10'
bat 'timeout /t 10'

Scripts can integrate with various other tools and frameworks in your environment—more to come on tools in the next section.

Section 4

Integrating Your Tools

For a real-life workflow, Jenkins needs to integrate with other tools, jobs, and the underlying environment.

Tools

Jenkins has a core capability to integrate with tools. Tools can be added and even automatically installed on your build nodes. From Workflow, you can simply use the tool DSL syntax:

def mvnHome = tool 'M3'
sh "${mvnHome}/bin/mvn -B verify"

In addition to returning the path where the tool is installed, the tool command ensures the named tool is installed on the current node.

Global Variables

The env global variable allows accessing environment variables available on your nodes:

echo env.PATH

Because the env variable is global, changing it directly is discouraged as it changes the environment globally, so the withEnv syntax is preferred (see example in the Workflow Full Syntax Reference below).

The currentBuild global variable can retrieve and update the following properties:

currentBuild.result
currentBuild.displayName
currentBuild.description

Existing Jobs

Existing jobs can be triggered from your workflow via the build command (e.g., build 'existing-freestyle-job'). You can also pass parameters to your external jobs as follows:

def job = build job: 'say-hello', parameters: [[$class: 'StringParameterValue', name: 'who', value: 'DZone Readers']]
Section 5

Controlling Flow

Because Jenkins Workflow is based on the Groovy language, there are many powerful flow control mechanisms familiar to developers and operations teams alike. In addition to standard Groovy flow control mechanisms like if statements, try/catch, and closures, there are several flow control elements specific to Workflow.

Handling Approvals

Jenkins Workflow supports approvals, manual or automated, through the input step:

input 'Are you sure?'

With the submitter parameter, the input step integrates Jenkins’ security system to restrict the allowed approvers.

The input step in Jenkins Workflow Stage View UI:

Image title

Timing

Timeouts allow workflow creators to set an amount of time to wait before aborting a build:

timeout(time: 30, unit: 'SECONDS') { ... } 

Parallel stages add a ton of horsepower to Workflow, allowing simultaneous execution of build steps on the current node or across multiple nodes, thus increasing build speed:

parallel 'quality scan': {
node {sh 'mvn sonar:sonar'}
}, 'integration test': {
node {sh 'mvn verify'}
}

Jenkins can also wait for a specific condition to be true:

waitUntil { ... } 

Handling Errors

Jenkins Workflow has several features for controlling flow by managing error conditions in your pipeline. Of course, because Workflow is based on Groovy, standard try/catch semantics apply:

try {
} catch (e) { 
}

Workflow creators can also create error conditions based on custom logic, if needed:

if(!sources) {
error 'No sources' 
}

Jenkins can also retry specific Workflow steps if there is variability in the steps for some reason:

retry(5) { ... }
Section 6

Script Security

As you've seen, Jenkins Workflow is quite powerful. Of course, with power comes risk, so Jenkins Workflow has a robust security and approval framework that integrates with Jenkins core security.

By default, when creating workflows as a regular user (that is, without the Overall/RunScripts permission), the Groovy Sandbox is enabled. When the Sandbox is enabled, Workflow creators will only be allowed to use pre-approved methods in their flow.

Image title

When adding pre-approved methods to a workflow, script changes do not require approval. When adding a new method (such as a Java API), users will see a RejectedAccessException and an administrator will be prompted to approve usage of the specific new API or method.

Deselecting the Use Groovy Sandbox option changes this behavior. When the Sandbox is disabled, workflow edits require administrator approval. Each change or update by a non-administrator user requires approval by an administrator. Users will see an UnnaprovedUsageException until their script is approved. Approving individual edits may not scale well, so the Groovy Sandbox is recommended for larger environments.

Section 7

Accessing Files

During your workflow development, you will very likely need to read and write files in your workspace.

Stashing Files

Stashing files between stages is a convenient way to keep files from your workspace in order to share them between different nodes:

stage 'build'
node{
git 'https://github.com/cloudbees/todo-api.git'
stash includes: 'pom.xml', name: 'pom'
}
stage name: 'test', concurrency: 3
node {
unstash 'pom'
sh 'cat pom.xml' 
}

Stash can be used to prevent cloning the same files from source control during different stages, while also ensuring the same exact files are used during compilation and tested in later pipeline stages.

Archiving

Like other Jenkins job types, workflows can archive their artifacts:

archive includes: '*.jar, excludes: '*-sources.jar'

Archives allow you to maintain binaries from your build in Jenkins for easy access later. Unlike stash, archive keeps artifacts around after a workflow execution is complete (whereas stash is temporary).

Beyond stashing and archiving files, the following Workflow elements also work with the file system (more details in full syntax reference):

pwd()
dir(''){}
writeFile file: 'target/results.txt', text: '' readFile 'target/results.txt'
fileExists 'target/results.txt'
Section 8

Scaling Your Workflow

As you build more of your DevOps pipelines with Jenkins Workflow, your needs will get more complex. The CloudBees Jenkins Platform helps scale Jenkins Workflow for more complex uses.

Checkpoints

One powerful aspect of the CloudBees extensions to Jenkins Workflow is the checkpoint syntax. Checkpoints allow capturing the workspace state so it can be reused as a starting point for subsequent runs:

checkpoint 'Functional Tests Complete' 

Checkpoints are ideal to use after a longer portion of your workflow has run (for example, a robust functional test suite).

Workflow Templates

The CloudBees Jenkins Platform has a robust template feature. CloudBees Jenkins Platform users can create template build steps, jobs, folders, and publishers. Since Workflows are a new job type, authors can create Workflow templates so that similar pipelines can simply leverage the same Workflow job template. More information on Templates is available on the CloudBees website:

https://www.cloudbees.com/products/cloudbees-jenkins- platform/enterprise-edition/features/templates-plugin

Section 9

Tying It Together: Example Pipeline

The following workflow is an example tying together several of the Workflow features we learned earlier. While not exhaustive, it provides a basic but complete pipeline that will help jump-start your workflow development:

stage 'build' 
node {
git 'https://github.com/cloudbees/todo-api.git' withEnv(["PATH+MAVEN=${tool 'm3'}/bin"]) {
sh "mvn -B –Dmaven.test.failure.ignore=true clean package"
}
stash excludes: 'target/', includes: '**', name: 'source'
}
stage 'test'
parallel 'integration': {
node {
unstash 'source' withEnv(["PATH+MAVEN=${tool 'm3'}/bin"]) {
sh "mvn clean verify" 
        }
}
}, 'quality': {
node {
unstash 'source' withEnv(["PATH+MAVEN=${tool 'm3'}/bin"]) {
sh "mvn sonar:sonar" 
        }
} 
}
stage 'approve'
timeout(time: 7, unit: 'DAYS') {
input message: 'Do you want to deploy?', submitter: 'ops'
}
stage name:'deploy', concurrency: 1
node {
unstash 'source' withEnv(["PATH+MAVEN=${tool 'm3'}/bin"]) {
sh "mvn cargo:deploy" 
    }
}
Section 10

Docker With Workflow

The Docker Workflow plugin exposes a docker global variable that provides a DSL for common Docker operations, only requiring a Docker client on the executor running the steps (use a label in your node step to target a Docker-enabled slave).

By default, the docker global variable connects to the local Docker daemon. You may use the docker.withServer step to connect to a remote Docker host. The image step provides a handle to a specific Docker image and allows executing several other image-related steps, including the image.inside step. The inside step will start up the specified container and run a block of steps in that container:

docker.image('maven:3.3.3-jdk8').inside('-v ~/.m2/repo:/ m2repo') {
sh 'mvn -Dmaven.repo.local=/m2repo clean package' 
}

When the steps are complete, the container will be stopped and removed. There are many more features of the Docker Workflow plugin; additional steps are outlined in the Workflow Full Syntax Reference.

Section 11

Extending Workflow

Like all Jenkins features, Workflow relies on Jenkins’ extensible architecture, allowing developers to extend Workflow’s features.

Plugin Compatibility

There are a large number of existing plugins for Jenkins. Many of these plugins integrate with Workflow as build steps, wrappers, and so on. Plugin maintainers must ensure their plugins are Workflow-compatible. The Jenkins community has documented the steps to ensure compatibility. More details on plugin development and Workflow compatibility are on the jenkins-ci.org Wiki:

https://wiki.jenkins-ci.org/display/JENKINS/Plugin+tutorial

Custom DSL

Beyond compatibility, plugin maintainers can also add specific statements to the Workflow DSL for their plugins’ behaviors. The Jenkins community has documented the steps to take to add plugin-specific statements to the Workflow DSL. Examples include the Credentials Binding Plugin, which contributes the withCredentials statement.

Section 12

Workflow Full Syntax Reference

Following is a full Jenkins Workflow syntax reference. Of course, as you add plugins—or as plugins are updated—new Workflow Script elements will become available in your environment. The Workflow snippet generator and UI will automatically add these and any associated help text so you know how to use them!

Basics

WORKFLOW SCRIPT

EXAMPLE(S)

stage

Stage

stage 'build'
stage concurrency: 3, name: 'test'a

node

Allocate a node

node('ubuntu') {

// some block }

ws

Allocate a workspace

ws('sub-workspace') {

// some block }

echo

Print a message

echo 'Hello Bees'

batch

Windows batch script

bat 'dir'

sh

Shell script

sh 'mvn -B verify'

checkout

General SCM

checkout([$class: 'GitSCM', branches: [[name: '*/master']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[url: 'http:// github.com/cloudbees/todo-api. git']]])

git

Git SCM

git 'http://github.com/cloudbees/ todo-api.git'

svn

Subversion SCM

svn 'svn://svn.cloudbees.com/repo/ trunk/todo-api'

step

General build step

step([$class: 'JUnitResultArchiver', testResults: 'target/test-reports/*.xml'])

step([$class: 'Mailer', notifyEveryUnstableBuild: true, recipients: 'info@cloudbees.com', sendToIndividuals: false])

wrap

wrap([$class:'Xvnc', useXauthority: true]){

sh 'make selenium-tests' }

tool

Install a tool

tool name: 'M3'

tool name: 'jgit', type: 'hudson. plugins.git.GitTool'

mail

Send an e-mail

mail, body: 'Uh oh.', charset: '', from: '', mimeType: '', replyTo: '', subject: 'Build Failed!', to: 'dev@ cloudbees.com'

Advanced

WORKFLOW SCRIPT

EXAMPLE(S)

build

Build an existing job

build job: 'hello-world'

build job: 'hello-world', parameters: [[$class: 'StringParameterValue', name: 'who', value: 'World']]

checkpoint

Capture the execution state so that it can be restarted later

checkpoint 'testing-complete'

withEnv

Set environment variables in a scope

withEnv(["PATH+MAVEN=${tool 'M3'}/ bin"]) {

sh 'mvn -B verify' }

load

Evaluate a Groovy source file into the workflow

load 'deploymentMethods.groovy'

File System


WORKFLOW SCRIPT

EXAMPLE(S)

dir

Change Directory

dir('src') {

// some block }

pwd

Get current Directory

def dir = pwd() echo dir

stash

Stash files for use later in the build

stash excludes: 'target/*-sources. jar', includes: 'target/*', name: 'source'

unstash

Restore files previously stashed

unstash 'source'

archive

Archive artifacts

archive includes:'*.jar', excludes:'*-sources.jar'

writeFile

Write file to Workspace

writeFile file: 'target/result. txt', text: 'Fail Whale'

readFile

Read file from the workspace

def file = readFile 'pom.xml'

fileExists

Verify if file exists in workspace

if(fileExists 'src/main/java/Main. java') {

// some block }


Flow Control

WORKFLOW SCRIPT

EXAMPLE(S)

sleep

Sleep

sleep 60
sleep time: 1000, unit: 'NANOSECONDS'

waitUntil

Wait for condition

waitUntil {

// some block }

retry

Retry body up to N times

retry(5) {

// some block }

input

Pause for manual or automated intervention

input 'Are you sure?'

input message: 'Are you sure?', ok: 'Deploy', submitter: 'qa-team'

parallel

Execute sub-flows in parallel

parallel “quality scan”: { // do something

}, “integration test”: {

// do something else },

failFast: true

timeout

Execute body without a timeout

timeout(time: 30, unit: 'SECONDS')

{
// some block

}

error

Stop build with an error

error 'No sources'

Docker

WORKFLOW SCRIPT

EXAMPLE(S)

image

Provides a handle to image

def image = docker. image('maven:3.3.3-jdk8')

image.inside

Runs steps inside image

image.inside('-v /repo:/repo') {

// some block }

image.pull

Pulls image

image.pull()

image.push

Push image to registry

image.push() image.push("latest")

image.run

Runs Docker image and returns container

def container = image.run("--name my-api -p 8080:8080")

container.stop()

image.withRun

Runs image and auto stops container

image.withRun {api -> testImg. inside("--link=${api.id}:api")

{
// some block

} }

image.tag

Records tag of image

image.tag("${tag}", false)

image.imageName()

Provides image name prefixed with registry info

sh "docker pull ${image. imageName()}"

container.id

ID of running container

sh "docker logs ${container.id}"

container.stop

Stops and removes container

container.stop()

build

Builds Docker image

docker.build("cb/ api:${tag}","target")

withServer

Runs block on given Docker server

docker.withServer('tcp://swarm. cloudbees.com:2376', 'swarm-certs')

{
// some block

}

withRegistry

Runs block using specified Docker registry

docker.withRegistry('https:// registry.cloudbees.com/', 'docker- registry-login') {

// some block }

withTool

Specifies name of Docker client to use

docker.withTool('toolName') {

// some block }


Publications

  • Featured
  • Latest
  • Popular
DOWNLOAD
Design Patterns
Learn design patterns quickly with Jason McDonald's outstanding tutorial on the original 23 Gang of Four design patterns, including class diagrams, explanations, usage info, and real world examples.
199.1k 538.6k
DOWNLOAD
Core Java
Gives you an overview of key aspects of the Java language and references on the core library, commonly used tools, and new Java 8 features.
122.4k 322.5k
DOWNLOAD
Getting Started with Ajax
Introduces Ajax, a group interrelated techniques used in client-side web development for creating asynchronous web applications.
100.5k 197k
DOWNLOAD
Getting Started with Git
This updated Refcard explains why so many developers are migrating to this exciting platform. Learn about creating a new Git repository, cloning existing projects, the remote workflow, and more to pave the way for limitless content version control.
109.2k 243.2k
DOWNLOAD
Spring Configuration
Catalogs the XML elements available as of Spring 2.5 and highlights those most commonly used: a handy resource for Spring context configuration.
101.8k 254.3k
DOWNLOAD
Core CSS: Part I
Covers Core principles of CSS that will expand and strengthen your professional ability to work with CSS. Part one of three.
88.8k 191.7k
DOWNLOAD
jQuery Selectors
Introduces jQuery Selectors, which allow you to select and manipulate HTML elements as a group or as a single element in jQuery.
92k 348.1k
DOWNLOAD
Foundations of RESTful Architecture
Introduces the REST architectural style, a worldview that can elicit desirable properties from the systems we deploy.
90.7k 133.6k
DOWNLOAD
The Ultimate Scrum Reference Card
Provides a concise overview of roles, meetings, rules, and artifacts within a Scrum organization. Updated for 2016.
84.7k 223k
DOWNLOAD
Core Java Concurrency
Helps Java developers working with multi-threaded programs understand the core concurrency concepts and how to apply them.
88.1k 179.7k
DOWNLOAD
Core CSS: Part II
Covers Core principles of CSS that will expand and strengthen your professional ability to work with CSS. Part two of three.
72.4k 138k
DOWNLOAD
Getting Started with Eclipse
Gives insights on Eclipse, the leading IDE for Java, which has a rich ecosystem of plug-ins and an open-source framework that supports other languages.
72.1k 182.7k
{{ card.title }}
{{card.downloads | formatCount }} {{card.views | formatCount }}

The best of DZone straight to your inbox.

SEE AN EXAMPLE
Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.
Subscribe

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}