Running Ansible Playbooks From Jenkins

Ansible Playbooks

Ansible playbooks are used to perform a multi-step procedure on one or more remote machines. The example of such procedures could be any of the following:

• A logical set of system levels changes.

• A deployment on the remote machines.

• Some configuration changes for an application already deployed on the remote machines.

A playbook doesn't have to be executed on a remote machine always. At times, it could be doing serverless operations such as doing something with an AWS S3 bucket or provisioning VMs using APIs provided by the virtualization tool or public cloud vendor.

The entire procedure that an Ansible playbook performs is divided into multiple tasks. Typically, each task is carried out by using an Ansible module or some system command. The tasks could be defined directly in the playbook or roles. A role is basically a reusable set of tasks that perform a complete procedure. For example, Java might be needed for running different applications. In such a situation, Java installation is defined as a role which could be reused by any playbook for installing an application that requires Java and thus the Java dependency could easily be enforced.

Managing Configurations in Ansible

Ansible is generally known as a Configuration Management (CM) system and it could be used to manage the environment information about all the application stacks and related resources in a company. The roles and playbooks should be designed to use the configuration data available within Ansible.

Well-designed roles and playbooks are parametrized to work with multiple environments. The environment specific configurations can be externalized and be maintained in vars modules, as in the following example:

envs:
   env1:
      app_server: app01.domain
      db_server: db01.domain
   env2:
      app_server: app02.domain
      db_server: db02.domain

In this sample case, a playbook can be run against any one of the environments, env1 or env2, and the playbook can use the related configuration parameters defined in Ansible to make changes such as these:

• Connect to the database and make some schema change.

• Update an application config file that has app_server and/or db_server config item in it.

The vars modules are loaded into Ansible as Python data structures. In the sample case, envs will be a multi-level dictionary and the db_server setting for env2 can be referred as:

envs['env2']['db_server']

When the playbook is executed, you need to indicate which environment it is working on. One way to do that is to provide the environment name from the command-line using Ansible playbook option extra-vars, as in this example:

$ ansible-playbook -i HOSTS-LIST install-app.yml -u USER -kK --extra-vars="env=env2"

With the environment known, it is easy to figure out the value of db_server within the playbook as:

envs[env]['db_server']

The example we saw is very simple. A real-life playbook might require few variables to be provided as extra-vars at execution time so it could be maintained to work on varied environments, and, providing extra-vars values could soon become unwieldy and error prone.

Also, there is a need to specify the list of hosts that the playbook will work on. If there is no Configuration Management Database (CMDB) or CMDB-like system in place that could provide host lists dynamically, it is a good idea to store that info in Ansible as below in a vars module:

envs:
   env1:
      app_server: app01.domain
      db_server: db01.domain
      hosts:
         - host1-env1.domain
         - host2-env1.domain
         - app01.domain
         - db01.domain
   env2:
      app_server: app02.domain
      db_server: db02.domain
      hosts:
         - host1-env2.domain
         - host2-env2.domain
         - app02.domain
         - db02.domain

Once you have the hosts list in a YAML data structure like this, it is easy to fetch that info as a list dynamically by using a simple Python script, as shown in the following example.

#!/usr/bin/python

import sys
import yaml

env=sys.argv[1]
f=open('envs.yml')
envs=yaml.load(f)
hosts=envs['envs'][env]['hosts']
for host in hosts: print host

The config item values and hosts list thus generated can be used as parameters when the playbook is executed.

Jenkins as UI to Playbook

Jenkins can be used as an interface to a playbook as follows:

• For gathering values of extra-vars required to run a playbook using Jenkins job form.

• For generating hosts lists and configuration items from Ansible vars modules or from any repository that could be accessed from Jenkins server.

• For managing credentials that can be retrieved and be injected into the environment where playbook runs the scripts.

Setting Up Playbook on Jenkins

It is hard to provide the exact steps to setup an Ansible playbook on Jenkins because it depends on how you manage the playbooks and how SSH access between hosts are configured in your organization. The following general guidelines are enough for someone who knows both Ansible and Jenkins to implement the strategies described here.

Set Up Passwordless Accesses

The Jenkins jobs are run by the user jenkins. So, the passwordless SSH access must be setup for jenkins to access the remote machine as jenkins or some other service account.

The user that logs into the remote machine must have passwordless sudo access on that machine also.

Passwordless access is discussed in detail in another article and read it if you are not fully conversant with setting up passwordless SSH and sudo accesses.

Create Jenkins Job to Run Playbook

Create a Freestyle project job on Jenkins and define parameters for every input item provided to the playbook using extra-vars option. The parameter type on the Jenkins job form will be based on the availability of data. In the example we have been looking at, the env could be a Choice Parameter with possible values env1 and env2.

Generate Input Values for Playbook

Before the playbook can be launched, values for required parameters must be gathered programmatically. These are in addition to the user inputs from the job form. Mainly, there are two items to be fetched:

1. Private key for SSH-ing to remote machines. The private key can be setup temporarily if the private keys are managed in some encrypted repository including Jenkin's credentials management feature. If the private key is available on the Jenkins server at a static location, then this step is not needed.

2. List of remote machines on which the playbook will be run. This could be derived from a vars module as explained before.

Setup Playbook Command-Line

The playbook execution is defined in the Build > Execute Shell section of the job. Generation of input values and playbook execution are scripted in this section and that would look like this:

#env is Jenkins build parameter
$ `gen-hosts-list $env` > /path/to/hosts_list

$ ansible-playbook /path/to/install-app.yml -i /path/to/hosts_list -u AUTO_USER --private-key=/path/to/private-key

Advantages of Using Jenkins

• Most of the complexities associated with running a playbook can be coded away in the Jenkins form and the Execute Shell. This would enable operations staff with little or no experience with Ansible to run playbooks.

• The Jenkins build log is an excellent resource to keep track and audit the changes pushed to a specific environment.

• Jenkins has a good interface to group jobs, tabbed views,  that would help with organizing your playbooks. Jenkins also lets you document a job and the parameters on the same form interface. It is more handy to refer to documentation on the form itself than look up for that info in a wiki.

Overheads With Using Jenkins

If setup as discussed here, there is no real disadvantage in using Jenkins to execute playbooks because playbooks don’t need any additional change to be able to run from Jenkins, and, they could still be executed from command-line if needed.

To be able to execute Ansible playbooks as proposed, here few extra preparations that are needed which add to the overhead:

• Install and configure Ansible on the Jenkins instance that would be used for setting up UI for the playbooks.

• Set up passwordless SSH access between Jenkins server and remote machines. As this cannot be done on the fly usually, this method of executing playbooks is better suited for running procedures against known machines repeatedly like periodic deployments.

• If secrets like password and API keys are not handled properly then there is a higher chance for those to end up in build logs which should be prevented. Taking extra security measures could add to the overhead.

Overall, using the Jenkins job UI is an excellent idea if team members with little or no knowledge of Ansible need to get involved in using them to get things done. Taking it one step further, users outside of operations team can be granted access to run such Jenkins jobs to make certain procedures self-service.