How to Build You Own Personal Jenkins CI Server

In a previous article, I discussed some of the benefits of running a personal CI server. For me, the benefits break down into a few simple points:

1. I have a CI server configured to prioritize my own branches and builds.
2. I can have an up-to-date development environment without manual repo updates and rebuilds.
3. I can build and deploy while failing tests.
4. I can start long running tests and other checks and ignore the results until I’m ready to review them with a Git commit.

In this article, I’m going to show you how I went about configuring Jenkins as a personal CI server.

Requirements

Be aware that a lot of the configuration shown here is based on my own somewhat unique requirements, and some tweaking will be required to get Jenkins working for your own workflow. In particular, I had to work around the following requirements and limitations:

• I don’t own the repos that I’m building, so I can’t check in files like the jenkinsfile.
• I don’t control the Git server, so I can’t set up hooks or anything else that requires server-side administration.
• Development is done using GitFlow, with all work done in branches.
• The apps I build are based on Gradle.
• The apps are deployed to WildFly.

With those requirements in mind, let’s take a look at how you can configure Jenkins.

Installing Linux Tools

To start, I have a fresh install of Ubuntu 16.10 running in VirtualBox. I’ve done this because exporting a VirtualBox image makes it very convenient to distribute a personal CI server image.

We need to install a bunch of handy tools that don’t come with the standard Ubuntu installation. Only git and openjdk-8-jdk are required for Jenkins, but I use the other tools commonly enough to warrant installing them as a matter of course.

sudo apt-get install htop vim iftop git openssh-server openjdk-8-jdk

Installing Jenkins

To install Jenkins, we’ll use the custom repo documented here.

wget -q -O - https://pkg.jenkins.io/debian/jenkins-ci.org.key | sudo apt-key add -

sudo sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'

sudo apt-get update

sudo apt-get install jenkins

Configuring the Port

By default, Jenkins will run on port 8080. This happens to be the same default port that WildFly runs on, so we need to change it to avoid port conflicts. I use port 9080 for Jenkins. To update the port, edit the file /etc/default/jenkins, and change the line that says HTTP_PORT=8080 to HTTP_PORT=9080.

Run Jenkins

To restart Jenkins with the new port, restart the Systemd service.

sudo systemctl restart jenkins.service

Initial Jenkins Configuration

Open up http://localhost:9080 in your web browser. You’ll see a message from Jenkins asking you to enter a password found in the file /var/lib/jenkins/secrets/initialAdminPassword.

You’ll then be prompted for the plugins that you want to install. At this point, I install the suggested plugins. We’ll add some more later.

Give Jenkins a few minutes to install the suggested plugins.

Create your first admin user.

And you’re done!

Installing Additional Plugins

Click Manage Jenkins > Manage Plugins.

Click the Available tab.

Tick the following plugins:

• Maven Integration Plugin.
• Copy Artifact Plugin.
• Multi-Branch Project Plugin.
• Global Variable String Parameter.
• Simple Theme Plugin.

Click Download now and install after restart.

Tick the Restart Jenkins when installation is complete and no jobs are running option.

After Jenkins restarts, you’ll have all the plugins you’ll need.

Install WildFly

Download WildFly from this link and extract the archive into the /opt folder. I am using WildFly 10.1.0, so I ended up with a directory of /opt/wildfly-10.1.0.Final. Rename this so you end up with a directory called /opt/wildfly.

Go to the directory /opt/wildfly/docs/contrib/scripts/systemd. This contains a number of scripts that we can use to configure WildFly as a Systemd service. There is a READMEfile with instructions to follow to install the WildFly Systemd service. I’ve copied this file below.

= How to configure WildFly as a systemd service

== Create a wildfly user

    # groupadd -r wildfly

    # useradd -r -g wildfly -d /opt/wildfly -s /sbin/nologin wildfly

== Install WildFly

    # tar xvzf wildfly-10.0.0.Final.tar.gz -C /opt

    # ln -s /opt/wildfly-10.0.0.Final /opt/wildfly

    # chown -R wildfly:wildfly /opt/wildfly

== Configure systemd

    # mkdir /etc/wildfly

    # cp wildfly.conf /etc/wildfly/

    # cp wildfly.service /etc/systemd/system/

    # cp launch.sh /opt/wildfly/bin/

    # chmod +x /opt/wildfly/bin/launch.sh

== Start and enable

    # systemd start wildfly.service

    # systemd enable wildfly.service

If I ran the systemd command as indicated in the README file, I got the error Excess arguments. So I ran systemctl start wildfly.service instead of systemd start wildfly.service and systemctl enable wildfly.service instead of systemd enable wildfly.service.

Now open /opt/wildfly/bin and run the command ./add-user.sh to create a user called myci with a password of password. We’ll use this account to deploy apps via Jenkins later.

Preparing Jenkins

Open up http://localhost:9080 in the browser. Then click Manage Jenkins > Global Tool Configuration.

On this page, we can install a number of tools that we’ll make use of when building our projects. In particular, we want to add a Gradle and a Maven instance to use in our builds. I find it easier to let Jenkins download and install these tools for me.

To support our builds when deploying to WildFly, we need to define some environment variables. Click Manage Jenkins > Configure System. Find the Global properties section, tick the Environment variables checkbox, and add the following values:

• wildfly_hostname = localhost
• wildfly_port = 9990
• wildfly_username = myci
• wildfly_password = password

The last thing we need to do is set the default shell. Click Manage Jenkins > Configure System. Find the Shell section and set the Shell executable to /bin/bash.

Make Jenkins Pretty

As powerful as Jenkins is, it looks like it was designed by a developer. Jenkins Material Themes is a great way to modernize the look of your Jenkins installation.

Click Manage Jenkins > Configure System, and add the URL http://afonsof.com/jenkins-material-theme/dist/material-<color>.css to the URL of theme CSS field. You can find a list of colors on the Jenkins Material Themes website to replace the <color> marker with.

I think you’ll agree that the new theme is much easier on the eyes.

Create New Item

Now it is time to configure a job. Click the New Item link on the left-hand side when you open http://localhost:9080. We’re going to create a Freestyle multi-branch project called Sample_Project_Build.

I try to avoid spaces in item names, as I have seen it cause issues with various scripts.

Click the Add Source button, select Git as the option, and enter the repo URL of https://github.com/mcasperson/gradle-sample-war.git.

Note that even though this is a GitHub repo, I want to demonstrate how you would work with an internal Git repo that has no special hooks or other niceties that you might get with GitHub.

Then click the Advanced button, and set the Include branches option to master <name>/*. I’ve used the name "casper" in the screenshot below.

This is the first example of where we have configured Jenkins to work specifically for us as individual developers. In a typical environment, you would expect a Git repo to have multiple branches, one for each developer working on a feature. We don’t want to build other developer's branches because this is a personal CI server and we don’t really care what other developers are doing until they merge back into master.

By configuring this project to only consider the master branch and any branches that are created with your chosen name in the branch name, we can configure Jenkins to only work with the branches that we, as individual developers, are interested in.

Scroll down and click the Add Build Step button, and select the Invoke Gradle Script option. Set the switches to -x test and the Tasks to clean build.

This is yet another case where a personal CI server differs from a central one. We have skipped the tests during the build (with the -x test switches) because we don’t want to break builds just because tests fail. Every development methodology that is based on testing expects tests to fail during development. We want to know what tests have failed but we don’t want to break the development flow because of failed tests.

Now that we have a way to build the script, we need to deploy the resulting WAR file to the WildFly server.

Click the Add Build Step button again, and select Execute Shell. Paste the following script into the Command box.

pushd build/libs
shopt -s nullglob
for file in *.war; do
  war_filename="${file}"
  break
done
popd

echo "<project><modelVersion>4.0.0</modelVersion><groupId>example.org</groupId><artifactId>wildfly-deploy</artifactId><version>1</version><build><plugins><plugin><groupId>org.wildfly.plugins</groupId><artifactId>wildfly-maven-plugin</artifactId><version>1.1.0.Alpha11</version><configuration><filename>build/libs/${war_filename}</filename><hostname>${wildfly_hostname}</hostname><port>${wildfly_port}</port><username>${wildfly_username}</username><password>${wildfly_password}</password><targetDir>.</targetDir><force>true</force></configuration></plugin><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-compiler-plugin</artifactId><executions><execution><id>default-compile</id><phase>none</phase></execution></executions></plugin><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><executions><execution><id>default-compile</id><phase>none</phase></execution></executions></plugin></plugins></build></project>" > deploy.xml

Click the Add Build Step button again and select Invoke top-level Maven targets. Select the Maven Version you installed in a previous step, and then set the Goals to wildfly:deploy --fail-never. Click the Advanced button and set the POM to deploy.xml.

What we have done here is create a script to build a Maven POM file that includes just enough information to deploy a WAR file to WildFly while excluding Maven plugins like test and compile that would ordinarily be run during such a deployment. Note that we don’t break the build if this deployment fails by using the Maven --fail-never switch.

You could achieve much the same result using the WildFly CLI tool, and Jenkins even comes with a WildFly deployment plugin. However, I find using the WildFly deployment tools available within Maven to be much less of a hassle because Maven downloads everything you need and you get easy access to updates.

So, while it may seem odd to be using Maven in a project that is built with Gradle, just remember that we’re only using Maven at build time as an alternative to maintaining our own deployment tool chain. Maven is not used to build the application or manage any dependencies.

Finally, we need to run this build on at regular intervals. Click the Build periodically checkbox and set the schedule to * * * * *. This results in the Git repo being reindexed every minute, allowing Jenkins to respond to any changes.

In an ideal world, Jenkins would not have to do this kind of polling, but one of my requirements is that Jenkins works with a Git server I have no control over, so polling is our only option here.

Finally, click the Save button to create our Jenkins Item.

Testing the Build

Once saved, Jenkins will scan the repo for matching branches and find the master branch. It will then create a project for that branch and begin building it.

Once built, our Maven script will be used to deploy the resulting WAR file to WildFly.

We can open the application in WildFly.

Conclusion

We’ve done a lot of work here to configure Jenkins and WildFly from scratch to create the beginnings of a personal CI server. At this point, we have the foundation of a system that will maintain a development environment that is automatically updated as other developers commit to master and as you commit to your own branches.

In the next article, we’ll take a look at some of the neat things you can now do with your personal CI server.

The exported VirtualBox image with Jenkins configured using these steps is available here. The username and password combination for Linux, Jenkins, and WildFly is myci and password.