Set up a Nightly Build Process with Jenkins, SVN and Nexus

we wanted to set up a nightly integration build with our projects so that we could run unit and integration tests on the latest version of our applications and their underlying libraries.  we have a number of libraries that are shared across multiple projects and we wanted this build to run every night and use the latest versions of those libraries even if our applications had a specific release version defined in their maven pom file.  in this way we would be alerted early if someone added a change to one of the dependency libraries that could potentially break an application when the developer upgraded the dependent library in a future version of the application.

the chart below illustrates our dependencies between our libraries and our applications.

updating versions nightly

both the crossdock-shared and messaging-shared libraries depend on the siesta framework library.  the crossdock web service and crossdockmessaging applications both depend on the crossdock-shared and messaging-shared libraries.  because of the dependency structure, we wanted the siestaframework library built first.  the crossdock-shared and messaging-shared libraries could be built in parallel, but we didn’t want the builds for the crossdock web service and crossdockmessaging applications to begin until all the libraries had finished building.  we also wanted the nightly build to tag a subversion with the build date as well as upload the artifact to our nexus “nightly build” repository.  the resulting artifact would look something like siestaframework-20120720.jar

also as i had mentioned, even though the crossdockmessaging app may specify in its pom file it depends on version 5.0.4 of the siestaframework library.  for the purposes of the nightly build, we wanted it to use the freshly built siestaframework-nightly-20120720.jar version of the library.

the first problem to tackle was getting the current date into the project’s version number.  for this i started with the jenkins zentimestamp plugin .  with this plugin the format of jenkin’s build_id timestamp can be changed.  i used this to specify using the format of yyyymmdd for the timestamp.

the next step was to get the timestamp into the version number of the project.  i was able to accomplish this by using the maven versions plugin.  one thing the versions plugin can do is allow you to dynamically override the version number in the pom file at build time.  the code snippet from the siestaframework’s pom file is below.

<plugin>
   <groupid>org.codehaus.mojo</groupid>
   <artifactid>versions-maven-plugin</artifactid>
   <version>1.3.1</version>
</plugin>

at this point the jenkins job can be configured to invoke the “versions;set” goal, passing in the new version string to use.  the ${build_id} jenkins variable will have the newly formatted date string.

this will produce an artifact with the name siestaframework-nightly-20120720.jar

uploading artifacts to a nightly repository

since this job needed to upload the artifact to a different repository from our release repository that's defined in our project pom files, the “altdeploymentrepository” property was used to pass in the location of the nightly repository.

the deployment portion of the siestaframework job specifies the location of the nightly repository where ${lynden_nightly_repo} is a jenkins variable containing the nightly repo url.

tagging subversion

finally, the jenkins subversion tagging plugin was used to tag svn if the project was successfully built.  the plugin provides a post-build action for the job with the configuration section shown below.

dynamically updating dependencies

so now that the main project is set up, the dependent projects are set up in a similar way, but need to be configured to use the siestaframework-nightly-20120720 of the dependency rather than whatever version they currently have specified in their pom file.  this can be accomplished by changing the pom to use a property for the version number of the dependency.  for example, if the snippet below was the original pom file—

<dependencies>
   <dependency>
      <groupid>com.lynden</groupid>
      <artifactid>siestaframework</artifactid>
      <version>5.0.1</version>
   </dependency>
</dependencies>

—changing it to the following would allow the siestaframework version to be set dynamically:

<properties>
   <siesta.version>5.0.1</siesta.version>
</properties>

<dependencies>
   <dependency>
      <groupid>com.lynden</groupid>
      <artifactid>siestaframework</artifactid>
      <version>${siesta.version}</version>
   </dependency>
</dependencies>

this version can then be overriden by the jenkins job. the example below shows the jenkins configuration for the crossdock-shared build.

enforcing build order

the final step in this process is setting up a structure to enforce the build order of the projects.  the dependencies are set up in such a way that siestaframework needs to be built first, and the crossdock-shared and messaging-shared libraries can be run concurrently once siestaframework finishes. the crossdock web service and crossdockmessaging application jobs can be run concurrently, too, but not until after both shared libraries have finished.

setting up the crossdock-shared and messaging-shared jobs to be built after the siestaframework finishes is pretty straightforward.  in the jenkins job configuration for both the shared libraries, the following build trigger is added:

to satisfy the requirement that the apps build only after all libraries have built, i enlisted the help of the join plugin .  the join plugin can be used to execute a job once all “downstream” jobs have completed.  what does this mean exactly?  looking at the diagram below, the crossdock-shared and the messaging-shared jobs are “downstream” from the siestaframework job.  once both of these jobs complete, a join trigger can be used to start other jobs.

in this case, rather than having the join trigger kick off other app jobs directly, i created a dummy join job.  in this way, as we add more application builds, we don’t need to keep modifying the siestaframework job with the new application job we just added.

to illustrate the configuration, siestaframework has a new post-build action (below):

join-build is a jenkins job i configured that does not do anything when executed.  then our crossdock web service and crossdockmessaging applications define their builds to trigger as soon as join-build has completed.

in this way we are able to run builds each night that will update to the latest version of our dependencies as well as tag svn and archive the binaries to nexus.  i’d love to hear feedback from anyone who is handling nightly builds via jenkins, and how they have handled the configuration and build issues.