Using Multi-Branch Pipelines in the Apache Maven Project

This is a post about how using Jenkins and Pipeline has enabled the Apache Maven project to work faster and better.

Most Java developers should have at least some awareness of the Apache Maven project. Maven is used to build a lot of Java projects. In fact, the Jenkins project and most Jenkins plugins are currently built using Maven.

After the release of Maven 3.3.9 in 2015 (at least from the outside), the project might have appeared to be stalled. In reality, the project was trying to resolve a key issue with one of its core components: Eclipse Aether. The Eclipse Foundation had decided that the Aether project was no longer active and had started termination procedures.

Behind the scenes, the Maven Project Management Committee was negotiating with the Eclipse Foundation and getting all the IP clearance from committers required in order to move the project to Maven. Finally, in the second half of 2016, the code landed as Maven Resolver.

But code does not stay still.

There had been other changes made to Maven since 3.3.9, and the integration tests had not been updated in accordance with the project conventions.

The original goal had been to get a release of Maven itself with Resolver and no other major changes in order to provide a baseline. This goal was no longer possible.

In January 2017, the tough decision was taken.

Reset everything back to 3.3.9 and merge in each feature cleanly, one at a time, ideally with a full clean test run on the main supported platforms: Linux and Windows, Java 7 and 8.

In a corporate environment, you could probably spend money to work your way out of trying to reconstruct a subset of 14 months of development history. The Apache Foundation is built on volunteers. The Maven project committers are all volunteers working on the project in their spare time.

What was needed was a way to let those volunteers work in parallel preparing the various feature branches while ensuring that they get feedback from the CI server so that there would be very good confidence of a clean test run before the feature branch is merged to master.

Enter Jenkins Pipeline Multibranch and the Jenkinsfile.

A Jenkinsfile was setup to do the following:

• Determines the current revision of the integration tests for the corresponding branch of the integration tests repository (falling back to the master branch if there is no corresponding branch).
• Checks out Maven itself and builds it with the baseline Java version (Java 7) and records the unit test results.
• Is in parallel on Windows and Linux build agents, with both Java 7 and Java 8. Checks out the single revision of the integration tests and runs those tests against the Maven distribution, recording all the results at the end.

There are more enhancements planned for the Jenkinsfile (such as moving to the declarative syntax) but with just this we were able to get all the agreed scope merged and cut two release candidates.

The workflow is something like this:

1. Developer starts working on a change in a local branch.
2. The developer recognizes that some new integration tests are required, so creates a branch with the same name in the integration tests repository.
3. When the developer is ready to get a full test run, they push the integration tests branch (integration tests have to be pushed first at present) and then push the core branch.
4. The Apache GitPubSub event notification system sends notification of the commit to all active subscribers.
5. The Apache Jenkins server is an active subscriber to GitPubSub and routes the push details into the SCM API plugin’s event system.
6. The Pipeline Multibranch plugin creates a branch project for the new branch and triggers a build
7. Typically, the build is started within five seconds of the developer pushing the commit.
8. As the integration tests run in parallel, the developer can get the build result as soon as possible.
9. Once the branch is built successfully and merged, the developer deletes the branch.
10. GitPubSub sends the branch deletion event and Jenkins marks the branch job as disabled (we keep the last three deleted branches in case anyone has concerns about the build result)

The general consensus among committers is that the multi-branch project is a major improvement to what we had before.