JVM Advent Calendar: Building Monorepo Projects With Gradle

According to Wikipedia, a monorepo is a software development strategy where many projects are stored in the same repository. This strategy allows for quick detection of potential problems and breakages caused by changes in dependencies, and it has been adopted by many organizations that work with large scale codebases, such as Google, Facebook, and Twitter.

You, too, can apply this strategy if you happen to use Gradle as your build tool of choice, thanks to a feature known as Composite Builds, which were introduced back in version 3.1 (at the time of writing the latest version is 5.0). Let’s have a look at a typical monorepo workflow when this feature is not in use.

Life Without Composite Builds

Let’s imagine you’ve just started work at a company where projects are kept in a single repository. Each project has a separate build, and the only relationship between them is via dependencies on one another, as it fit their needs. Some projects will have more dependencies than others, and some projects may not even have dependencies to the others.

The number of projects is important; when it’s low, you could say that all of them can fit under an umbrella project, just like it’s done with Maven and its reactor feature. Gradle has a similar feature; except, it’s easier to target a particular build without triggering all other projects. In a way, you can say that Gradle’s reactor is smarter and picking the targets to be executed.

But what happens when a project number goes above a dozen, say a couple hundreds? Even with a smarter reactor, Gradle would have to read the configuration of all projects and then resolve the appropriate targets. This will certainly take precious time of your daily work, and that’s a big no-no.

The solution would be to break down each project into individual builds. Gone is the reactor feature; thus, we don’t have to pay the price of reading and configuring all projects to later discard most of them. However, now, we lost the opportunity to react when a dependency may have introduced a bug or a binary incompatibility, which is one of the reasons to organize code in a monorepo.

Now, we have to follow the old and tried workflow of:

• Make a change on the dependency project.
• Build and publish artifacts to a repository. Most people rely on snapshot artifacts.
• Make sure the dependent project consumes the freshly published artifacts/snapshots.
• Compile and run tests to figure out if the code works again.
• Rinse and repeat until it works.

The problem with this approach is that we waste time publishing intermediate artifacts, and from time to time, we’ll forget to publish a snapshot release and spend hours in a debugging session until we realize the binaries are incorrect, ugh.

Composite Builds to the Rescue

Let’s look now at how Composite Builds may solve the problem we’re found ourselves in. We begin by looking at the following projects and their dependencies between them

Project1

Project2 <– depends — Project1

Project3 <– depends — Project2

This small dependency graph tells us that any changes made to Project1 will affect Project2 and, by consequence, to Project3, as well. This is because changes to Project2 also affect Project3. The directory structure for this monorepo looks like this:

├── project1
│   └── build.gradle
├── project2
│   └── build.gradle
└── project3
    └── build.gradle

Here, we can see the three projects with their respective builds files. Each project has its own release lifecycle and version, as we can observe in their build files

project1/build.gradle

apply plugin: 'java'

group   = 'com.acme'
version = '1.0.0'

project2/build.gradle

apply plugin: 'java'

group   = 'com.acme'
version = '2.3.0'

dependencies {
    compile 'com.acme:project1:1.0.0'

project3/build.gradle

apply plugin: 'java'

group   = 'com.acme'
version = '1.2.0'

dependencies {
    compile 'com.acme:project2:2.3.0'
}

Activating the Composite Builds feature requires configuring the link between projects in a file named settings.gradle. Projects 2 and 3 required this file; thus, our repository looks like this:

.
├── project1
│   └── build.gradle
├── project2
│   ├── build.gradle
│   └── settings.gradle
└── project3
    ├── build.gradle
    └── settings.gradle

Next, we write down the links between projects, like so:

project2/settings.gradle

includeBuild '../project1'

project3/settings.gradle

includeBuild '../project2'

Great. With this setup in place, we can now build Project3 by issuing the following commands:

$ cd project3
$ pwd
/tmp/project3
$ gradle classes
> Task :processResources
> Task :project2:processResources
> Task :project1:compileJava
> Task :project1:processResources
> Task :project1:classes
> Task :project1:jar
> Task :project2:compileJava
> Task :project2:classes
> Task :project2:jar
> Task :compileJava
> Task :classes

As you can appreciate, both project1 and project2 were built as well. Making a change in project1 and triggering the build on project3, once again, will build all three projects as expected. Now, imagine growing this monorepo to dozens or hundreds of projects, and you’ll quickly realize that there’s little need to have snapshot releases, if any. Gradle has other features up its sleeve, like task caching of inputs/outputs, which make builds faster as well. Similarly, the recently announced build cache feature speeds up builds by “yoinking” outputs that have been computed by other nodes in a CI farm.

If you enjoyed this article, you may find other interesting posts about Gradle and build tools in general on my blog. Go check it out!