Multi-Project Builds With Gradle and Fat JARs With Shadow

When deciding on your dependency manager in Java, you have two main choices: Maven and Gradle (or something more complex like Google's Bazel). Both manage dependencies well, have robust plugin systems, support checkstyles, run tests, and build/publish JARs and sources. Pick whatever you are comfortable with. Gradle is a little less verbose and is what we will be using.

Multi-Project Builds

Multi-project builds are very useful for splitting a project into separate dependencies. For example, you may have a REST service that is split into three projects core for common models/logic, client for the HTTP client that interacts with the server, and the server. You wouldn't want database dependencies in the client library, so this is a clean separation of concerns. We will be using the StubbornJava projects in the example, but the separation of logic still holds.

Parent Project

The root project for StubbornJava is the root on the StubbornJava GitHub repository. Parent projects generally only have a few Gradle files and no source code.

settings.gradle

This file is responsible for setting the root project name and including all child projects.

rootProject.name = 'stubbornjava-parent'

include ':stubbornjava-undertow'
include ':stubbornjava-common'
include ':stubbornjava-examples'
include ':stubbornjava-webapp'

def rootProjectDescriptor = settings.rootProject
settings.createProjectDescriptor(rootProjectDescriptor, 'stubbornjava-private', file('../stubbornjava'))

View on GitHub.

gradle/

The gradle/ directory is the default location for including Gradle scripts. This is a convenient location to split out our dependencies. The build.gradle file tends to get a bit cluttered, since dependencies are one of the most updated sections, and, self-contained, it's a great idea to split into its own file gradle/dependencies.gradle. We will be using Gradle's ext tag, which is used for extra properties. This is a good spot for shared variables. Normally, projects only store the version numbers here, but we also store the full dependency strings so they can be reused.

ext {
    versions = [
        jackson           : '2.8.5',       // Json Serializer / Deserializer
        okhttp            : '3.6.0',       // HTTP Client
        slf4j             : '1.7.21',      // Logging 
        logback           : '1.1.8',       // Logging
        undertow          : '1.4.8.Final', // Webserver
        metrics           : '3.1.2',       // Metrics
        guava             : '19.0',        // Common / Helper libraries
        typesafeConfig    : '1.3.1',       // Configuration
        handlebars        : '4.0.6',       // HTML templating
        htmlCompressor    : '1.4',         // HTML compression
        hikaricp          : '2.6.0',       // JDBC connection pool
        jool              : '0.9.12',      // Functional Utils
        hsqldb            : '2.3.4',       // In memory SQL db
        aws               : '1.11.98',     // AWS Java SDK
        contentful        : '7.4.0',       // Contentful java sdk
        flyway            : '4.1.2',       // DB migrations
        connectorj        : '5.1.41',       // JDBC MYSQL driver
        jooq              : '3.9.1',       // jOOQ
        hashids           : '1.0.1',       // Id hashing

        junit             : '4.12',        // Unit Testing
    ]
    libs = [
        okhttp                    : "com.squareup.okhttp3:okhttp:$versions.okhttp",
        okhttpUrlConnection       : "com.squareup.okhttp3:okhttp-urlconnection:$versions.okhttp",
        loggingInterceptor        : "com.squareup.okhttp3:logging-interceptor:$versions.okhttp",
        jacksonCore               : "com.fasterxml.jackson.core:jackson-core:$versions.jackson",
        jacksonDatabind           : "com.fasterxml.jackson.core:jackson-databind:$versions.jackson",
        jacksonAnnotations        : "com.fasterxml.jackson.core:jackson-annotations:$versions.jackson",
        jacksonDatatypeJdk8       : "com.fasterxml.jackson.datatype:jackson-datatype-jdk8:$versions.jackson",
        jacksonDatatypeJsr310     : "com.fasterxml.jackson.datatype:jackson-datatype-jsr310:$versions.jackson",
        jacksonDataformatCsv      : "com.fasterxml.jackson.dataformat:jackson-dataformat-csv:$versions.jackson",
        metricsCore               : "io.dropwizard.metrics:metrics-core:$versions.metrics",
        metricsJvm                : "io.dropwizard.metrics:metrics-jvm:$versions.metrics",
        metricsJson               : "io.dropwizard.metrics:metrics-json:$versions.metrics",
        metricsLogback            : "io.dropwizard.metrics:metrics-logback:$versions.metrics",
        metricsHealthchecks       : "io.dropwizard.metrics:metrics-healthchecks:$versions.metrics",
        undertowCore              : "io.undertow:undertow-core:$versions.undertow",
        slf4j                     : "org.slf4j:slf4j-api:$versions.slf4j",
        slf4jLog4j                : "org.slf4j:log4j-over-slf4j:$versions.slf4j",
        logback                   : "ch.qos.logback:logback-classic:$versions.logback",
        guava                     : "com.google.guava:guava:$versions.guava",
        typesafeConfig            : "com.typesafe:config:$versions.typesafeConfig",
        handlebars                : "com.github.jknack:handlebars:$versions.handlebars",
        handlebarsJackson         : "com.github.jknack:handlebars-jackson2:$versions.handlebars",
        handlebarsMarkdown        : "com.github.jknack:handlebars-markdown:$versions.handlebars",
        handlebarsHumanize        : "com.github.jknack:handlebars-humanize:$versions.handlebars",
        handlebarsHelpers         : "com.github.jknack:handlebars-helpers:$versions.handlebars",
        htmlCompressor            : "com.googlecode.htmlcompressor:htmlcompressor:$versions.htmlCompressor",
        hikaricp                  : "com.zaxxer:HikariCP:$versions.hikaricp",
        jool                      : "org.jooq:jool:$versions.jool",
        hsqldb                    : "org.hsqldb:hsqldb:$versions.hsqldb",
        s3                        : "com.amazonaws:aws-java-sdk-s3:$versions.aws",
        contentful                : "com.contentful.java:java-sdk:$versions.contentful",
        flyway                    : "org.flywaydb:flyway-core:$versions.flyway",
        connectorj                : "mysql:mysql-connector-java:$versions.connectorj",
        jooq                      : "org.jooq:jooq:$versions.jooq",
        jooqCodegen               : "org.jooq:jooq-codegen:$versions.jooq",
        hashids                   : "org.hashids:hashids:$versions.hashids",

        junit                     : "junit:junit:$versions.junit",
    ]
}

View on GitHub

build.gradle

The build.gradle file is where we will load all plugins and include our previous gradle/dependencies.gradle file. This is also where we handle building our fat JAR using the Shadow JAR plugin.

Ever run into issues where Maven/Gradle have multiple versions of the same library from different transitive dependencies? Turning on failOnVersionConflict() will help track down and resolve all these issues. Since we also stored all of our dependency strings in a variable, we can iterate them and force their versions to always be used libs.each { k, v -> force(v) }. This means we only need to override library versions if multiple transitive dependencies share the same library with different versions.

buildscript {
  repositories {
      jcenter()
  }
  // buildscript dependencies can be used for build time plugins.
  dependencies {
    classpath "com.github.jengelman.gradle.plugins:shadow:1.2.3"
  }
}

// Include a gradle script that has all of our dependencies split out.
apply from: "gradle/dependencies.gradle"

allprojects {
    // Apply the java plugin to add support for Java
    apply plugin: 'java'
    apply plugin: 'idea'
    apply plugin: 'eclipse'
    apply plugin: 'maven-publish'

    // Using Jitpack so I need the repo name in the group to match.
    group = 'com.stubbornjava.StubbornJava'
    version = '0.1.14-SNAPSHOT'

    repositories { 
        mavenLocal()
        mavenCentral()
        maven { url 'https://jitpack.io' } // This allows us to use jitpack projects
    }

    configurations.all {
        resolutionStrategy {
            // fail eagerly on version conflict (includes transitive dependencies)
            // e.g. multiple different versions of the same dependency (group and name are equal)
            failOnVersionConflict()

            // Auto force all of our explicit dependencies.
            libs.each { k, v -> force(v) }
            force('io.reactivex:rxjava:1.1.2')
            force('com.google.code.gson:gson:2.6.2')
            force('commons-logging:commons-logging:1.2')

            // cache dynamic versions for 10 minutes
            cacheDynamicVersionsFor 10*60, 'seconds'
            // don't cache changing modules at all
            cacheChangingModulesFor 0, 'seconds'
        }
    }

    // Maven Publish Begin
    task sourceJar(type: Jar) {
        from sourceSets.main.allJava
    }

    // This publishes sources with our jars.
    publishing {
        publications {
            mavenJava(MavenPublication) {
                from components.java
                artifact sourceJar {
                    classifier "sources"
                }
            }
        }
    }
    // Maven Publish End
}

subprojects {
    apply plugin: 'com.github.johnrengelman.shadow'

    shadowJar {
       classifier = null
    }
}

View on GitHub

stubbornjava-undertow/build.gradle

This project is for StubbornJava-specific undertow helper classes. We only need to reference the libs.{library name} because we stored all the dependency strings in the ext tag in the parent project.

dependencies {
    compile libs.undertowCore
    compile libs.slf4j
    compile libs.logback

    testCompile libs.junit
}

View on GitHub

stubbornjava-common/build.gradle

This project is for StubbornJava-specific common code. Notice stubbornjava-undertow is a dependency.

dependencies {
    // Project reference
    compile project(':stubbornjava-undertow')
    compile libs.slf4j
    compile libs.logback
    compile libs.jacksonCore
    compile libs.jacksonDatabind
    compile libs.jacksonDatabind
    compile libs.jacksonAnnotations
    compile libs.jacksonDatatypeJdk8
    compile libs.jacksonDatatypeJsr310
    compile libs.jacksonDataformatCsv
    compile libs.metricsCore
    compile libs.metricsJvm
    compile libs.metricsJson
    compile libs.metricsLogback
    compile libs.metricsHealthchecks
    compile libs.guava
    compile libs.typesafeConfig
    compile libs.handlebars
    compile libs.handlebarsJackson
    compile libs.handlebarsMarkdown
    compile libs.handlebarsHelpers
    compile libs.handlebarsHumanize
    compile libs.htmlCompressor
    compile libs.hikaricp
    compile libs.jool
    compile libs.okhttp
    compile libs.okhttpUrlConnection
    compile libs.loggingInterceptor

    testCompile libs.junit
}

View on GitHub

stubbornjava-examples/build.gradle

This project is for StubbornJava specific examples.

dependencies {
    compile project(':stubbornjava-undertow')
    compile project(':stubbornjava-common')
    compile libs.hsqldb
    compile libs.hashids
    testCompile libs.junit
}

View on GitHub

Building a Fat JAR With Shadow

Now that we have a working multi-project build, let's create an executable JAR. For our example embedded REST service. (Assume we are in the root Gradle directory.)

gradle shadowJar
Configuration on demand is an incubating feature.
:stubbornjava-undertow:compileJava UP-TO-DATE
:stubbornjava-undertow:processResources UP-TO-DATE
:stubbornjava-undertow:classes UP-TO-DATE
:stubbornjava-undertow:jar
:stubbornjava-common:compileJava
:stubbornjava-common:processResources UP-TO-DATE
:stubbornjava-common:classes
:stubbornjava-common:shadowJar
:stubbornjava-common:jar
:stubbornjava-examples:compileJava
:stubbornjava-examples:processResources UP-TO-DATE
:stubbornjava-examples:classes
:stubbornjava-examples:shadowJar
:stubbornjava-undertow:shadowJar

BUILD SUCCESSFUL

Total time: 6.638 secs

You should now be able to run the self-contained JAR:

java -Denv={env} -Xmx{max-heap} -cp '{path-to-jar}' {fully-qualified-class-with-main}

What is very nice about this style of passing the main class instead of using a manifest is that the same JAR can be used to run any main method. In this case, any of the example servers can be run with this JAR.

java -Denv=local -Xmx640m -cp 'stubbornjava-examples/build/libs/stubbornjava-examples-0.1.2-SNAHOT.jar' com.stubbornjava.examples.undertow.rest.RestServer
2017-02-20 15:37:54.760 [main] DEBUG c.s.common.undertow.SimpleServer - ListenerInfo{protcol='http', address=/0:0:0:0:0:0:0:0:8080, sslContext=null}

curl -X POST "localhost:8080/users" -d '
{
  "email": "[email protected]",
  "roles": ["USER"]
}
';
{"email":"[email protected]","roles":["USER"],"dateCreated":"2017-01-16"}

curl -X POST "localhost:8080/users" -d '
{
  "email": "[email protected]",
  "roles": ["ADMIN"]
}
';
{"email":"[email protected]","roles":["ADMIN"],"dateCreated":"2017-01-16"}