How to Test Gradle Plugins

In this article, I share my experience of creating functional tests for a custom Gradle plugin and how to configure the plugin project to collect code coverage metrics from tests.

In the previous article, I described how to build a custom Gradle plugin. Here, we will continue to work with it. Before we start, I’d recommend recapping things in the previous article to get a better understanding of where we started.

0. Clone the Project 

git clone -b chapter-1 https://github.com/SurpSG/code-lines-counter-gradle-plugin.git

1. Configuration

Create a separate source set that will contain functional tests. Start with the creation of a simple directory, src/funcTests/kotlin.

After creation, the directory looks like a usual folder, and it's not recognized as a code source by the IDE.

It's time to explain to Gradle that this directory will contain a code or in other words make it a "sourceSet." 

Open the 'build.gradle' file in the root of the project, add a new source set configuration, and reimport the project:

x

sourceSets {

    functionalTest { // declares a new sourceset

        // specifies code source dir

        kotlin.srcDir file('src/funcTests/kotlin') 

        // specifies resource dir

        resources.srcDir file('src/funcTests/resources') 

        // specifies dependencies to compile test classes

        compileClasspath += sourceSets.main.output + configurations.testRuntimeClasspath

        // specifies dependencies to run tests

        runtimeClasspath += output + compileClasspath

    }

}

Create a Gradle task that will run functional tests and add a Kotlin std lib dependency for the test configuration:

xxxxxxxxxx

task functionalTest(type: Test) {

    description = 'Runs the functional tests.'

    group = 'verification'

    testClassesDirs = sourceSets.functionalTest.output.classesDirs

    classpath = sourceSets.functionalTest.runtimeClasspath

}

check.dependsOn functionalTest

2. Tests Creation

You can use any testing framework you like. In this example, I use JUnit 4.12. Create com.github.CodeLinesPluginTest Kotlin class:

xxxxxxxxxx

class CodeLinesPluginTest {

    // creates temp directory for a gradle project                      <-------- (1)

    @get:Rule

    var testProjectDir = TemporaryFolder()

    private lateinit var buildFile: File

    private lateinit var gradleRunner: GradleRunner

    @Before

    fun setup() {

        // creates empty build.gradle file in the test gradle project   <-------- (2)

        buildFile = testProjectDir.newFile("build.gradle")

        // creates and configures gradle runner                         <-------- (3)

        gradleRunner = GradleRunner.create()

            .withPluginClasspath()

            .withProjectDir(testProjectDir.root)

            .withTestKitDir(testProjectDir.newFolder())

    }

    @Test

    fun `check test setup`() {

        // runs `tasks` gradle task                                     <-------- (4)

        val result = gradleRunner

            .withArguments("tasks")

            .build()

        println(result.output)

    }

}

It is a simple functional test in the example above. The test creates a Gradle project and runs the "tasks" Gradle task. Let's explore the test step by step:

1. Declare the rule that cares about temporary directory creation. The directory is used as the project root.
2. Create an empty build.gradle file in the project's root directory.
3. Create a Gradle Runner that will help us to set up/build/run a test Gradle project.
4. Execute the Gradle task, `tasks`. The Gradle Runner returns the execution result that is used for assertions.

Run the test and observe that basic Gradle tasks are printed to the console. We've just checked that our configuration is correct.

Apply the "code-lines" plugin:

xxxxxxxxxx

@Before

fun setup() {

    buildFile = testProjectDir.newFile("build.gradle")

    // add common configuration for all tests in this class

    buildFile.appendText("""

        plugins {

            id 'java'                      // `code-lines` plugin is dependent on `java` plugin

            id 'com.github.code-lines'

        }

    """.trimIndent())

    ...

}

Then, update the test:

xxxxxxxxxx

@Test

fun `codeLines task should print '0' when there is no source code`() {

    val result = gradleRunner

        .withArguments("codeLines")                             // <------- (1)

        .build()

    assertEquals(SUCCESS, result.task(":codeLines")!!.outcome)  // <------- (2)

    assertTrue(result.output.contains("Total lines: 0"))        // <------- (3)

}

Now, the test does the following steps:

1. Invokes the codeLines task.
2. Verifies codeLines' execution status.
3. Verifies the output contains an expected message. The result is "Total lines: 0" because the test project doesn't have any code.

It's the simplest happy path test for the plugin. Let's add another one:

1. Add a Java class to verify the plugin counts lines properly.
2. Apply the non-default plugin's configuration.

Create a simple Java class by location code-lines-counter-gradle-plugin/src/funcTests/resources/TestClass.java 

xxxxxxxxxx

public class TestClass {

    public static void main(String[] args) {

        System.out.println("Hello world");

    }

}

Add a new test:

xxxxxxxxxx

@Test

fun `codeLines task should print 'Total lines 6'`() {

    // creates folders in the temp project

    val testClassLocation: File = testProjectDir.newFolder("src", "main", "java").resolve("TestClass.java")

    CodeLinesPluginTest::class.java.classLoader

        .getResource("TestClass.java")!!.file.let(::File)

        .copyTo(testClassLocation)  // copies test file from resources to test project 

    val result = gradleRunner

        .withArguments("codeLines")

        .build()

    assertEquals(SUCCESS, result.task(":codeLines")!!.outcome)

    assertTrue(result.output.contains("Total lines: 6"))

}

Add one more test to check the plugin's configuration:

xxxxxxxxxx

@Test

fun `codeLines should skip blank lines`() {

    val testClassLocation: File = testProjectDir.newFolder("src", "main", "java").resolve("TestClass.java")

    CodeLinesPluginTest::class.java.classLoader

        .getResource("TestClass.java")!!.file.let(::File)

        .copyTo(testClassLocation)

    // apply `code-lines` plugin configuration

    buildFile.appendText("""

        codeLinesStat {

            sourceFilters.skipBlankLines = true 

        }

    """.trimIndent())

    val result = gradleRunner

        .withArguments("codeLines")

        .build()

    assertEquals(SUCCESS, result.task(":codeLines")!!.outcome)

    assertTrue(result.output.contains("Total lines: 5"))

}

3. Code Coverage

Apply the Jacoco plugin. Open build.gradle and update it with:

plugins {  
    id 'jacoco'
}

jacocoTestReport {
    reports.html.enabled = true
    executionData.setFrom fileTree(buildDir).include("/jacoco/*.exec")  // <------ (1) 
}

jacoco {
    toolVersion = '0.8.7'     // <----- (2)
}

1. Specify what coverage data files to analyze. As we have two separate test source sets, we need to tell Jacoco about it. 
2. Set Jacoco version. I prefer the latest version, especially if I'm using Kotlin.

At this step, we should add some extra configurations. Tests executed with the TestKit are run in daemon JVM. That's why we need to tell daemon JVM to use the Jacoco Java agent.

Luckily, we can use Jacoco-gradle-testkit-plugin that helps us here:

plugins {
    id "pl.droidsonroids.jacoco.testkit" version "1.0.8"
}

functionalTest.dependsOn generateJacocoTestKitProperties
jacocoTestKit {
    applyTo("functionalTestRuntimeOnly", tasks.named("functionalTest"))
}

Update the GradleRunner configuration in the CodeLinesPluginTest class:

gradleRunner = GradleRunner.create()
            .withPluginClasspath()
            .withProjectDir(testProjectDir.root)
            .withTestKitDir(testProjectDir.newFolder())
            .apply {
                // gradle testkit jacoco support
                javaClass.classLoader.getResourceAsStream("testkit-gradle.properties")?.use { inputStream ->
                    File(projectDir, "gradle.properties").outputStream().use { outputStream ->
                        inputStream.copyTo(outputStream)
                    }
                }
            }            

Now, we can run tests and check coverage:

xxxxxxxxxx

gradlew check jacocoTestReport

Open {PROJECT-ROOT}/build/reports/jacoco/test/html/index.html in your favorite browser:

There’s one more plugin I recommend to use for your project. DiffCoverage builds a coverage report based on new or modified code. It may help to keep a high percentage of code coverage.

Full Working Example

Follow by the link or clone the project:

xxxxxxxxxx

git clone -b chapter-2-testing https://github.com/SurpSG/code-lines-counter-gradle-plugin.git

Conclusions

Functional tests are important because they show you if your plugin works correctly. You can check your plugin, but don't overdo the creation of tests because they are very poor in terms of performance. I prefer common happy path scenarios, like checking a plugin's configuration, plugin's task lifecycle, interaction with other plugins.

Code coverage tools help to detect untested code, so potentially, you can find and fix defects before your code is committed. Don't fully rely on such tools because they cannot guarantee that all corner cases are covered. They just show you if a code was invoked on tests run.

References

• Testing Gradle plugins (Gradle official guide).
• Code lines counter plugin (Github).
• Jacoco Gradle plugin.
• Diff coverage plugin.