How to Integrate FitNesse Test into Jenkins

In an ideal continuous integration pipeline different levels of testing are involved. Individual software modules are typically validated through unit tests, whereas aggregates of software modules are validated through integration tests. When a continuous integration build tool like Jenkins is used it is natural to define different build steps, each step returning feedback and generating test reports and trend charts for a specific level of testing.

FitNesse is a lightweight testing framework that is meant to implement integration testing in a highly collaborative way, which makes it very suitable to be used within agile software projects. With Jenkins and Maven it is quite easy to trigger the execution of FitNesse integration tests automatically. When properly configured and bootstrapped, Jenkins can treat the FitNesse test results in a very similar way as it treats regular JUnit test results.

Now lets suppose within a Maven project we have a FitNesse suite that contains the integration tests we want to be executed by a Jenkins job. With the Maven Failsafe Plugin and the help of some convenient FitNesse built-in JUnit utility classes this can be accomplished really easily.

First of all we need to create a JUnit integration test class that will actually bootstrap the FitNesse tests. Lets says this class is named FitNesseIT. Within this class we need to instantiate a JUnitXMLTestListener and a JUnitHelper in such a way that Jenkins will automatically recognize the test results as regular JUnit test results:

import fitnesse.junit.*;
 
resultListener = new JUnitXMLTestListener("target/failsafe-reports");
jUnitHelper = new JUnitHelper(".", "target/fitnesse-reports", resultListener);

The port property of the JUnitHelper does not need to be set when using the SLIM test system. However, if the FIT test system is used, this port must be set to an appropriate value as it specifies the port number of the FitServer that will be launched to execute the FIT tests. It is recommended to assign a random free available port, as it is considered a good practice to avoid using any fixed port on the executing Jenkins node:

// if test system == FIT
socket = new ServerSocket(0);
jUnitHelper.setPort(socket.getLocalPort());
socket.close();

The debugMode property of the JUnitHelper should not be changed. It is set to true by default, which means that the SlimService or FitServer will efficiently run within the same Java process that is created by the Maven Failsafe Plugin to run the integration test.

The JUnitHelper will be used to kick off the execution of the actual FitNesse tests:

@Test
public void assertSuitePasses() throws Exception {
    jUnitHelper.assertSuitePasses(suiteName);
}

The execution of the FitNesseIT test class itself can be triggered through the use of the Maven Failsafe Plugin. In this way the FitNesse suite will be executed automatically as part of the Maven lifecycle integration-test build phase.

The FitNesseIT test class can also be executed from your IDE, which makes it really easy to actually debug the FitNesse tests by stepping through the fixture classes.

Instead of instantiating a JUnitHelper ourself, we could have used the JUnit runner class FitNesseSuite and specified by annotation the actual FitNesse suite that needs to be executed as a JUnit test. However this runner class does not create the JUnit XML report files that need to be processed by Jenkins.

As the JUnitXMLTestListener will already create report files for all individual FitNesse tests, there is no need to have a separate report file for the bootstrapping FitNesseIT test class itself. Therefore, the disableXmlReport configuration property of the Maven Failsafe Plugin need to be enabled. In this way the Jenkins job will only take the results of the individual FitNesse tests into account when generating its test report and trend chart.

Furthermore, the system property variables TEST_SYSTEM and SLIM_PORT need to be configured appropriately:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-failsafe-plugin</artifactId>
    <executions>
        <execution>
            <goals>
                <goal>integration-test</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <disableXmlReport>true</disableXmlReport>
        <systemPropertyVariables>
            <TEST_SYSTEM>slim</TEST_SYSTEM>
            <SLIM_PORT>0</SLIM_PORT>
        </systemPropertyVariables>
    </configuration>
</plugin>

By setting the SLIM_PORT to 0, the SLIM executor will run on a random free available port, so no fixed port will be used on the executing Jenkins node. Obviously, when using FIT the TEST_SYSTEM variable must be set to fit instead of slim and the SLIM_PORT variable is not needed.

Alternatively, the TEST_SYSTEM and SLIM_PORT variables can be defined with the Fitnesse define keyword:

!define TEST_SYSTEM {slim}
!define SLIM_PORT {0}

As Jenkins automatically scans the failsafe-reports directories “**/target/failsafe-reports”, the FitNesse test results will be processed out of the box. No additional Jenkins plugins are required.

The JUnitHelper also creates a nice HTML report that consist of a summary including some useful statistics as well as detailed test result pages for all executed tests. This report can be found in the “target/fitnesse-reports” directory and can be published by a post-build action with the HTML Publisher Plugin.

In a continuous integration pipeline it makes sense to trigger the execution of the integration tests in an individual build step. This can be accomplished typically by activating the Maven Failsafe Plugin using a Maven profile. In this way the integration test results and unit test results are not mixed into the same reports and trend charts by Jenkins.