DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Last call! Secure your stack and shape the future! Help dev teams across the globe navigate their software supply chain security challenges.

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workloads.

Releasing software shouldn't be stressful or risky. Learn how to leverage progressive delivery techniques to ensure safer deployments.

Avoid machine learning mistakes and boost model performance! Discover key ML patterns, anti-patterns, data strategies, and more.

Related

  • Maven Plugin Testing - in a Modern Way - Part I
  • The Most Popular Technologies for Java Microservices Right Now
  • Micronaut With Relation Database and...Tests
  • Go: Unit and Integration Tests

Trending

  • Why High-Performance AI/ML Is Essential in Modern Cybersecurity
  • A Modern Stack for Building Scalable Systems
  • Kubeflow: Driving Scalable and Intelligent Machine Learning Systems
  • Revolutionizing Financial Monitoring: Building a Team Dashboard With OpenObserve
  1. DZone
  2. Software Design and Architecture
  3. Integration
  4. Separating Integration and Unit Tests with Maven, Sonar, Failsafe, and JaCoCo

Separating Integration and Unit Tests with Maven, Sonar, Failsafe, and JaCoCo

Execute the slow integration tests separately from unit tests and show as much information about them as possible in Sonar.

By 
Jakub Holý user avatar
Jakub Holý
·
Feb. 08, 12 · Tutorial
Likes (1)
Comment
Save
Tweet
Share
56.3K Views

Join the DZone community and get the full member experience.

Join For Free

Goal: Execute the slow integration tests separately from unit tests and show as much information about them as possible in Sonar.

The first part – executing IT and UT separately – is achieved by using the maven-failsafe-plugin and by naming the integration tests *IT (so that the unit test running surefire-maven-plugin will ignore them while failsafe will execute them in the integration-test phase and collect results in the verify phase).

The second part – showing information about integration tests in Sonar – is little more tricky. Metrics of integration tests will not be included in the Test coverage + Unit tests success widget. You can add Integration test coverage (IT coverage) widget if you enable JaCoCo but there is no alternative for the test success metrics. But don’t despair, read on!

Important notice: The integration of Sonar, JaCoCo and Failsafe evolves quite quickly so this information may easily get outdated with the next releases of Sonar

Versions: Sonar 2.12, Maven 3.0.3


Prerequisity: How Sonar Works

It’s important to understand how Sonar works and how it is integrated into the build process. Briefly:

  • Sonar is run after the project is built (and thus various artifacts are already generated): either as a post- action in Jenkins or after executing mvn install manually
  • Sonar comes bundled with various integrated quality analysis plugins such as PMD, Checkstyle, Findbugs (depending on the quality profile chosen). You do not need to add them manually to your POM (but you can, if you need to configure them). However if you need something it doesn’t do yet such as collecting coverage for integration tests, you have to do it manually.
  • Sonar may either reuse existing reports (checkstyle, …) or generate its own.
  • F.ex. if you choose JaCoCo as the default code coverage provider, Sonar will automatically rerun unit tests with JaCoCo enabled to collect the coverage metrics. You can see the file target/sonar/sonar-pom.xml that it generates to see what it does.

Fooling Sonar to Display IT Test Success

Executing unit tests via Surefire and integration tests via Failsafe results in not beeing able to see when integration tests fail. (Because this is shown by the Unit test success widget, which doesn’t support Failsafe.) However it’s possible to fool Sonar to show test success for both unit and integration tests together by instructing Failsafe to store its test reports to the same directory as Surefire instead of the default failsafe-reports, and that’s what we do:

<!-- pom.xml, the build/plugins section -->
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-failsafe-plugin</artifactId>
    <configuration>
        <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
    </configuration>
</plugin>

How to display IT Code Coverage in Sonar

Sonar 2.12 currently isn’t able to compute test coverage for integration tests automatically and thus we have to do it manually. Summary:

  1. Add Failsafe to your Maven configuration as shown above.
  2. Add JaCoCo to Maven and combine it with Failsafe to produce the coverage report when integration tests are run
  3. Instruct Sonar to execute the verify phase and pass the path to the integration test report to it

1. Add Failsafe to your Maven configuration as shown above

See the pom.xml fragment under “Fooling Sonar to Display IT Test Success” above.

2. Add JaCoCo to Maven and combine it with Failsafe to produce the coverage report when integration tests are run

As Sonar 2.12 doesn’t automatically reconfigure Failsafe to collect code coverage, we have to instruct Failsafe manually to load the JaCoCo java agent that will collect and store the coverage report (to target/jacoco.exec by default):

<!-- pom.xml fragment: -->
<build>
 ...
     <!-- Run integration tests (*IT) -->
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-failsafe-plugin</artifactId>
        <configuration>
            <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
            <argLine>${jacoco.agent.argLine}</argLine>
        </configuration>
    </plugin>
    <!--
        Compute integration test coverage for Sonar
        BEWARE: Sonar doesn't run the verify phase, it has to be forced by setting -Dsonar.phase=verify
    -->
    <plugin>
        <groupId>org.jacoco</groupId>
        <artifactId>jacoco-maven-plugin</artifactId>
        <configuration>
            <propertyName>jacoco.agent.argLine</propertyName> <!-- default: argLine -->
            <includes>
                <include>com/comoyo/**</include>
            </includes>
            <destFile>${project.build.directory}/jacoco-integration.exec</destFile> <!-- agent -->
            <dataFile>${project.build.directory}/jacoco-integration.exec</dataFile> <!-- report -->
        </configuration>
        <executions>
            <execution>
                <id>agent</id>
                <goals><goal>prepare-agent</goal></goals>
            </execution>
        </executions>
    </plugin>
</build>

The key lines are 10 (argLine) and 28 (prepare-agent before verify – the default phase, if unspecified, is init).

3. Instruct Sonar to execute the verify phase and pass the path to the integration test report to it

Important: If you use Sonar prior to 2.12 then you must install the Sonar JaCoCo plugin manually.

Whether running Sonar from Jenkins or locally we have to make sure that the verify phase is run and pass to Sonar the path to the generated JaCoCo integration test coverage report. This is best done in Sonar in the JaCoCo plugin configuration (on the project level?) but you could also do it manually by passing it to maven via “-Dsonar.jacoco.itReportPath=target/jacoco-integration.exec” or by setting the property in pom.xml under project/properties.

Jenkins Sonar plugin does not require any special configuration if path to the report is set inside Sonar’s JaCoCo plugin.

Command line (the recommended way; the first line runs among others the phase verify and thus generates the report):

payment$ mvn clean install -Dtest=false -DfailIfNoTests=false
payment$ mvn sonar:sonar

We could actually invoke Maven with a single line, though that isn’t recommended because it would run tests multiple times:

payment$ mvn -Dsonar.phase=verify sonar:sonar
NOTE: If you set JaCoCo as the default code coverage provider in Sonar than it will produce coverage for unit tests overriding the integration test coverage from the verify phase. Solutions: 1) Don’t enable JaCoCo, 2) Configure JaCoCo to use a different file for integration tests (which we do)

Some logs of interest from running mvn sonar:sonar:

...
JaCoCo agent (version 0.5.3.201107060350) extracted: /var/folders/k0/2842tm752zv1dh4q77_gmgdr0000gn/T/jacocoagent2548420105762793132.jar
JVM options: -javaagent:/var/folders/k0/2842tm752zv1dh4q77_gmgdr0000gn/T/jacocoagent2548420105762793132.jar=destfile=target/jacoco.exec,excludes=*_javassist_*
Initializer FindbugsMavenInitializer...
Initializer FindbugsMavenInitializer done: 4 ms
Execute maven plugin maven-surefire-plugin...
Execute org.apache.maven.plugins:maven-surefire-plugin:2.8.1:test...
...
Execute maven plugin maven-surefire-plugin done: 9856 ms
Initializer JacocoMavenInitializer...
..
Sensor SquidSensor done: 2207 ms
Sensor JaCoCoSensor...
Analysing /myproject/target/jacoco.exec
Sensor JaCoCoSensor done: 559 ms
Sensor JaCoCoItSensor...
Analysing /myproject/target/jacoco-integration.exec
Sensor JaCoCoItSensor done: 89 ms
Sensor SurefireSensor...
  • Notice that I’ve JaCoCo set as my code coverage provider in Sonar and Sonar does use its own copy of it  (the line JaCoCo agent .. extracted), which it uses in the test phase
  • Notice that Sonar runs surefire:test automatically (with instrumentation) to collect code coverage
  • Notice that JaCoCo processed both *.exec files (the first generated by Sonar for unit tests, the other generated by Maven in verify prior to calling Sonar)

Tip: Compute the total code coverage of unit + integration tests

Unit and integration test coverage are computed separately; to see the total code coverage we would need to merge the two (notice we can’t just sum them as both kinds of tests can cover some of the same lines). It would be possible by using JaCoCo both to compute unit test coverage (supported out of the box) and integration test coverage into different files and using its Ant task to merge the two coverage files, passing the resulting file as the IT test coverage file to Sonar (for we cannot get a 3rd widget to display this summed coverage). However I haven’t tried it.

Note on Testing JaCoCo, Failsafe and Maven Integration

If Sonar doesn’t show the IT coverage widget though it is on the dashboard (click on Configure widgets while logged in) or shows 0% though it should be higher, you may check the data that JaCoCo is generating by producing HTML report from them. There is both an Ant task for that, which didn’t work for me, and Maven goal. This is how you instruct JaCoCo to generate the report when “mvn site” is run – notice the lines 13, 14 (remember that you must run “mvn verify” first to generate the binary coverage report):

<!-- pom.xml build/plugins fragment -->
<plugin>
    <groupId>org.jacoco</groupId>
    <artifactId>jacoco-maven-plugin</artifactId>
    <executions>
        <execution>
            <id>agent</id>
            <phase>pre-integration-test</phase>
            <goals><goal>prepare-agent</goal></goals>
        </execution>
        <execution>
            <id>report</id>
            <phase>site</phase>
            <goals><goal>report</goal></goals>
        </execution>
    </executions>
</plugin>

References

JaCoCo Maven plugin has minimal documentation but you can get useful info via:

mvn help:describe -Dplugin=org.jacoco:jacoco-maven-plugin -Ddetail

Sonar: Measure Code Coverage by Integration Tests with Sonar (9/2010) – doesn’t use the JaCoCo Maven plugin and thus has to configure the argLine manually

The Complete POM

See the complete POM at GitHub.

 

From http://theholyjava.wordpress.com/2012/02/05/separating-integration-and-unit-tests-with-maven-sonar-failsafe-and-jacoco/

unit test Integration Apache Maven integration test Code coverage

Opinions expressed by DZone contributors are their own.

Related

  • Maven Plugin Testing - in a Modern Way - Part I
  • The Most Popular Technologies for Java Microservices Right Now
  • Micronaut With Relation Database and...Tests
  • Go: Unit and Integration Tests

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!