Java Microbenchmark Harness (JMH)

Introduction

In my previous article, we established that microbenchmarking is hard with jvm. It is not enough to surround the code in a loop with System.out.println() and gather the time measurements. While benchmarking, a developer should consider warm-up cycles, JIT compilations, JVM optimizations, avoiding usual pitfalls, and even more.

Thankfully, OpenJDK has a great tool Java Microbenchmark Harness (JMH) that can help us generated benchmarking stats. In this article, we will discuss how JMH can help us avoid the pitfalls that we have discussed earlier.

Getting Started With JMH

A quick way to start with JMH is to use the Maven archetype. The command below will generate a new Java project benchmark. The project will have com/gaurav/MyBenchmark.java class and pom.xml. The Maven pom.xml includes all the required dependencies to support JMH.

mvn archetype:generate -DarchetypeGroupId=org.openjdk.jmh -DarchetypeArtifactId=jmh-java-benchmark-archetype -DinteractiveMode=false -DgroupId=com.gaurav -DartifactId=benchmark -Dversion=1.0

Good Benchmarks With JMH

Let us discuss how JMH can help us to write better microbenchmarks.

• JMH by default makes several warm-up cycles before collecting the stats. Thus, it makes sure that the results are not completely random and jvm has performed optimizations.
• @benchmark runs iteration over the code, and then collects the average. The more runs you make through the code, the better stats you will collect.
• Use Blackhole class of JMH can avoid deal code elimination by jvm. If you pass the calculated results to blackhole.consume(), it would trick the jvm. jvm will never drop the code, thinking that consume() method uses the result.

Writing First Benchmark

Maven has already provided with us a template in MyBenchmark class to fill in. I am going to utilize the same class.

package com.gaurav;

import org.openjdk.jmh.annotations.Benchmark;

public class MyBenchmark {

    @Benchmark

    public void testMethod() {

        // This is a demo/sample template for building your JMH benchmarks. Edit as needed.

        // Put your benchmark code here.

    }

}

I would like to keep my first benchmark simple. Let us start by iterating over all the elements of a list and sum them up using a conventional for loop. As discussed, I will use Blackhole to fool the compiler and return the result. Here, I am asking JMH to calculate the average time, using @BenchmarkMode, it takes to run the testMethod().

x

@Benchmark

@BenchmarkMode(Mode.AverageTime)

public static double testMethod(Blackhole blackhole) {

  double sum = 0;

  for(int i=0; i<list.size(); i++) {

    sum += list.get(i);

  }

  blackhole.consume(sum);

  return sum;

}

Compiling the JMH Project

You can compile and build the project like any other Maven project, using the below Maven command:

mvn clean install

The command will create a fully executable jar file under benchmark/target the directory. Please note that Maven will always generate a jar file named benchmarks.jar, regardless of the project name.

Next step is to execute the jar using the below command:

java -jar target/benchmarks.jar

It produced the below result for me. It means that test operation is taking approx. 0.053 seconds on the current hardware.

xxxxxxxxxx

# Run progress: 80.00% complete, ETA 00:01:41

# Fork: 5 of 5

# Warmup Iteration   1: 0.052 s/op

# Warmup Iteration   2: 0.051 s/op

# Warmup Iteration   3: 0.053 s/op

# Warmup Iteration   4: 0.056 s/op

# Warmup Iteration   5: 0.055 s/op

Iteration   1: 0.054 s/op

Iteration   2: 0.053 s/op

Iteration   3: 0.053 s/op

Iteration   4: 0.054 s/op

Iteration   5: 0.059 s/op

Result "com.example.MyBenchmark.testMethod":

  0.053 ±(99.9%) 0.002 s/op [Average]

  (min, avg, max) = (0.052, 0.053, 0.061), stdev = 0.002

  CI (99.9%): [0.051, 0.055] (assumes normal distribution)

# Run complete. Total time: 00:08:27

Benchmark Modes

In the previous example, I used @BenchmarkMode(Mode.AverageTime). If you try to decompile the JMH jar, you will find enum Mode with the below values:

The default Mode is Throughput. 

Time Measurement

It is evident from the console output above that calculations are in seconds. But, JMH allows you to configure the time units, using @OutputTimeUnit annotation. The @OutputTimeUnit accepts java.util.concurrent.TimeUnit, as shown below:

@OutputTimeUnit(TimeUnit.SECONDS)

The TimeUnit enum has the following values:

NANOSECONDS
MICROSECONDS
MILLISECONDS
SECONDS
MINUTES
HOURS
DAYS

The default TimeUnit is SECONDS. 

Configure Fork, Warmup, and Iterations

The benchmark is currently executing 5 times, with 5 warmup iterations and 5 measurement iterations. JMH even allows to configure these values using @Fork, @Warmup and @Measurement annotations. The code snippet below would execute the test method twice, with a couple of warmup iterations and 3 measurement iterations.

xxxxxxxxxx

@Fork(value = 2)

@Warmup(iterations = 2)

@Measurement(iterations = 3)

@Warmup and @Measurement annotations also accepts parameters:

• batchSize - configures the number of test method calls to be performed per operation.
• time - time spent for each iteration.

Practice

You can play around to compare execution times of different for loops i.e. a conventional for loop, a forEach loop and an stream iterator. Something like:

xxxxxxxxxx

private static final List<Integer> list = IntStream.rangeClosed(1, Integer.MAX_VALUE/100)

            .boxed().collect(Collectors.toList());

    @Benchmark

    @BenchmarkMode(Mode.AverageTime)

    public static double conventionalLoop(Blackhole blackhole) {

        double sum = 0;

        for(int i=0; i<list.size(); i++) {

            sum += list.get(i);

        }

        blackhole.consume(sum);

        return sum;

    }

    @Benchmark

    @BenchmarkMode(Mode.AverageTime)

    public static double enhancedForLoop(Blackhole blackhole) throws InterruptedException {

        double sum = 0;

        for (int integer : list) {

            sum += integer;

        }

        blackhole.consume(sum);

        return sum;

    }

    @Benchmark

    @BenchmarkMode(Mode.AverageTime)

    public static double streamMap(Blackhole blackhole) {

        double sum = list.stream().mapToDouble(Integer::doubleValue).sum();

        blackhole.consume(sum);

        return sum;

    }

Conclusion

In this article, we have gone through a hands-on example of creating a JMH project. We have seen how can we configure our JMH project to suit our needs. You can refer to JMH Github Samples for more in-depth examples.

We have seen that JMH is a jvm tool. In the next article, we will try to explore if it can help us with other jvm based languages.