Spring Boot: Where Do the Default Metrics Come From?

Have you ever noticed all of the default metrics Spring Boot and Micrometer generate for your app? If not — you can add the actuator dependency to your project and hit the /actuator/metrics endpoint where you will find useful info about the JVM, processes, Tomcat, traffic, and so on. Then, add some cache, data sources, or JPA dependencies and even more metrics appear. If you have ever wondered how they end up there, and where we can find an explanation about what params they describe, then this is a post for you.

Displaying the Metrics

Just to keep it organized, let’s start with how to display the metrics in a Spring Boot app. If you already know it,  feel free to skip this section.

Metrics in Spring Boot are handled by micrometer.io, but if you use an actuator, there is no need to add a micrometer dependency to your project as the actuator already depends on it. An actuator is something you want you to use, even if you are not interested in the endpoints it provides, as this is the module that registers a lot of the metrics through its AutoConfigurations. But more about that a little later.

So to start, simply add the actuator dependency to your project (here to build.gradle.kts)

dependencies {
    implementation("org.springframework.boot:spring-boot-starter-actuator")
}

And to display metric names in the actuator endpoint, hit http://localhost:8080/actuator/metrics.

{
  "names": [
    "jvm.threads.states",
    "process.files.max",
    "jvm.memory.used",
    "jvm.gc.memory.promoted",
    "jvm.memory.max",
    "system.load.average.1m",
    ...
  ]
}

Then, to see the details, add a metric name to the URL path, eg. http://localhost:8080/actuator/metrics/system.cpu.count.

{
  "name": "system.cpu.count",
  "description": "The number of processors available to the Java virtual machine",
  "baseUnit": null,
  "measurements": [
    {
      "statistic": "VALUE",
      "value": 8
    }
  ],
  "availableTags": [
  ]
}

These metrics can be regularly sent to a metrics system of your choice (Prometheus, New Relic, CloudWatch, Graphite, etc) by providing a specific meter registry. Let’s do it with the simplest registry ever — LoggingMeterRegistry, which just periodically logs all the metrics.

@Configuration
class MetricsConfig {

    @Bean
    LoggingMeterRegistry loggingMeterRegistry() {
        return new LoggingMeterRegistry();
    }
}

Now, the metrics are displayed in logs as well:

2019-07-17 11:07:09.406  INFO 91283 --- [trics-publisher] i.m.c.i.logging.LoggingMeterRegistry     : jvm.buffer.count{id=direct} value=0 buffers
2019-07-17 11:07:09.406  INFO 91283 --- [trics-publisher] i.m.c.i.logging.LoggingMeterRegistry     : jvm.buffer.count{id=mapped} value=0 buffers
2019-07-17 11:07:09.406  INFO 91283 --- [trics-publisher] i.m.c.i.logging.LoggingMeterRegistry     : jvm.buffer.memory.used{id=direct} value=0 B
2019-07-17 11:07:09.406  INFO 91283 --- [trics-publisher] i.m.c.i.logging.LoggingMeterRegistry     : jvm.buffer.memory.used{id=mapped} value=0 B
2019-07-17 11:07:09.408  INFO 91283 --- [trics-publisher] i.m.c.i.logging.LoggingMeterRegistry     : jvm.classes.loaded{} value=8530 classes
2019-07-17 11:07:09.408  INFO 91283 --- [trics-publisher] i.m.c.i.logging.LoggingMeterRegistry     : jvm.gc.live.data.size{} value=0 B
2019-07-17 11:07:09.408  INFO 91283 --- [trics-publisher] i.m.c.i.logging.LoggingMeterRegistry     : jvm.gc.max.data.size{} value=0 B
2019-07-17 11:07:09.410  INFO 91283 --- [trics-publisher] i.m.c.i.logging.LoggingMeterRegistry     : jvm.memory.committed{area=nonheap,id=Compressed Class Space} value=6.25 MiB
2019-07-17 11:07:09.410  INFO 91283 --- [trics-publisher] i.m.c.i.logging.LoggingMeterRegistry     : jvm.memory.committed{area=heap,id=G1 Eden Space} value=168 MiB
...

Metrics Supply

So, how are these metrics are being provided? One example might be WebMvcMetricsFilter, adding performance metrics to all your Spring Web MVC endpoints (http.server.requests metric).

But this case is easy. When all your requests are handled by the Spring Framework, it’s a no brainer to add a call generating metrics inside (just check the WebMvcMetricsFilter.record() method).

But what about cases when you use pure ehcache or hibernate or some other data source, and then the metrics are generated anyway?

And what about cache.* metrics, which are generated even if I  @Autowired pure  net.sf.ehcache.Cache?

And what about hibernate.* metrics which are generated even if I  @Autowired pure  org.hibernate.SessionFactory?

And then, how are the  jvm.*,  process.*,  tomcat.*, etc. being generated?

It appears that it’s simpler than one might think, as these statistics are provided by monitored components themselves. Sometimes, it will be provided directly, like with  cache.getStatistics() providing StatisticsGateway for EhCache, or  sessionFactory.getStatistics() providing statistics for Hibernate SessionFactory, and so on.

And sometimes, this can occure by some other means, like managed beans. For example, using RuntimeMXBean for JVM  process.* metrics and  Tomcat mbeans (like GlobalRequestProcessor, Servlet, etc.) for  tomcat.* metrics.

To access these statistics and translate them to specific metrics, Micrometer introduces the concept of a MeterBinder.

Check the MeterBinder implementation hierarchy and you will learn more about what groups of metrics are available.

You can also check it directly in micrometer repo.

Open, for example, EhCache2Metrics, and you will find what and how Ehcache statistics are mapped to specific Micrometer metrics

cache.size -> StatisticsGateway:getSize
cache.gets{result=miss} -> StatisticsGateway:cacheMissCount
cache.gets{result=hit} -> StatisticsGateway:cacheHitCount
cache.puts -> StatisticsGateway:cachePutCount
cache.evictions -> StatisticsGateway:cacheEvictedCount
cache.remoteSize -> StatisticsGateway::getRemoteSize
cache.removals -> StatisticsGateway::cacheRemoveCount
cache.puts.added{result=added} -> StatisticsGateway::cachePutAddedCount
cache.puts.added{result=updated} -> StatisticsGateway::cachePutAddedCount
cache.misses{reason=expired} -> StatisticsGateway::cacheMissExpiredCount)
cache.misses{reason=notFound} -> StatisticsGateway::cacheMissNotFoundCount)
cache.xa.commits{result=readOnly} -> StatisticsGateway::xaCommitReadOnlyCount
cache.xa.commits{result=exception} -> StatisticsGateway::xaCommitExceptionCount
cache.xa.commits{result=committed} -> StatisticsGateway::xaCommitCommittedCount
cache.xa.rollbacks{result=exception} -> StatisticsGateway::xaRollbackExceptionCount
cache.xa.rollbacks{result=success} -> StatisticsGateway::xaRollbackSuccessCount
cache.xa.recoveries{result=nothing} -> StatisticsGateway::xaRecoveryNothingCount
cache.xa.recoveries{result=success} -> StatisticsGateway::xaRecoveryRecoveredCount
cache.local.offheap.size -> StatisticsGateway::getLocalOffHeapSize)
cache.local.heap.size -> StatisticsGateway::getLocalHeapSizeInBytes
cache.local.disk.size -> StatisticsGateway::getLocalDiskSizeInBytes

Registering MeterBinders is extremely easy, and examples can be found in the micrometer documentation.

Keep in mind, you can do it manually:

new ClassLoaderMetrics().bindTo(registry);
new JvmMemoryMetrics().bindTo(registry);
new EhCache2Metrics(cache, Tags.of("name", cache.getName())).bindTo(registry)
new TomcatMetrics(manager, tags).bindTo(registry)
...

Or, you can use Spring Boot, which will do it for you under the hood.

As I mentioned before actuator will provide many  AutoConfigurations and MetricsBinders that will register MeterBinders whenever a given dependency is added.

For example, TomcatMetricsBinder will register TomcatMetrics (for your embedded container).
MeterRegistryConfigurer will register JVM, uptime, and other system metrics.

Now, let’s say you want to use Ehcache in your app. You can add two dependencies:

    implementation("org.springframework.boot:spring-boot-starter-cache")
    implementation("net.sf.ehcache:ehcache")

Then, register the cache (you can also do it by ehcache.xml)

    @Bean
    Cache playCache(EhCacheCacheManager cacheManager) {
        CacheConfiguration cacheConfiguration = new CacheConfiguration()
            .name(CACHE_NAME)
            .maxEntriesLocalHeap(MAX_ELEMENTS_IN_MEMORY);
        Cache cache = new Cache(cacheConfiguration);
        cacheManager.getCacheManager().addCache(cache);
        cacheManager.initializeCaches();
        return cache;
    }

Now, CacheMetricsRegistrarConfiguration will register EhCache2Metrics for every cache managed by Spring Cache Managers.

If you don’t want to use Spring cache managers, you can also register EhCache2Metrics by yourself.

Now, launch the app and you will see additional ehcache metrics.

2019-07-17 13:08:45.113  INFO 93052 --- [trics-publisher] i.m.c.i.logging.LoggingMeterRegistry     : cache.gets{cache=playCache,cacheManager=cacheManager,name=playCache,result=hit} throughput=12.95/s
2019-07-17 13:08:45.124  INFO 93052 --- [       Thread-4] i.m.c.i.logging.LoggingMeterRegistry     : cache.misses{cache=playCache,cacheManager=cacheManager,name=playCache,reason=notFound} throughput=3.7/s
2019-07-17 13:08:45.124  INFO 93052 --- [trics-publisher] i.m.c.i.logging.LoggingMeterRegistry     : cache.gets{cache=playCache,cacheManager=cacheManager,name=playCache,result=miss} throughput=3.7/s
2019-07-17 13:08:48.840  INFO 93052 --- [       Thread-4] i.m.c.i.logging.LoggingMeterRegistry     : cache.puts{cache=playCache,cacheManager=cacheManager,name=playCache} throughput=16.65/s
2019-07-17 13:08:48.840  INFO 93052 --- [trics-publisher] i.m.c.i.logging.LoggingMeterRegistry     : cache.misses{cache=playCache,cacheManager=cacheManager,name=playCache,reason=notFound} throughput=3.7/s
2019-07-17 13:08:48.841  INFO 93052 --- [trics-publisher] i.m.c.i.logging.LoggingMeterRegistry     : cache.puts{cache=playCache,cacheManager=cacheManager,name=playCache} throughput=16.65/s
2019-07-17 13:08:48.841  INFO 93052 --- [       Thread-4] i.m.c.i.logging.LoggingMeterRegistry     : cache.puts.added{cache=playCache,cacheManager=cacheManager,name=playCache,result=updated} throughput=0.116667/s
2019-07-17 13:08:48.841  INFO 93052 --- [trics-publisher] i.m.c.i.logging.LoggingMeterRegistry     : cache.puts.added{cache=playCache,cacheManager=cacheManager,name=playCache,result=updated} throughput=0.116667/s
2019-07-17 13:08:48.841  INFO 93052 --- [       Thread-4] i.m.c.i.logging.LoggingMeterRegistry     : cache.puts.added{cache=playCache,cacheManager=cacheManager,name=playCache,result=added} throughput=0.116667/s
2019-07-17 13:08:48.842  INFO 93052 --- [trics-publisher] i.m.c.i.logging.LoggingMeterRegistry     : cache.puts.added{cache=playCache,cacheManager=cacheManager,name=playCache,result=added} throughput=0.116667/s
2019-07-17 13:08:48.847  INFO 93052 --- [trics-publisher] i.m.c.i.logging.LoggingMeterRegistry     : cache.local.disk.size{cache=playCache,cacheManager=cacheManager,name=playCache} value=0 B
2019-07-17 13:08:48.847  INFO 93052 --- [       Thread-4] i.m.c.i.logging.LoggingMeterRegistry     : cache.local.disk.size{cache=playCache,cacheManager=cacheManager,name=playCache} value=0 B
2019-07-17 13:08:48.908  INFO 93052 --- [       Thread-4] i.m.c.i.logging.LoggingMeterRegistry     : cache.local.heap.size{cache=playCache,cacheManager=cacheManager,name=playCache} value=1.039062 KiB
2019-07-17 13:08:48.908  INFO 93052 --- [trics-publisher] i.m.c.i.logging.LoggingMeterRegistry     : cache.local.heap.size{cache=playCache,cacheManager=cacheManager,name=playCache} value=1.039062 KiB
2019-07-17 13:08:48.909  INFO 93052 --- [trics-publisher] i.m.c.i.logging.LoggingMeterRegistry     : cache.local.offheap.size{cache=playCache,cacheManager=cacheManager,name=playCache} value=0 B
2019-07-17 13:08:48.909  INFO 93052 --- [       Thread-4] i.m.c.i.logging.LoggingMeterRegistry     : cache.local.offheap.size{cache=playCache,cacheManager=cacheManager,name=playCache} value=0 B
2019-07-17 13:08:48.909  INFO 93052 --- [       Thread-4] i.m.c.i.logging.LoggingMeterRegistry     : cache.remoteSize{} value=0
2019-07-17 13:08:48.909  INFO 93052 --- [trics-publisher] i.m.c.i.logging.LoggingMeterRegistry     : cache.remoteSize{} value=0
2019-07-17 13:08:48.909  INFO 93052 --- [       Thread-4] i.m.c.i.logging.LoggingMeterRegistry     : cache.size{cache=playCache,cacheManager=cacheManager,name=playCache} value=7
2019-07-17 13:08:48.909  INFO 93052 --- [trics-publisher] i.m.c.i.logging.LoggingMeterRegistry     : cache.size{cache=playCache,cacheManager=cacheManager,name=playCache} value=7

In this case, the responsibility of each component in the context of metrics can be summarized as:

You can check all these concepts in an example app provided here.

Happy coding!