Getting Jaeger’s Java Client Internal Metrics Into Prometheus

We recently integrated Micrometer into the internal metrics collection mechanism for the Jaeger Java Client, making it easier to get them into Prometheus.

The Jaeger Java Client, like other Jaeger clients, already had an internal metrics mechanism for collecting data such as “number of spans started”. With the support for Micrometer included in the client version 0.25.0, it’s now easier to get this data fed into a range of backend metrics platforms supported by Micrometer, like Prometheus, JMX and/or StatsD, among others.

To demonstrate this feature, we developed a simple Vert.x application that would just accept a request on a given port, create a span, write back a “Hello” message and finish the span.

Our example makes use of the Prometheus Registry as the concrete backend for Micrometer, available under the artifact coordinates io.micrometer:micrometer-registry-prometheus.

package io.vertx.starter;

import com.uber.jaeger.Configuration;
import com.uber.jaeger.micrometer.MicrometerMetricsFactory;
import com.uber.jaeger.samplers.ConstSampler;
import io.micrometer.core.instrument.Metrics;
import io.micrometer.prometheus.PrometheusConfig;
import io.micrometer.prometheus.PrometheusMeterRegistry;
import io.opentracing.Span;
import io.opentracing.Tracer;
import io.vertx.core.AbstractVerticle;
import io.vertx.core.Vertx;

public class MainVerticle extends AbstractVerticle {

  public static void main(String[] args) {
      Vertx vertx = Vertx.vertx();
      vertx.deployVerticle(new MainVerticle());
  }

  @Override
  public void start() {
    MicrometerMetricsFactory metricsFactory = new MicrometerMetricsFactory();
    Configuration configuration = new Configuration("jaeger-client-java-tester");
    Tracer tracer = configuration
        .withReporter(
            new Configuration.ReporterConfiguration()
                .withLogSpans(true)
        )
        .withSampler(
            new Configuration.SamplerConfiguration()
                .withType(ConstSampler.TYPE)
                .withParam(1)
        )
        .getTracerBuilder()
        .withMetricsFactory(metricsFactory)
        .build();

    vertx.createHttpServer()
        .requestHandler(req -> {
          Span span = tracer.buildSpan("new-request").start();
          req.response().end("Hello Vert.x!");
          span.finish();
        })
        .listen(8080);

    PrometheusMeterRegistry registry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
    Metrics.addRegistry(registry);

    vertx.createHttpServer()
      .requestHandler(req -> req.response().end(registry.scrape()))
      .listen(8081);

  }

}

For reference, this is the build.gradle file for this project. It’s pretty much the same as the starter project for Vert.x, with the addition of Jaeger and Micrometer libraries.

plugins {
  id 'application'
  id 'com.github.johnrengelman.shadow' version '2.0.1'
}

repositories {
  mavenLocal()
  jcenter()
}

version = '1.0-SNAPSHOT'
sourceCompatibility = '1.8'
mainClassName = 'io.vertx.core.Launcher'

def vertxVersion = '3.5.0'
def mainVerticleName = 'io.vertx.starter.MainVerticle'
def watchForChange = 'src/**/*'
def doOnChange = './gradlew classes'

dependencies {
  compile "com.uber.jaeger:jaeger-core:0.25.0"
  compile "com.uber.jaeger:jaeger-micrometer:0.25.0"
  compile 'io.micrometer:micrometer-registry-prometheus:latest.release'
  compile group: 'org.apache.logging.log4j', name: 'log4j-slf4j-impl', version: '2.10.0'

  compile "io.vertx:vertx-core:$vertxVersion"
  testCompile "junit:junit:4.12"
  testCompile "io.vertx:vertx-unit:$vertxVersion"
}

shadowJar {
  classifier = 'fat'
  manifest {
      attributes "Main-Verticle": mainVerticleName
  }
  mergeServiceFiles {
    include 'META-INF/services/io.vertx.core.spi.VerticleFactory'
  }
}

run {
  args = ['run', mainVerticleName, "--redeploy=$watchForChange", "--launcher-class=$mainClassName", "--on-redeploy=$doOnChange"]
}

task wrapper(type: Wrapper) {
  gradleVersion = '4.3.1'
}

Run the example above with ./gradlew run and it should print out a log entry like this:

13:45:47.634 [vert.x-eventloop-thread-0] WARN  io.vertx.starter.MainVerticle - Registered tracer: GlobalTracer{Tracer(...)}
INFO: Succeeded in deploying verticle

After that, just hit the port 8080to create spans:

$ curl http://localhost:8080
Hello Vert.x!

The metrics collected are exposed to Prometheus on port 8081:

$ curl http://localhost:8081
# HELP jaeger:sampler_updates_total 
# TYPE jaeger:sampler_updates_total counter
jaeger:sampler_updates_total{result=”err”,} 0.0
jaeger:sampler_updates_total{result=”ok”,} 0.0
# HELP jaeger:baggage_restrictions_updates_total 
# TYPE jaeger:baggage_restrictions_updates_total counter
jaeger:baggage_restrictions_updates_total{result=”err”,} 0.0
jaeger:baggage_restrictions_updates_total{result=”ok”,} 0.0
# HELP jaeger:span_context_decoding_errors_total 
# TYPE jaeger:span_context_decoding_errors_total counter
jaeger:span_context_decoding_errors_total 0.0
# HELP jaeger:sampler_queries_total 
# TYPE jaeger:sampler_queries_total counter
jaeger:sampler_queries_total{result=”ok”,} 0.0
jaeger:sampler_queries_total{result=”err”,} 0.0
# HELP jaeger:baggage_updates_total 
# TYPE jaeger:baggage_updates_total counter
jaeger:baggage_updates_total{result=”err”,} 0.0
jaeger:baggage_updates_total{result=”ok”,} 0.0
# HELP jaeger:baggage_truncations_total 
# TYPE jaeger:baggage_truncations_total counter
jaeger:baggage_truncations_total 0.0
# HELP jaeger:traces_total 
# TYPE jaeger:traces_total counter
jaeger:traces_total{sampled=”n”,state=”started”,} 0.0
jaeger:traces_total{sampled=”n”,state=”joined”,} 0.0
jaeger:traces_total{sampled=”y”,state=”started”,} 3.0
jaeger:traces_total{sampled=”y”,state=”joined”,} 0.0
# HELP jaeger:reporter_spans_total 
# TYPE jaeger:reporter_spans_total counter
jaeger:reporter_spans_total{result=”dropped”,} 0.0
jaeger:reporter_spans_total{result=”err”,} 0.0
jaeger:reporter_spans_total{result=”ok”,} 0.0
# HELP jaeger:started_spans_total 
# TYPE jaeger:started_spans_total counter
jaeger:started_spans_total{sampled=”y”,} 3.0
jaeger:started_spans_total{sampled=”n”,} 0.0
# HELP jaeger:finished_spans_total 
# TYPE jaeger:finished_spans_total counter
jaeger:finished_spans_total 3.0

At this point, you just need to get Prometheus to scrape the port 8081. A minimalist configuration file could look like this:

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

scrape_configs:
  - job_name: 'app-using-jaeger-java-client'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:8081']

Then, start Prometheus using this file:

$ prometheus --config.file=/tmp/prometheus.yml

Prometheus should be available at http://0.0.0.0:9090 , ready to display the metrics reported by the Jaeger Client Java.

Summary

In this article, we’ve seen that it’s now very easy to get the internal metrics from the Jaeger Client for Java published by a backend supported by Micrometer, like Prometheus. Give it a try and don’t forget to give us your feedback!