Extra Micrometer Practices With Quarkus

Metrics emitted from applications might contain parameters (i.e., tags or labels) for which a specific metric is measured. Micrometer provides a facade over the instrumentation clients for many widespread monitoring systems like Atlas, Datadog, Graphite, Ganglia, Influx, JMX, and Prometheus. This article's scope explores some extra practices when combining Micrometer and Quarkus to tailor those metrics or give you some ideas of what you could extra measure.

For simplicity and given Prometheus popularity, the Micrometer Prometheus registry will be used as an example to demonstrate most of the use cases.

Prerequisites

• You can apply the concepts from this article by generating your project via https://code.quarkus.io/.
• JDK 11 installed with JAVA_HOME configured appropriately. 
• If interested in concepts about working with Quarkus and Micrometer, please take a look at https://quarkus.io/guides/micrometer.
• To further observe the metrics in Prometheus and creating Grafana panels, please consider provisioning those tools. However, you can run the code locally as well.

Global Tags Setup

By simply adding micrometer-registry-prometheus extension as a dependency to your Quarkus code, your application code will benefit from some Out Of The Box metrics, accessible on localhost via localhost:8080/q/metrics/q/metrics.

In case your application is deployed across multiple regions and the business logic inside it needs to consider specific details (like connecting to third-party services that offer data in different formats), you can add your own global tags to help further you inspecting performance statistics per region.  In the example below, the global metrics should be all properties prefixed with  global and defined separately in a class.

xxxxxxxxxx

@ConfigProperties(prefix = "global")

public class GlobalTagsConfig {

    public static final String PROFILE = "profile";

    public static final String REGION = "region";

    public String region;

    public String customMeter;

}

Furthermore, those configuration properties are defined in  application.properties file with a default value (for local development work), but the expectation is for them to be configured via environment variables passed from Docker or Kubernetes.

xxxxxxxxxx

global.region=${REGION:CEE}

global.custom-meter=${CUSTOM_METER:''}

The last step was to instruct the Micrometer extension about the CustomConfigurationand to use its  MeterFilter CDI beans when initializing MeterRegistry instances: 

x

@Singleton

public class CustomConfiguration {

    @Inject

    GlobalTagsConfig tagsConfig;

    @Produces

    @Singleton

    public MeterFilter configureAllRegistries() {

        return MeterFilter.commonTags(Arrays.asList(

                Tag.of(GlobalTagsConfig.PROFILE, ProfileManager.getActiveProfile()),

                Tag.of(GlobalTagsConfig.REGION, tagsConfig.region)));

    }

    @Produces

    @Singleton

    public MeterFilter configureMeterFromEnvironment() {

        return new MeterFilter() {

            @Override

            public Meter.Id map(Meter.Id id) {

                if(id.getName().startsWith(tagsConfig.customMeter)) {

                    return id.withName(tagsConfig.customMeter+"." + id.getName())

                            .withTag(new ImmutableTag(tagsConfig.customMeter+".tag", "value"));

                }

                return id;

            }

        };

    }

        };

}

A transform filter is used to add a name prefix and an additional tag conditionally to meters, starting with the runtime environment's name.

These metrics can help generate insights about how a particular part of your application performed for a region. In Grafana, you can display a graph of database calls via a specific function (tagged by   database_calls_find_total ) per region by the query:

xxxxxxxxxx

sum by(region)(rate(database_calls_find_total[5m]))

Measuring Database Calls 

If you are trying to measure the database calls, you may want to do the following:

• Add  @Counted annotation on top of the methods to check the number of invocations. 
• Add  @Timed annotation on top of the expected methods to run long-running operations and deserve a closer inspection.

In the example below, @Countedwas added to trace the number of invocations done:

x

    @Transactional

    @Counted(value = "database.calls.add", extraTags = {"db", "language", "db", "content"})

    public void createMessage(MessageDTO dto) {

        Message message = new Message();

        message.setLocale(dto.getLocale());

        message.setContent(dto.getContent());

        em.persist(message);

    }

This metric can further help in observing the distribution in a Grafana panel based on the following query:

xxxxxxxxxx

sum(increase(database_calls_add_total[1m]))

In the case of complex database queries or where multiple filters are applied, adding  @Timed  would give insights into those behaviors under the workload. In the example below, filtering messages by language is expected to be a long-running task:

    @Timed(value = "database.calls.filter", longTask = true, extraTags = {"db", "language"})

    public List<Message> filterMessages(Locale locale) {

        TypedQuery<Message> query = em.createQuery(

                "SELECT m from Message m WHERE m.locale = :locale", Message.class).

                setParameter("locale", locale);

        return query.getResultList();

    }

Measuring Performance of Requests

Depending on the complexity of the methods from your API and how you would like to measure their performance, you may want to do the following:

• Add @Countedannotation on top of the methods to check the number of invocations.
• Add @Timedannotation on top of the expected methods to run long-running operations and deserve a closer inspection.
• Measure code performance with complex business (present either inside those methods or in separate classes), you might want to consider a dynamic tagging strategy.

The simplest scenario for an API method is to be straightforward in its behavior. In such a case, applying the @Counted and/or @Timed should be enough, and of course, more tags can be added, like in the example below:

xxxxxxxxxx

    @GET

    @Path("find")

    @Produces(MediaType.TEXT_PLAIN)

    @Timed(value = "greetings.all", longTask = true, extraTags = {URI, API_GREET})

    @Counted(value = "http.get.requests", extraTags = {URI, API_GREET})

    public List<Message> findAll() {

        return messageService.findAll();

    }

Yet, implementations with more business logic would need separate metrics with more tags. The goal in such cases is to reuse some metrics and decorate them with more tags. For example, the method below should have additional metrics attached to it to measure the performance of each greeting handled in a given language:

xxxxxxxxxx

    @GET

    @Path("find/{content}")

    @Produces(MediaType.TEXT_PLAIN)

    @Timed(value = "greetings.specific", longTask = true, extraTags = {URI, API_GREET})

    @Counted(value = "http.get.specific.requests", extraTags = {URI, API_GREET})

    public List<Message> findGreetings(@PathParam("content") String content) {

        AtomicReference<List<Message>> messages = new AtomicReference<>();

        if (!content.isEmpty()) {

            List<Message> greetings = messageService.findMessages(content);

            messages.set(greetings);

        }

        return messages.get();

    }

For the previous API request, the average greeting retrieved over time can be measured in a Grafana panel through the query:

x

avg(increase(greetings_specific_seconds_duration_sum[1m])) without(class, endpoint, pod,instance,job,namespace,method,service,exception,uri)

But the above method can also get no result for the given content, and that case should be closely observed. For this extra case, a custom Counter  was implemented were firstly the existence of a  Counter with similar features is checked and if not, increment a new instance of  Counter with additional tags attached:

xxxxxxxxxx

public class DynamicTaggedCounter extends CommonMetricDetails {

    private final String tagName;

    public DynamicTaggedCounter(String identifier, String tagName, MeterRegistry registry) {

        super(identifier, registry);

        this.tagName = tagName;

    }

    public void increment(String tagValue) {

        Counter counter = registry.counter(identifier, tagName, tagValue);

        counter.getId().withTag(new ImmutableTag(tagName, tagValue));

        counter.increment();

    }

}

Note: Similar customization was be applied for DynamicTaggedTimer.

In the  ExampleEndpoint, the following changes were added:

xxxxxxxxxx

@Path("/api/v1")

public class ExampleEndpoint {

    @Inject

    protected PrometheusMeterRegistry registry;

    private DynamicTaggedCounter dynamicTaggedCounter;

    @PostConstruct

    protected void init() {

        this.dynamicTaggedCounter = new DynamicTaggedCounter("another.requests.count", CUSTOM_API_GREET, registry);

    }

    @GET

    @Path("find/{content}")

    @Produces(MediaType.TEXT_PLAIN)

    @Timed(value = "greetings.specific", longTask = true, extraTags = {URI, API_GREET})

    @Counted(value = "http.get.specific.requests", extraTags = {URI, API_GREET})

    public List<Message> findGreetings(@PathParam("content") String content) {

        AtomicReference<List<Message>> messages = new AtomicReference<>();

        if (!content.isEmpty()) {

            List<Message> greetings = messageService.findMessages(content);

            messages.set(greetings);

        }

        if (messages.get().size() > 0) {

            dynamicTaggedCounter.increment(content);

        } else {

            dynamicTaggedCounter.increment(EMPTY);

        }

        return messages.get();

    }

}

Based on the above implementation, the ratio of empty results requests can be  outlined in Grafana via the query:

xxxxxxxxxx

sum(increase(another_requests_count_total{custom_api_greet="empty"}[5m]))/sum(increase(another_requests_count_total[5m]))

When multiple tags and values need to be added per action, the previous implementation for DynamicTaggedCounter got additional customizations: DynamicMultiTaggedCounterand DynamicMultiTaggedTimer. The DynamicMultiTaggedTimer below checks, if there is an existing Timer a with the same tags and values, and if it returns a new instance of a Timer with additional tags attached:

xxxxxxxxxx

public class DynamicMultiTaggedTimer extends CommonMetricDetails {

    protected List<String> tagNames;

    public DynamicMultiTaggedTimer(String name, MeterRegistry registry, String... tags) {

        super(name, registry);

        this.tagNames = Arrays.asList(tags.clone());

    }

    public Timer decorate(String ... tagValues) {

        List<String> adaptedValues = Arrays.asList(tagValues);

        if(adaptedValues.size() != tagNames.size()) {

            throw new IllegalArgumentException("Timer tag values mismatch the tag names!"+ 

                                               "Expected args are " + tagNames.toString() +

                                               ", provided tags are " + adaptedValues);

        }

        int size = tagNames.size();

        List<Tag> tags = new ArrayList<>(size);

        for(int i = 0; i<size; i++) {

            tags.add(new ImmutableTag(tagNames.get(i), tagValues[i]));

        }

        Timer timer = registry.timer(identifier, tags);

        timer.getId().withTags(tags);

        return timer;

    }

}

The above-implementation is used below to determine the creation of exceptional content of messages in a different language: 

xxxxxxxxxx

    @PUT

    @Path("make/{languageTag}/{content}")

    @Consumes(MediaType.TEXT_PLAIN)

    @Produces(MediaType.TEXT_PLAIN)

    @Timed(value = "greeting.generator", extraTags = {URI, API_GREET})

    @Counted(value = "http.put.requests", extraTags = {URI, API_GREET})

    public String generateGreeting(@PathParam("languageTag") String languageTag, @PathParam("content") String content) {

        Locale locale = Locale.forLanguageTag(languageTag);

        MessageDTO dto = new MessageDTO(locale, content);

        if (Math.random() > 0.8) {

            String exceptionalTag = "exceptional" + content;

            dynamicMultiTaggedTimer.decorate(languageTag, exceptionalTag).record(() -> {

                try {

                    Thread.sleep(1 + (long)(Math.random()*500));

                } catch (InterruptedException e) {

                    LOGGER.error("Error occured during long running operation ", e);

                }

            });

            dynamicMultiTaggedCounter.increment(languageTag, exceptionalTag);

        } else {

            dynamicMultiTaggedCounter.increment(languageTag, content);

        }

        messageService.createMessage(dto);

        return content;

    }

A Grafana panel can be created to some exceptional messages by language and content:

xxxxxxxxxx

sum by (language, custom_api_greet)(increase(other_requests_total{custom_api_greet=~"exceptional.+"}[1m]))

Final Thoughts

For sure, you can do many more tweaks with Micrometer, yet I hope this piqued your interest in Quarkus and Micrometer. The code is available at https://github.com/ammbra/micrometering-with-quarkus.