{{announcement.body}}
{{announcement.title}}

Spring Boot Metrics with Dynamic Tag Values

DZone 's Guide to

Spring Boot Metrics with Dynamic Tag Values

Tweak around with the new Spring Boot Micrometer to get a semi-dynamic tagged metrics.

· Java Zone ·
Free Resource

Metrics are essential tools for every scalable application.

Spring Boot 2.0 introduced a new level of metrics with the Micrometer library, making integration of our code with monitoring systems like Prometheus and Graphite simpler than ever.

One of the features we found missing is dynamic tag values. Tag name and value are declared at counter creation. In fact tag values are treated as a name decorator: tags with the same name and tag name, but different tag values are two different standalone counters. 

Java
 




xxxxxxxxxx
1


 
1
Counter counterWithTag1 = Counter.builder(name).tags(tagName, tagValue1).register(registry);
2
Counter counterWithTag2 = Counter.builder(name).tags(tagName, tagValue2).register(registry);


Our system is multi-tenant: it serves messages from multiple customers. We want to know the rate of messages we process per customer. The list of names of the active customers is dynamic, as they connect and disconnect. We do not want to hold a counter for a customer that exists in the system but does not send data, we do not want to sync our counters with a db holding the list of customers to detect newly added customers. We want to have a counter only if and when a customer is sending data.

For this case (and many others) we implemented the TaggedCounter

Java
 




xxxxxxxxxx
1
27


 
1
import io.micrometer.core.instrument.Counter;
2
import io.micrometer.core.instrument.MeterRegistry;
3
 
          
4
import java.util.HashMap;
5
import java.util.Map;
6
 
          
7
public class TaggedCounter {
8
    private String name;
9
    private String tagName;
10
    private MeterRegistry registry;
11
    private Map<String, Counter> counters = new HashMap<>();
12
 
          
13
    public TaggedCounter(String name, String tagName, MeterRegistry registry) {
14
        this.name = name;
15
        this.tagName = tagName;
16
        this.registry = registry;
17
    }
18
 
          
19
    public void increment(String tagValue){
20
        Counter counter = counters.get(tagValue);
21
        if(counter == null) {
22
            counter = Counter.builder(name).tags(tagName, tagValue).register(registry);
23
            counters.put(tagValue, counter);
24
        }
25
        counter.increment();
26
    }
27
}


In our message receiver constructor, we create a TaggedCounter, noting the name of the counter and the name of the tag. No values. Note that MeterRegistry can be autowired — you do not have to explicitly create it.

Java
 




x


 
1
private TaggedCounter perCustomerMessages;
2
 
          
3
@Autowired
4
public ReaderMessageReceiver(MeterRegistry meterRegistry) {
5
    this.perCustomerMessages = new TaggedCounter("per-customer-messages", "customer", meterRegistry);
6
}


Later on, when a message is received (in this case, a pubsub message), we increment the counter, now noting the tag value.

Java
 




xxxxxxxxxx
1


 
1
public void receiveMessage(PubsubMessage pubsubMessage, AckReplyConsumer ackReplyConsumer) {
2
        String customer = pubsubMessage.getAttributesMap().get("customer_id");
3
        perCustomerMessages.increment(customer);
4
    try {
5
      //process the message...
6
    } finally {
7
        ackReplyConsumer.ack();
8
    }
9
}


And that's it. Our monitoring server is Prometheus, this is what we get in /actuator/prometheus endpoint (note that all delimiters in counter name or tag name are replaced with underscores):

C#
 




xxxxxxxxxx
1


 
1
# TYPE per_customer_messages_total counter
2
per_customer_messages_total{customer="0f43e152",} 1291460.0
3
per_customer_messages_total{customer="93c2adbb",} 118899.0
4
per_customer_messages_total{customer="1eab1589",} 301311.0
5
per_customer_messages_total{customer="270e5ca0",} 1710188.0


In Grafana we can see a graph of messages per customer by the query

CSS

With {{customer}} in the Legend format field, we get the graph we wanted: a line per customer.

Another use case is processing different types of messages. Each type has its own processing time, and we want to measure by a timer the processing latency per type. (Although the message types are predefined in the system, some of them are not in use: deprecated or just rarely used. We want the counters to be triggered only if used.)

So we have a similar class of TaggedTimer:

Java
 




xxxxxxxxxx
1
28


 
1
import io.micrometer.core.instrument.MeterRegistry;
2
import io.micrometer.core.instrument.Timer;
3
 
          
4
import java.util.HashMap;
5
import java.util.Map;
6
 
          
7
public class TaggedTimer {
8
    private String name;
9
    private String tagName;
10
    private MeterRegistry registry;
11
    private Map<String, Timer> timers = new HashMap<>();
12
 
          
13
    public TaggedTimer(String name, String tagName, MeterRegistry registry) {
14
        this.name = name;
15
        this.tagName = tagName;
16
        this.registry = registry;
17
    }
18
 
          
19
    public Timer getTimer(String tagValue){
20
        Timer timer = timers.get(tagValue);
21
        if(timer == null) {
22
            timer = Timer.builder(name).tags(tagName, tagValue).register(registry);
23
            timers.put(tagValue, timer);
24
        }
25
        return timer;
26
    }
27
 
          
28
}


Usage is similar: Create with name and tag name in constructor
Java
 




xxxxxxxxxx
1


1
TaggedTimer perTypeTimer = new TaggedTimer("per-type-processing-timer", "message-type", meterRegistry);


And then use it, in this case for the "heartbeat" message type, by

Java
 




x
2


 
1
perTypeTimer.getTimer("heartbeat").record(() -> {
2
    //process the message...
3
});


In Grafana, we display the process latency per message type by

C#

(We did not create a tagged Gauge as we did not need it, but one can be created in the same manner.)

The above TaggedCounter and TaggedTimer are solving the most frequent case of a counter with a single tag. But what if you need to count by multiple tags? For example, in order to profile the functional difference between our customers, we wanted to get the rate of messages of specific type per customer.

For this, we created the MultiTaggedCounter:

Java
 




xxxxxxxxxx
1
38


1
import io.micrometer.core.instrument.Counter;
2
import io.micrometer.core.instrument.ImmutableTag;
3
import io.micrometer.core.instrument.MeterRegistry;
4
import io.micrometer.core.instrument.Tag;
5
 
          
6
import java.util.*;
7
 
          
8
public class MultiTaggedCounter {
9
    private String name;
10
    private String[] tagNames;
11
    private MeterRegistry registry;
12
    private Map<String, Counter> counters = new HashMap<>();
13
 
          
14
    public MultiTaggedCounter(String name, MeterRegistry registry, String ... tags) {
15
        this.name = name;
16
        this.tagNames = tags;
17
        this.registry = registry;
18
    }
19
 
          
20
    public void increment(String ... tagValues){
21
        String valuesString = Arrays.toString(tagValues);
22
        if(tagValues.length != tagNames.length) {
23
            throw new IllegalArgumentException("Counter tags mismatch! Expected args are "+Arrays.toString(tagNames)+", provided tags are "+valuesString);
24
        }
25
        Counter counter = counters.get(valuesString);
26
        if(counter == null) {
27
            List<Tag> tags = new ArrayList<>(tagNames.length);
28
            for(int i = 0; i<tagNames.length; i++) {
29
                tags.add(new ImmutableTag(tagNames[i], tagValues[i]));
30
            }
31
            counter = Counter.builder(name).tags(tags).register(registry);
32
            counters.put(valuesString, counter);
33
        }
34
        counter.increment();
35
    }
36
 
          
37
}
38
 
          


When building the counter we note only the tag names

Java
 




x


1
MultiTaggedCounter perCustomerPerTypeMessages = new MultiTaggedCounter("per-customer-per-type", meterRegistry, "customer", "message-type");
2
 
          


And then using the counter, we increment noting the tag values

Java
 




xxxxxxxxxx
1


 
1
perCustomerPerTypeMessages.increment(message.getCustomerName(), message.getType());


In Grafana we sum by both customer and message_type tags:

C#

(Legend format {{customer}}-{{message_type}} is the short notation of that values in the legend)

We can also get the data for specific customer or specific type by

Java
 




xxxxxxxxxx
1


 
1
sum by (message_type)(rate(per_customer_per_type_total{customer="89450f"}[1m]))
2
 
          


Hope you found this helpful!

Topics:
dynamic metrics, java, metrics, microservice best practices, monitoring and performance, spring boot

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}