Prometheus Metrics for MicroProfile Microservices in Istio

for cloud-native applications, kubernetes and istio deliver a lot of important functionality out of the box, for example, certain metrics , which are independent from the actual service implementations, can be collected automatically.

however, other metrics are application specific, for example, how many times certain methods have been invoked and how long they take. eclipse microprofile allows java developers to easily expose these metrics . check out the list of available java annotations .

the application metrics can be displayed in prometheus which comes with istio. other istio metrics can be displayed in the prometheus dashboard as well.

i've implemented a little sample that shows how to generate metrics from java-based microservices and how to access them in the prometheus dashboard.

get the code from github.

microprofile

in the microprofile code , java annotations can be used. in this example, the amount of times the '/getmultiple' endpoint is invoked and the duration of this method are measured.

@timed(name = "getarticlestimed",
  absolute = true,
  displayname = "web-api /getmultiple timer",
  description = "time taken by com.ibm.webapi.apis.getarticles.getarticles")
@counted(name = "getarticlescounted",
  absolute = true,
  displayname = "web-api /getmultiple count",
  description = "number of times com.ibm.webapi.apis.getarticles.getarticles has been invoked",
  monotonic = true)
@metered(name = "getarticlesmetered",
  displayname = "web-api /getmultiple frequency",
  description = "rate the throughput of com.ibm.webapi.apis.getarticles.getarticles")
@get
@path("/getmultiple")

in order to use these annotations in java code, i've defined the dependencies in pom.xml . to keep it simple, i use the two 'umbrella' features, javaee-api and microprofile.

<dependencies>
  <dependency>
    <groupid>javax</groupid>
    <artifactid>javaee-api</artifactid>
    <version>8.0</version>
    <scope>provided</scope>
  </dependency>
  <dependency>
    <groupid>org.eclipse.microprofile</groupid>
    <artifactid>microprofile</artifactid>
    <version>2.1</version>
    <scope>provided</scope>
    <type>pom</type>
  </dependency>
</dependencies>

in order to run the microprofile service on kubernetes, you need a docker image . check out my previous article on how to dockerize microprofile services .

next, in the server.xml file, a couple of settings need to be made. you need to define the monitor feature and disable authentication. i have read in several articles that https is required, but, for me, http works just as well. to keep it as simple as possible, i also disabled authentication. however, in a production application, you'd use https and require authentication.

<?xml version="1.0" encoding="utf-8"?>
<server description="openliberty server">
  <featuremanager>
    <feature>webprofile-8.0</feature>
    <feature>microprofile-2.1</feature>
    <feature>usr:opentracingzipkin-0.31</feature>    
    <feature>monitor-1.0</feature> 
  </featuremanager>
  <httpendpoint id="defaulthttpendpoint" host="*" httpport="9080" httpsport="9443"/>
  <mpmetrics authentication="false"/>    
</server>

in order to see the metrics, you can trigger endpoints like http://192.168.99.100:31223/metrics/application . note that you have to replace the ip address with the ip address of your cluster and the port with the port of your service.

prometheus

in order to see the metrics in prometheus in istio, some additional configuration needs to be made.

first, you need to use an annotation like "webapi: 'true'" in your service definition. you can choose whatever name you like.

kind: service
apiversion: v1
metadata:
  annotations:
    webapi: "true"
  name: web-api
  labels:
    app: web-api
spec:
  selector:
    app: web-api
  ports:
    - port: 9080
      name: http
  type: nodeport

in order to point prometheus to the metrics endpoint of your service, you need to change the configmap of prometheus. the following snippet shows the simplest possible configuration (http and anonymous). note that the job_name 'webapi' needs to match with the annotation of your service above.

- job_name: 'webapi'
  scheme: 'http'
  tls_config:
    insecure_skip_verify: true
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_webapi]
    action: keep
    regex: true

there are different ways to update prometheus with this configuration. since my goal is to provide starter code for cloud-native applications, i've written a little script to automate this process. the prometheus configuration is read, then the additional configuration is merged into the original configuration and at last the new configuration is applied.

kubectl get configmap prometheus -n istio-system -oyaml > ${root_folder}/istio/prometheus-config-org.yaml
sed -e '/kind: configmap/r ./prometheus-config.yaml' -e '/kind: configmap/d' ${root_folder}/istio/prometheus-config-org.yaml > ${root_folder}/istio/prometheus-config-new.yaml
kubectl replace --force -f ${root_folder}/istio/prometheus-config-new.yaml
pod=$(kubectl get pods -n istio-system | grep prometheus | awk ' {print $1} ')
kubectl delete pod $pod -n istio-system

this image shows the prometheus dashboard displaying the amount of times '/getmultiple' has been invoked.

this image shows the rate per second:

if you want to try this yourself, follow the instructions in the readme of cloud-native-starter .