Utilize OpenShift to Manage External Services Metrics
Openshift 4 has great metrics monitor suites used to monitor and create dashboards for infra and application layer services; we can leverage that to manage external services and infra as well.
Join the DZone community and get the full member experience.Join For Free
Openshift 4 provided great metrics monitor suites, we can use them to monitor and create dashboards for both infra and application layer services; What's more, we can leverage that to manage external services and infra as well.
The benefits why I want to address this issue are:
- Establish and maintaining a sophisticated monitoring solution can be very hard and time-consuming.
- The monitoring components are almost coming free with embedded in Openshift. They are managed by the Openshift Cluster Monitoring operator which is stable, self-healing, and well-structured.
- Manage and scale infra configuration like monitoring components in OpenShift is easier than doing it in VM because of its declarative nature and we can even manage them by GitOps.
- Of course, if you have both the OpenShift Platform to manage the container world and the VM world to manage other businesses, you don't have to manage two complicated Monitoring solutions for time and cost-saving.
In this article I'm going to cover the following topic in order to demonstrate how to monitor external services on OpenShift Monitoring Component:
- Exporting Quarkus applications via MicroProfile metrics.
- Configuring OpenShift to enable user-defined monitoring components.
- Proxy external service in Openshift via Kubernetes service.
- Create My Grafana Instance to host application dashboard for external service metrics.
- (Optional) Utilize node exporter to monitor external infra metrics.
Although the issue I am addressing is targeted to external services, you can surely apply the same steps for your applications hosted in OpenShift.
Exporting Quarkus Applications by Micro Profile Metrics
In this lab I'm going to use the Quarkus framework to demonstrate the application metric monitoring, the application example is a quarkus-todo-application in my GitHub repo.
Quarkus application supports 2 ways to export metrics:
- MicroProfile metrics.
- Micrometer metrics.
I'm using the MicroProfile metric.
The key part of exporting metrics in a quarkus application are:
Build and Deploy
If you are familiar with build and dev concepts, you can directly get a binary image in my quay.io repo.
Here are the steps to deploy the Todo service on a virtual machine.
My target VM IP is
192.168.2.10 OS: rhel7
#Execute on my laptop mvn quarkus:dev
#Test my metrics is actual exporting curl http://localhost:8080/metrics|grep Prime application_io_quarkus_sample_PrimeNumberChecker_checksTimer_rate_per_second gauge
#Build container images and push into a remote repo, you need to change to your accessible repo podman build -f src/main/docker/Dockerfile.ubi . -t quay.io/rzhang/quarkus:todo-app-jvm-11-nodb
# log into my target virtual machine 192.168.2.10 ssh 192.168.2.10 podman run -d --name todo-app -p 8080:8080 quay.io/rzhang/quarkus:todo-app-jvm-11-nodb
curl http://192.168.2.10:8080/metrics|grep Prime application_io_quarkus_sample_PrimeNumberChecker_checksTimer_rate_per_second gauge
OK, the application is deployed in a VM and metrics have been exported to: http://192.168.2.10:8080/metrics
Let's see how to configure OpenShift to collect these metrics.
Configuring OpenShift to Enable User-defined Monitoring Components
First, let's have a briefing on how Monitoring components are organized in OpenShift.
User-Defined Project's metric would be managed by the right part of Monitoring components (i.e.,
On the left side, all OpenShift Infra-related metrics would be managed by
openshift-monitoring namespaces, so they are well separated and have different storage and components for Platform and User-defined.
There are few components that are shared between Platform and User-Defined monitoring components.
Thanos Querier — Aggregates both Platform metrics and User-Defined and provide a unique query interface for searching metrics; In the next part of Grafana configuration, we would use the Thanos Querier URL to display the metrics;
AlertManager — Receiving alert from both sides of Prometheus and sending alert to the external system;
Cluster Monitoring Operator - Provides easy install and maintenance of monitoring components;
The left part is installed out of the box if you installed OpenShift 4.
However, the right part would not be installed by default. We would utilize the User-defined monitoring components to manage the external services metrics. And the steps are the same if you want to manage your own application metrics which run in OpenShift instead of outside.
To enable user-workload-monitoring:
oc apply -f cluster-monitoring-config.yaml
After enabling this, you can see that a new namespace and related monitoring component would be created in these new namespaces:
>oc projects|grep monitoring
>oc get po -n openshift-user-workload-monitoring
NAME READY STATUS RESTARTS AGE
prometheus-operator-6bf7fbbbdd-m8fsl 2/2 Running 0 9d
prometheus-user-workload-0 5/5 Running 0 23d
prometheus-user-workload-1 5/5 Running 0 23d
thanos-ruler-user-workload-0 3/3 Running 1 23d
thanos-ruler-user-workload-1 3/3 Running 1 23d
OK, now Openshift User-Defined Monitoring components have been ready.
Let's move on and create a new namespace to host a proxy to external services metrics.
Proxy External Service in OpenShift via Kubernetes Service
For readers who are familiar with the following concepts, you can just check my lab code at https://github.com/ryanzhang/openshift-monitor-external-service.
1. Create a new project to host my proxy and metrics collecting configuration for external metrics and configuration.
>oc new-project external-service
2. Create a Kubernetes service to integrate external service which points to external metric rest endpoints.
Now you can monitor the external metric in the embedded dashboard:
First trigger some load via ab test tools:
Search metrics in OpenShift integrated UI:
Create Grafana Instance to Display External Service Metrics
Wouldn't it be great if we can view our user-defined metric in Grafana?
Although OpenShift has installed Grafana out of the box, they are not supposed to be used by user-defined workloads. Currently, there is no way to edit and add a user-defined dashboard or create more user accounts for the default Grafana instance.
So let me show you how to install a user-defined Grafana to integrate our own metrics.
Install the Grafana Operator Into
oc new-project my-grafana
2. Create a
my-grafana instance to host my metrics dashboards:
Now you can access
my-grafana instance via: https://my-grafana.apps-crc.testing/
Login in with admin/admin.
Add my todo-app quarkus MicroProfile Metrics dashboard.
To find more information on how to generate MicroProfile Metrics for your application, I recommend you to check this: https://github.com/jamesfalkner/microprofile-grafana
Here we go:
(I also trigger a load test at the backend for monitoring the metrics.)
What's more, you could monitor external infra metrics by utilizing the
(Optional) Utilize Node Exporter to Monitor External Infra Metrics
Please follow Prometheus official documentation to install node_exporter to your VM or bare metal, then you would have the metrics available at, for example in my case:
Repeat the above steps, or take a look at the YAML resource in my GitHub repo:
You would get the external infra metrics managed in user-defined monitoring components.
Here is the node_exporter dashboard I used in the following graph.
OpenShift monitoring components are easy to install and operate. I hope it's useful to show you that we can not only integrate the container-based workload's metrics inside OpenShift but also external services and infra metrics outside OpenShift.
Opinions expressed by DZone contributors are their own.