Monitor Applications Using Prometheus Operator on Kubernetes

You can make the Prometheus configuration aware of the Kubernetes environment your applications are running in. I’ve described how to do that manually, in a previous blog post. Prometheus Operator is an extension to Kubernetes that manages Prometheus monitoring instances in a more automated and effective way.

Prometheus Operator allows you to define and manage monitoring instances as Kubernetes resources. If you know how to manage Kubernetes, there’s a low threshold to get started and effectively define the monitoring of your applications.

In order to enable our Kubernetes for Prometheus operators, we set up the resource and RBAC definitions that you can find here. This enhances our cluster with more Kubernetes resources types, such as ServiceMonitor, or Prometheus. Similarly, you can use the Prometheus Operator helm chart.

We define the operators of our config-example application, similar to the previous post:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: config-example
  labels:
    team: example
spec:
  selector:
    matchLabels:
      app: config-example
  endpoints:
  - basicAuth:
      password:
        name: basic-auth
        key: password
      username:
        name: basic-auth
        key: username
    port: https
    scheme: https
    path: '/metrics/'
    tlsConfig:
      insecureSkipVerify: true

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
spec:
  serviceAccountName: prometheus
  serviceMonitorSelector:
    matchLabels:
      team: example
  resources:
    requests:
      memory: 400Mi

apiVersion: v1
kind: Service
metadata:
  name: prometheus
spec:
  ports:
  - port: 9090
    name: http
  selector:
    prometheus: prometheus

apiVersion: v1
kind: Secret
metadata:
  name: basic-auth
data:
  password: YWRtaW5hZG1pbg==
  username: YWRtaW4=

This sets up a Prometheus instance that will scrape applications that are deployed with the app: config-example label using the provided configuration to access it. It also creates a prometheusservice to access the monitoring instances.

You can find a full description of the Prometheus Operator API in the documentation.

After we applied all resources, we can see the running monitoring instances in our cluster:

$> kubectl get pods
NAME                                   READY   STATUS    RESTARTS   AGE
config-example-7db586bb95-jdmsx        1/1     Running   0          12m
config-example-7db586bb95-z4ln8        1/1     Running   0          12m
[...]
prometheus-prometheus-0                3/3     Running   0          14m

This enables us to simply monitor all application instances without manually configuring the Prometheus instances.

Have a look at the full example on GitHub (deployment/ directory).