Microservices With Observability on Kubernetes
A Complete Guide to deployment, logging, distributed tracing, performance, and metrics monitoring including cluster health
Join the DZone community and get the full member experience.Join For Free
Are you looking to implement observable microservices but clueless as to how to make them work with Kubernetes? Well, well… This could be the blog you have been searching for.
First, let us understand what observability is. This term originated in control system engineering and was defined as “a measure of how well internal states of a system can be inferred by knowledge of its external outputs.” In simple terms, observability means adequate insights into a system facilitating corrective action.
There are three main pillars of observability:
- Event Logs: A record of an event that happened in the system. Events are discrete and contain metadata about the system when they occur.
- Tracing: The system usually consists of many parts/components that work cohesively to provide certain meaningful functionality. For a particular functionality, tracing of request and response flow across distributed components is critical for effective debugging/troubleshooting.
- Metrics: Performance of the system measured over a period of time. These indicate the service level of a system.
Now the question is, how do we get this implemented in a Kubernetes cluster for our microservices components?
Microservices - Kubernetes App
Below is an example that can act as a guide for your microservices-based production app.
Let us consider a microservice app that provides weather information for a given city.
- Weather-front: Component that comprises frontend UI to input a city name to view the weather info. Please see the screenshot above.
- Weather-services: Component that takes the city as an input and calls an external weather API to retrieve weather details
- Weather-db: This component is the Maria database component where weather data, that is pulled in the background for the city being followed, is stored.
Using a Kubernetes Deployment object, the microservices above are deployed, and below is the output of the kubectl get deploy command.
Below are the images to be used to deploy microservices components.
Observability - Pillar One - Event Logs
To get the first pillar of observability, let's install the EFK stack: Elasticsearch, Fluentd, and Kibana, and below are simple installation steps.
Elasticsearch and Kibana:
Post-installation, you can launch the Kibana dashboard, and below is what you would see:
Once Fluentd starts, below is what you would see. Fluentd is run as Daemonset (Ref: Image 4)
And soon, logs will start getting put into Elasticsearch that can be explored on Kibana as below:
Now the first pillar of observability is implemented, so let's focus on the next - distributed tracing.
Observability - Second Pillar - (Distributed) Tracing
For distributed tracing, we have few alternatives in place for java apps like Zipkin, Jaeger, Elasticsesarch APM, etc.
Since we have the EFK stack already in place, let's use the APM provided by Elasticsearch. First, let's start the APM server as a Kubernetes Deployment.
Elastic APM Server deployment snippet:
Once the APM server is running, then add the APM agent in the microservices in a non-invasive method. See the code snippet below used for the weather-front microservice. Similar code snippets should be used for the weather-services component as well.
Snippet of APM agent in microservice weather-front:
Post redeployment of the microservices components, you can navigate to Observability -> APM console on Kibana to see services appearing (Ref: Image 6)
Once you click on the weather-front service, you can see the transactions:
Click on any of the transactions and you would see a drill-down view with more details on the transaction, including latency, throughput, trace Sample, etc.
The above screenshot captures the distributed tracing where the linkage of weather-front and weather-services microservices is clearly depicted. Clicking on the Trace Sample would take you to the transaction details.
In the transaction details, the Actions dropdown provides an option to select traversing the logs for this particular transaction
So by now, the two pillars of observability are covered.
Observability - Third Pillar - Metrics
To implement the third pillar, i.e., metrics, we can use the APM Services dashboard where Latency, Throughput, and Error rates are captured.
Besides, we can use the Spring Boot Prometheus Actuator plugin to collect the metrics data. For this, first, install Prometheus and Grafana using the simple commands below:
Prometheus and Grafana:
Once Prometheus and Grafana are up, then add the code below to the microservices and redeploy:
Prometheus metrics injection - Spring boot microservice:
Once the microservices are redeployed, open the Grafana and import dashboard id 12685 and select the microservice that you want to see the metrics for. See below for weather-front:
And to see the whole cluster's metrics, import the Grafana dashboard id 6417 and you will see something like below:
Opinions expressed by DZone contributors are their own.