Monitoring Health of ASP.NET Core Background Services With TCP Probes on Kubernetes
Many microservices applications require background tasks and scheduled jobs to process requests asynchronously. In the .NET Core ecosystem, background servic...
Join the DZone community and get the full member experience.Join For Free
Many microservices applications require background tasks and scheduled jobs to process requests asynchronously. In the .NET Core ecosystem, background services are called Hosted services because a single host, such as a web host or a console host, can run several such services in the background while it is alive. In terms of implementation, a hosted service is required to implement the
You can implement the
IHostedService interface yourself, or even better, leverage the
BackgroundService class that implements some common concerns such as cancellation token management and error propagation to the host for you. A class inheriting from the
BackgroundService abstract class only needs to implement the
ExecuteAsync method to define the background task's business logic. If you want to read more about the internals of ASP.NET Core background tasks, please refer to the Microsoft documentation.
Kubernetes relies on probes in your application to assess whether your application is healthy. If you are not familiar with the health probes used by Kubernetes, you can read more about them on the Kubernetes documentation. If your application continuously fails to respond or responds with an error to several probe requests, Kubernetes restarts the pod.
In ASP.NET Core, you can use the implicitly referenced package
Microsoft.AspNetCore.Diagnostics.HealthChecks to add health checks to your application. The core health check package does not support probing external services such as a database. If your application's health depends on external services, you can add custom probes by implementing the
IHealthCheck interface. We will shortly see how simple it is to implement one. Refer to the official Microsoft guidance on health checks for ASP.NET Core applications that describes this concept.
The default implementation of health checks on ASP.NET Core comprises a middleware, a hosted service, and a few libraries. The health check probes are exposed by your application over HTTP. Since there is a lot of goodness packaged in the ASP.NET Core health check framework, we will leverage it to expose health checks over TCP. But why should you consider the modified TCP based implementation of health checks over the default HTTP based one? You might not want to include the whole shebang of routing and middleware in your background services on which the ASP.NET Core health checks depend. In such cases, exposing a single port attached to a TCP listener is a suitable choice.
The Nine to Five Application
There are days at the office when you are just counting minutes until the clock strikes five. For pushing through such days, I built a simple background service that displays the number of minutes left in the workday from Monday to Friday between 9 A.M and 5 P.M. The source code of the application is available on my GitHub repository.
We will configure health checks on this service and deploy it to our local Kubernetes cluster. I recommend using either Docker Desktop or KinD for running a local Kubernetes cluster. Let's discuss the details of implementing this service next.
The following code listing presents the hosted service/background service that logs the time left in the workday every minute. It also logs a message every minute for weekends and before and after work hours.
To execute this service, you need to register it with the host using the
Let's now add health checks to our service and deploy it to Kubernetes.
Health Check Service
You can add custom health checks to your application by implementing the
IHealthCheck interface, which requires defining the
CheckHealthAsync method. The following health check implementation always reports the application state as healthy. You can add custom logic to this method to return an appropriate response, denoting the actual health of the service.
To register the custom health check, you need to add the service type and the name of the check to the health check services using the
AddCheck method as follows.
We now need to attach a TCP listener to the health check port to report the service's health when Kubernetes executes a liveness or readiness probe on the pod hosting the service. Following is the implementation of the
TcpHealthProbeService which is a hosted service that does that.
UpdateHeartbeatAsync method, which is invoked every second, we call the
CheckHealthAsync method of the
HealthCheckService class that requests all the registered health check services to report the health of the service. Based on the response that we receive, we either respond to the probe or halt the TCP listener and do not report the service's health. Kubernetes treats unresponded requests as a permanent error after reaching the consecutive failure threshold. Unlike HTTP probes that give you the option to respond to every probe request with the appropriate status code to indicate the health of the service, with TCP probes, you can either accept or reject a TCP connection from Kubernetes to indicate the health of the service.
Deploy to Kubernetes
Use the following command to build a container image named
nine-to-five:1.0.0 using the Dockerfile present in the
Next, use the following Kubernetes specification to deploy the background service as a pod. In the repository, you will find this specification in the file named deploy.yml.
To apply the manifest, execute the following command.
Following is a screenshot of the background service in action. I use the K9S CLI for viewing the cluster statistics and logs.
In the previous output, you can also see the log events generated on the execution of TCP health probes. Execute the following shell commands to monitor the Kubernetes events generated in the
demo-ns namespace continuously.
The following screenshot presents the output generated on the execution of the previous command. You wouldn't see any error events yet because the service works as expected and the health of the service is good.
Let's now update the custom health check to change the health state of the service and inspect the logs again.
Simulating Health Check Failure
Navigate to the
CustomHealthCheck class and update the code as follows to simulate an unhealthy state of service.
Let's create another docker image of the application by executing the following command.
Use the following command to redeploy the existing pod with the new container image that you just created.
Let's revisit the Kubernetes events that we are monitoring with the
In the previous output, you can see that Kubernetes determined that the service is unhealthy because of repeated probe failure. Let's check the state of the application by inspecting its logs now.
You can view the logs generated by failed health checks. Because of the repeated failed probes, Kubernetes restarted the pod, and hence our application's final log statement indicates the shutdown of the application.
In this post, we saw how we could add custom health checks to ASP.NET Core applications. We customized the default ASP.NET Core health checks to respond to probes over TCP.
Finally, we configured the liveness and readiness probe settings for our application and deployed it to a local Kubernetes cluster.
Published at DZone with permission of Rahul Rai, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.