System Memory Health Check for ASP.NET Core

I found a temporary cross-platform solution for .NET Core to read system memory metrics until framework-level libraries appear. This blog post shows how to build an ASP.NET Core health check for system memory metrics.

Getting Started

We are using the MemoryMetricsClient class defined in the blog post referred to above. The class with MemoryMetrics result is given here again.

public class MemoryMetrics
{    
  public double Total;    
  public double Used;    
  public double Free;
} 

public class MemoryMetricsClient
{    
  public MemoryMetrics GetMetrics()    
  {        
    MemoryMetrics metrics;         
    if (IsUnix())        
    {            
      metrics = GetUnixMetrics();        
    }        
    else        
    {            
      metrics = GetWindowsMetrics();        
    }          

    return metrics;    
  }     

  private bool IsUnix()    
  {        
    var isUnix = RuntimeInformation.IsOSPlatform(OSPlatform.OSX) ||                     
      RuntimeInformation.IsOSPlatform(OSPlatform.Linux);         

    return isUnix;    
  }     

  private MemoryMetrics GetWindowsMetrics()    
  {        
    var output = "";         
    var info = new ProcessStartInfo();        
    info.FileName = "wmic";        
    info.Arguments = "OS get FreePhysicalMemory,TotalVisibleMemorySize /Value";        
    info.RedirectStandardOutput = true;         

    using (var process = Process.Start(info))        
    {            
      output = process.StandardOutput.ReadToEnd();       
    }         

    var lines = output.Trim().Split("\n");        
    var freeMemoryParts = lines[0].Split("=", StringSplitOptions.RemoveEmptyEntries);        
    var totalMemoryParts = lines[1].Split("=", StringSplitOptions.RemoveEmptyEntries);         
    var metrics = new MemoryMetrics();        

    metrics.Total = Math.Round(double.Parse(totalMemoryParts[1]) / 1024, 0);        
    metrics.Free = Math.Round(double.Parse(freeMemoryParts[1]) / 1024, 0);        
    metrics.Used = metrics.Total - metrics.Free;         

    return metrics;    
  }     

  private MemoryMetrics GetUnixMetrics()    
  {        
    var output = "";         
    var info = new ProcessStartInfo("free -m");        
    info.FileName = "/bin/bash";        
    info.Arguments = "-c \"free -m\"";        
    info.RedirectStandardOutput = true;         

    using (var process = Process.Start(info))        
    {            
      output = process.StandardOutput.ReadToEnd();            
      Console.WriteLine(output);        
    }         

    var lines = output.Split("\n");        
    var memory = lines[1].Split(" ", StringSplitOptions.RemoveEmptyEntries);         

    var metrics = new MemoryMetrics();        
    metrics.Total = double.Parse(memory[1]);        
    metrics.Used = double.Parse(memory[2]);        
    metrics.Free = double.Parse(memory[3]);         

    return metrics;    
  }
}

For Windows, we are using wimc to get memory metrics. On Linux, we are using a command line utility called “free.”

NB! This memory metrics client also works on Windows subsystem for Linux (WSL). Solutions using the Process class may return incorrect results under WSL. 

Health Status

As we are writing health checks, it’s a good idea to stop for a moment and think about how to translate memory metrics to health status. We can return memory metrics with a healthy status no matter what numbers say, but it doesn’t seem very intelligent to me.

I took rough numbers I have seen in online conversations by admin guys and defined health status based on used memory as shown here:

• Healthy — up to 80%.
• Degraded — 80% to 90%.
• Unhealthy — over 90%.

System Memory Health Check

Here is the health check. Notice how I use a data dictionary for memory metrics.

public class SystemMemoryHealthcheck : IHealthCheck
{    
public async Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default)    
    {        
    var client = new MemoryMetricsClient();        
        var metrics = client.GetMetrics();        
        var percentUsed = 100 * metrics.Used / metrics.Total;         
        var status = HealthStatus.Healthy;        

        if (percentUsed > 80)        
        {            
        status = HealthStatus.Degraded;        
        }        

        if(percentUsed > 90)        
        {            
        status = HealthStatus.Unhealthy;        
        }         

        var data = new Dictionary<string, object>();        
        data.Add("Total", metrics.Total);        
        data.Add("Used", metrics.Used);        
        data.Add("Free", metrics.Free);         

        var result = new HealthCheckResult(status, null, null, data);         
        return await Task.FromResult(result);    
    }
}

NB! If you see the Append() extension method on a data dictionary when adding metrics, then don't use it. It doesn't generate an error, but it doesn't work correctly.

Enabling ASP.NET Core Health Checks

To make health checks work, we have to add them to ASP.NET Core request pipeline. In the ConfigureServices() method of the Startup class, we add services to request pipeline and configure them.

public void ConfigureServices(IServiceCollection services)
{    
  services.AddHealthChecks()            
    .AddCheck("ERP", new PingHealthCheck("www.google.com", 100))            
    .AddCheck("Accounting", new PingHealthCheck("www.bing.com", 10))            
    .AddCheck("Database", new PingHealthCheck("www.__Dbing1.com", 100))            
    .AddCheck<SystemMemoryHealthcheck>("Memory");     

  services.AddControllersWithViews();    
  services.AddRazorPages();
}

You can comment out PingHealthCheck rows. If you need ping health checks too, then take a look at these blog posts:

• ASP.NET Core health checks quick start.
• Avoiding ping flood in ASP.NET Core health checks.

The Configure() method of the Startup class also needs a few changes. Take a careful look at the options.ResponseWriter line. This is where the formatting of the health check result happens.

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{    
  if (env.IsDevelopment())    
  {        
    app.UseDeveloperExceptionPage();    
  }    
  else    
  {        
    app.UseExceptionHandler("/Home/Error");    
  }     

  var options = new HealthCheckOptions();    
  options.ResponseWriter = async (c, r) => {         

    c.Response.ContentType = "application/json";        
    var result = new List<ServiceStatus>();        
    result.Add(new ServiceStatus {Service = "OverAll", Status = (int)r.Status});        
    result.AddRange(            
      r.Entries.Select(                
        e => new ServiceStatus                
        {                    
          Service = e.Key,                    
          Status = (int)e.Value.Status,                    
          Data = e.Value.Data.Select(k => k).ToList()                
        }            
      )        
    );                

    var json = JsonConvert.SerializeObject(result);         

    await c.Response.WriteAsync(json);    
  };     

  app.UseHealthChecks("/hc", options);    
  app.UseStaticFiles();    
  app.UseRouting();    
  app.UseAuthorization();     

  app.UseEndpoints(endpoints =>    
  {        
    endpoints.MapControllerRoute(            
      name: "default",            
      pattern: "{controller=Home}/{action=Index}/{id?}");        
    endpoints.MapRazorPages();    
  });
}

Health checks are available as /hc end-point in our application. Let’s run the application and navigate to the health checks page.

[  
  {    
    "Service": "OverAll",    
    "Status": 0,    
    "Data": null  
  },  
  {    
    "Service": "ERP",    
    "Status": 2,    
    "Data": []  
  },  
  {    
    "Service": "Accounting",    
   "Status": 1,    
    "Data": []  
  },  
  {    
    "Service": "Database",    
    "Status": 0,   
    "Data": []  
  },  
  {    
    "Service": "Memory",    
    "Status": 0,    
    "Data": [      
      {        
        "Key": "Total",        
        "Value": 8117.0      
      },      
      {        
        "Key": "Used",        
        "Value": 7462.0      
      },      
      {       
        "Key": "Free",        
        "Value": 655.0      
      }    
    ]  
  }
]

We can visualize health checks, as shown in my blog post Displaying ASP.NET Core health checks with Grafana and InfluxDB. It’s not much effort to add support for memory metrics to the data collector.

How Long Did it Take to Get Memory Metrics?

As our current solution makes use of external shell applications I would like to see how long it took to get system memory metrics. To get this data to metrics I add Duration property to Memory metrics class.

public class MemoryMetrics
{    
  public double Total;    
  public double Used;    
  public double Free;    
  public long Duration;
}

To get the amount of time it took to get memory metrics, we modify the GetMetrics() method of the MemoryMetricsClient class().

public MemoryMetrics GetMetrics()
{    
  MemoryMetrics metrics;    
  var watch = new Stopwatch();   

  watch.Start();    
  if (IsUnix())    
  {        
    metrics = GetUnixMetrics();    
  }    
  else    
  {       
    metrics = GetWindowsMetrics();    
  }    
  watch.Stop();  

  metrics.Duration = watch.ElapsedMilliseconds;     

  return metrics;
}

To make duration appear in memory health check data, we have to add it to the data collection in the health check class.

var data = new Dictionary<string, object>();
data.Add("Total", metrics.Total);
data.Add("Used", metrics.Used);
data.Add("Free", metrics.Free);
data.Add("Duration", metrics.Duration);

It’s time to check out the health check results again and see what the approximate duration of system memory health check.

[  
  {    
    "Service": "OverAll",    
    "Status": 0,    
    "Data": null  
  },  
  {    
    "Service": "ERP",    
    "Status": 2,    
    "Data": []  
  },  
  {    
    "Service": "Accounting",    
    "Status": 1,    
    "Data": []  
  },  
  {    
    "Service": "Database",    
    "Status": 0,    
    "Data": []  
  },  
  {    
    "Service": "Memory",    
    "Status": 0,    
    "Data": [      
      {        
        "Key": "Total",        
        "Value": 8117.0      
      },      
      {        
        "Key": "Used",        
        "Value": 7462.0      
      },      
      {        
        "Key": "Free",        
        "Value": 655.0      
      },      
      {        
        "Key": "Duration",        
        "Value": 186      
      }    
    ]  
  }
]

Getting system memory metrics only takes a few hundred milliseconds on my machine, and it clearly tells us there is minimal overhead in getting memory metrics.

Wrapping Up

Like what often happens in this blog, one thing leads to another. This time I wrote a cross-platform class to get system memory metrics with .NET Core, and then I took the client class to ASP.NET Core web application and used it to write memory metrics health check. I use this health check in some Azure VMs that run Linux where a guest OS agent is unstable on reporting memory metrics. With this health check, I always get valid data about VMs memory.