Member Spotlight: Shamsher Khan

There’s always more to our contributors than what you see in their author profiles. For our latest Member Spotlight, I sat down with Shamsher Khan to learn more about his newest project. What started as a frustrating Kubernetes troubleshooting problem has since grown into published research, a new way of thinking about operational evidence, and ongoing open-source work.

What first got you interested in digging into complex infrastructure and systems problems?

"I’ve always been interested in problems where the visible symptom is not necessarily the real cause. In infrastructure, especially distributed systems, a service can look healthy from one angle while something important is already failing underneath. Troubleshooting becomes less about finding one bad log line and more about understanding how the application, container, node, network, scheduler, and platform interacted over time.

That is what made Kubernetes particularly interesting to me. It automates a lot of recovery, which is great operationally, but that also means the system can change very quickly while you are still trying to understand what happened. Over time, I found myself increasingly interested not just in fixing incidents, but in understanding what information engineers actually have available during and after those incidents, what disappears, and where existing tooling helps or still leaves gaps. That curiosity has shaped a lot of my writing and open-source work."

Your DZone article, “When Kubernetes Forgets: The 90-Second Evidence Gap,” ended up becoming the starting point for Operational Memory Architecture (OMA). What were you seeing in Kubernetes that made you think, “There’s a bigger problem here”?

It came from a very specific frustration during incidents. A pod would crash, Kubernetes would restart it, and by the time I got there to investigate, some of the information I wanted was already gone or had changed.

One example is LastTerminationState. Kubernetes keeps information about a container’s most recent termination, but when that container fails again, the previous termination context is replaced. In a fast crash loop, that can happen repeatedly in a short period of time. You can arrive at a pod that has restarted thousands of times and still have only a very small window into how that sequence began.

What made me think the problem was bigger was realizing that this was not really a Kubernetes bug. Kubernetes is primarily designed to maintain desired state and restore workloads. Preserving a complete forensic history is a different concern. Once I started looking more systematically, I saw similar boundaries elsewhere. Kubernetes Events have limited retention, short-lived workloads can exist entirely between monitoring samples, and some node- or runtime-level evidence can become difficult or impossible to reconstruct after the underlying state changes.

There are already strong observability tools that help with logs, metrics, traces, and events, so the question was not, “Why doesn’t Kubernetes keep everything forever?” That would not be realistic or necessarily desirable. The question became more specific: are there predictable points after which certain diagnostic evidence can no longer be recovered, and can we reason about those points explicitly? I started calling those points evidence horizons. OMA grew from trying to characterize those horizons and explore what evidence may need to be captured before they are crossed." 

Now that the research is being published in IEEE Access, what do you hope people working with these systems take away from it? And where would you like to see OMA go from here?

"The main thing I hope people take away is that recovery and diagnosis are related, but they are not the same problem. A platform can successfully restore an application while still losing some of the context that would have helped explain why it failed. I think many engineers have experienced this without necessarily having a name for it. If you have ever finished an incident review with, “We’re not completely sure what actually triggered this,” disappearing or short-lived evidence may be one reason.

I also want to be careful not to suggest that OMA replaces existing observability platforms. Tools for logs, metrics, traces, events, and distributed tracing are already essential. OMA is better thought of as a way of reasoning about when different kinds of evidence remain available and when they may cross a point where recovery becomes difficult or impossible.

There is also a practical side to this. Teams doing post-incident reviews, reliability analysis, or audit and compliance work may need to reconstruct what happened after the system has already recovered. Thinking explicitly about evidence retention and recovery boundaries can help teams decide what information is worth preserving.

As for where OMA goes next, the research is still early. The work evolved in stages: I first published the foundational OMA idea on arXiv, then extended it with a broader evidence-horizon taxonomy and additional validation before developing it into the peer-reviewed IEEE Access paper. The implementation and experiments are public, and the most useful next step is independent validation in environments different from the ones I tested. There are also limitations in the current work. For example, some node-level evidence across kubelet or node restart boundaries requires deeper integration than the current architecture provides. I documented that rather than trying to claim the problem was solved.

Some of these ideas have also influenced practical work I’m doing in OpsCart, an open-source Kubernetes operational triage project. OpsCart is not a replacement for OMA or for established observability tools. I use it more as an engineering testbed for exploring how incident context, workload history, and diagnostic evidence can be surfaced in a way that is useful during everyday Kubernetes troubleshooting.

I would like to see other engineers test both the research assumptions and the practical tooling, challenge the model, and point out where it does not hold up. That kind of feedback is more valuable at this stage than claiming the architecture is complete."

Research: https://ieeexplore.ieee.org/document/11656328
OMA implementation: https://github.com/opscart/k8s-causal-memory
OpsCart: https://github.com/opscart/opscart-k8s-watcher

After spending so much time thinking about Kubernetes, what’s your ideal way to completely unplug for a weekend?

The first requirement is definitely no Kubernetes dashboards. I spend a lot of time during the week thinking about systems, debugging, writing, and experimenting, so on weekends I like doing almost the opposite: spending time with family, getting outside, going somewhere for the day, or just having time where I’m not trying to solve a technical problem.

Infrastructure problems have a way of staying in your head even after you close the laptop, so sometimes the best reset is doing something that has absolutely nothing to do with technology.

To see more of Shamsher's content, here's the link to his DZone profile.