Monitoring and Observability

When putting their model into production, every team or organization encounters the same issue. Failures go unnoticed for days at first because there is no monitoring. As teams begin to fix the issues, they identify areas where production results deviate from the training data, create dashboards for every metric, and set alerts for every threshold. This results in engineers being paged at two in the morning for a bug that fixes itself within an hour, and when an important alert arises, it goes unanswered due to alert fatigue, creating a pipeline that silently feeds garbage into the model. When a team learns to disregard 95% of the issues, they are very likely to disregard the remaining 5% that are actually important, and the solution to this isn’t less monitoring. The good solution to this problem is monitoring, which is tiered, routed, and pruned differently from the infrastructure monitoring that most teams already know. The Problem With Applying Old Monitoring Rules To AI Traditionally, application monitoring used to be binary, which is whether the application or service is up or down, latency is high or low, etc. But AI models don’t fail with these signs; they usually degrade over time. For instance, a recommendation model does not show exceptions when the user behavior shifts; it just silently gets worse at what it was supposed to do. A classifier model does not throw an error when its input distribution changes; it just returns answers confidently with increasingly wrong predictions. An AI application does not crash when it hallucinates; instead, it returns a normal HTTP 200 response with incorrect content. This creates two problems: When AI models fail, the reason for failure is invisible to classical infrastructure monitoring, which causes teams to bolt on multiple checks like data quality checks, drift detectors, and output scorers, each introducing a new source of noise. AI models are statistical in behavior and not deterministic, so setting threshold alerts on them leads to them firing constantly, and training teams have to tune the model. As a result, thorough AI monitoring does not make the application safer; beyond a certain point, it only makes things worse. What to Actually Monitor Monitoring issues that no one will ever take action on is often the first step towards alert fatigue. It is useful to consider it in four layers, each with its own owner and mode of failure. Infrastructure and service: Metrics like inference latency, throughput, Graphics Processing Unit (GPU)/Central Processing Unit (CPU) utilization, error rates, and cost per request and token consumption for anything calling a hosted large language model (LLM) API are classic operational metrics and can usually be monitored with the existing Application Performance Monitoring (APM) tools. Data quality: This is another important thing to keep an eye on because it can cause broken feature pipelines, upstream schema changes, input formats being changed without getting noticed, and null-rate spikes. These are usually the worst failures because you can't see them unless you're looking for them, and the model keeps making predictions based on bad data. Model quality: This can be tracked by looking at changes in the Confidence Score or how much the prediction distribution has changed from what was seen during training. This can be used instead of measuring accuracy because it's hard to tell right away how measures like accuracy are calibrating, because to measure accuracy, you would have to compare the predicted result to the actual correct answer, which doesn't always exist at the time of prediction. Generative artificial intelligence/large language model quality: Metrics like hallucination rate, coherence, factual grounding, toxicity, and susceptibility to prompt injection need different types of tooling to identify them because they are not like traditional metrics and would require human-in-the-loop sampling or an LLM as a judge for identifying them. The mistake many teams make is that they apply the same alerting techniques to all four layers, which is the infrastructure one, as that is the traditional way of setting up monitoring for applications, but issues related to data quality and model quality require a trend-based review. How to Alert Without the Noise Replace static thresholds with adaptive baselines. When systems learn a baseline from historical behavior and trigger alerts on deviations from it, like “alert if latency exceeds 200ms,” this ignores the daily and weekly traffic patterns, and the same is valid for data volume and null rates, which leads to a large number of false alarms being raised. So, teams that have made this switch from static thresholds to adaptive baselines have reportedly reduced noisy alerts by 60–90%. Introduce real severity tiers. When an alert is critical and poses an instant business risk, it is sent to an on-call engineer so that the problem can be fixed right away. Warnings about poor performance that are not critical are sent to a Teams chat channel during business hours, and signals about long-term trends land on the dashboard to be looked at from time to time. This helps to make sure that the notification's urgency matches its real urgency. Correlate and deduplicate before notifying. One change to the schema upstream can cause a dozen problems downstream. Sending a dozen alerts for one root cause either makes the team too busy or forces them to mentally group alerts together, which your tools should be doing for you. Route alerts to whoever can act on them. Misrouting is a common cause of tiredness. If the central platform team doesn't know about the business, they might ignore a spike they can't understand, and the domain team that would be able to understand it would never see the alert. Both problems are solved by linking alerts to the right person by domain, based on where the problem starts. Prioritize by business impact. A system that looks for unusual events handles all alerts the same way because it doesn't know which parts of your system are important to the business. When you think about how important each problem is before choosing how loud to alert, you get a lot fewer alerts overall, and a lot more of them are ones that you should actually act on. Conclusion It's important to understand that all of the ideas we've talked about work together; none of them can be used on their own. For example, adaptive thresholds only give out fewer alerts that aren't differentiated by severity. Without proper routing, severity tiers send the wrong messages about how important something is to the incorrect individuals. To avoid alert fatigue, teams need to take comprehensive actions, which include proper alert designs and organizational practices. They should also ensure that every alert can be acted on, which is better than monitoring everything, because AI monitoring only scales, and not having anyone see a model fail could have serious consequences. Good monitoring means building a system that sends alerts only when it matters, so when it does, people actually act on it.

Key Takeaways In regulated industries, cloud migration success is determined less by technology selection and more by how deliberately you decouple risk vectors — compliance risk, organizational hesitation, user adoption gaps, and integration changes — so no single failure can derail the whole program.You can successfully migrate an application to AWS while keeping data on-premises by routing through a REST API abstraction (e.g., IBM’s DB2 REST API layer) paired with dedicated AWS security groups controlling cloud-to-on-prem traffic, allowing the data migration to proceed on its own compliance and trust-building timeline.The most dangerous compliance gap in regulated applications isn’t declared sensitive fields — it’s free-form text fields where users may inadvertently type SSNs, credit cards, or other regulated identifiers; proactive tokenization in the application’s write path closes this gap before any audit finds it.Long-tenured business users carry a decade of UX muscle memory that QA testing cannot replicate; allocating real production validation time (such as a 15-day dark deployment cohort) is essential when migrating systems users have relied on daily for 10+ years.Before starting a regulated cloud migration, ask which risk vector each architectural decision is decoupling and whether your team is aligned on why — this single question reframes "cloud migration" from a technology project into a coordinated risk-management exercise. Introduction Most published writing on legacy-to-cloud migration treats it as a technical exercise: pick the stack, plan the cutover, flip the switch. In regulated industries, that framing fails — and the failure mode isn’t a missed deployment window. It’s a stalled program, a failed compliance audit, or a client who pulls back from the cloud strategy entirely. A cloud migration in healthcare insurance is as much about regulatory risk management, organizational trust-building, and user adoption as it is about microservices and Fargate. Get the technology right and miss the risk choreography, and the project doesn’t ship. I led the first WebSphere-to-AWS migration in the health division of a Fortune 50 insurer — a multi-year program touching PHI data, long-tenured business partners, and downstream services concurrently migrating to the cloud. Over that program, six architectural patterns emerged as decisive. Not for the technology they enabled, but for the risks they made manageable. None are individually novel. What’s distinctive is how they work together — as a coordinated set of risk-decoupling decisions in a first-of-its-kind regulated cloud migration. Pattern 1: Strangler Fig With Dark Deployment When migrating critical production systems to the cloud, the temptation is a hard cutover — flip the switch at 2 AM on a Sunday and hope for the best. We chose a different path: a 15-day dark deployment on AWS production, accessible only to a designated cohort of business partners. Three factors drove this decision. 1. First-mover risk in the department. This was the first WAS-to-AWS migration in this Fortune 50 insurer’s health division. There was no internal precedent to draw from — no playbook, no lessons learned from a prior AWS rollout. A "big bang" cutover would have exposed our full user base to whatever unknowns we hadn’t anticipated. Dark deployment let us pioneer the path with limited blast radius. 2. Regulatory exposure on PHI data. The application processes Protected Health Information. Any data integrity issue — a missed field, a misformatted record, a sync gap — could have triggered regulatory scrutiny. By exposing the new AWS environment to a small group of business partners first, we could validate end-to-end data flow in real production conditions without putting the full user base or compliance posture at risk. 3. UX learning curve. We had explicitly rejected a lift-and-shift approach. The new application wasn’t just re-hosted — the UI had been redesigned, the APIs restructured, and user workflows updated. Even excellent technical execution couldn’t eliminate the learning curve our users would face. Dark deployment gave us 15 days of real-world UX observation: where do users hesitate, what do they misunderstand, which workflows feel awkward? By the time we cut over publicly, we had already addressed the rough edges. The result: When we replaced the WAS production URL with the AWS production URL, end users perceived the change as a routine UI update, not a foundational technology migration. Pattern 2: Decouple Application Migration From Data Migration The default assumption in cloud migration is that application and data should move together. We made the opposite choice: migrate the application to AWS while keeping the underlying DB2 data on-premises. Three factors made this the right call. 1. PHI/HIPAA compliance complexity. The application processes Protected Health Information governed by HIPAA. Moving regulated healthcare data to a new environment raises a long list of compliance questions — encryption-at-rest configurations, audit logging, access control policies, business associate agreements with the cloud provider, breach notification readiness. None of these are insurmountable, but they take months of compliance review. Treating data migration as a separate workstream with its own compliance approval cycle was significantly less risky than bundling it into the application cutover. 2. Client comfort and trust-building. Cloud migration is as much a psychological transition for the client as a technical one. Moving an application to AWS is one decision; moving sensitive data off the client’s own infrastructure is a much larger one — it changes their security perimeter, their incident response posture, and in some cases their regulatory filings. Insisting on moving both at once would have either delayed the program waiting for full executive comfort, or risked a "no" on the entire initiative. Application-first let us demonstrate the new architecture working successfully before the data migration conversation began. 3. Parallel team enablement. Decoupling created room for a separate analytics team to independently assess which data could move to the cloud, on what timeline, and under what compliance framework. The application architecture was designed from day one to support a hybrid future — partial data on AWS, other data on-prem — so the analytics team’s work didn’t block application progress. How the technical decoupling works. The natural temptation when keeping data on-prem is to expose a direct database connection from the AWS application back to the on-prem DB2 instance. We rejected that — opening database ports across the cloud-to-on-prem boundary is a security liability, a latency problem, and a fragile dependency. Instead, we used IBM’s DB2 REST API layer to expose data access through authenticated HTTPS-based service calls. The AWS application talks to data through an API, not a database connection. This abstraction also positions the application to seamlessly switch to AWS-resident data later, without any application code change — only the API endpoint moves. Network-layer security follows the same decoupling principle. We provisioned dedicated AWS security groups on the Fargate side specifically for the IMS and DB2 connections back to the on-premises environment — only requests from those approved security groups can traverse the firewall to the on-prem data tier. Combined with the REST API abstraction, this gives us both application-layer (authenticated HTTPS) and network-layer (security-group-controlled) protection across the cloud-to-on-prem boundary. The result: A successful cloud migration with regulatory exposure isolated to a single workstream, and a forward path that doesn’t force the client into uncomfortable decisions before they’re ready. Pattern 3: EJB Monolith → Containerized Microservices on Fargate The original application was a Java EJB monolith running on WebSphere. The "lift-and-shift" temptation would have been to containerize the existing EJB code as-is into AWS Fargate — preserving the architecture, just moving the deployment substrate. We rejected that and instead decomposed the monolith into bounded REST microservices. Three reasons drove this decision. 1. Downstream services were also migrating. The application integrated with 5–7 SOAP-based services owned by adjacent teams — agreement service, customer service, sensitive data masking, and others. Those teams were simultaneously migrating their own services from WAS to AWS, which meant interface contracts, protocols, and endpoints would inevitably change. Inside an EJB monolith, every downstream integration change forces a recompile-redeploy-retest cycle of the entire application. Inside microservices, only the integration adapter for the affected service needs to change. With multiple active migration interfaces, the flexibility difference compounds quickly. 2. EJB development velocity is structurally slow. Even routine changes to EJB code require a full WAR/EAR build, redeployment to the WAS instance, and a heavy test cycle. The technology wasn’t designed for the iteration speed we needed to support a multi-year migration alongside actively changing downstream dependencies. Microservices on Fargate gave us a development model — fast container builds, independent deployments, isolated test environments — that matched the pace of the work. 3. Future data migration optionality. As noted in Pattern 2, the underlying data was kept on-premises for now, but a phased data migration to AWS was planned. By isolating database calls and IMS calls into dedicated microservices, the change required when the data eventually moves is localized — swap one service’s data access logic rather than reworking the monolith. The architecture is positioned for the data move whenever the client is ready. How we sized the decomposition. The boundaries followed natural integration points: each external SOAP integration became its own bounded microservice with a thin REST API. Data access calls (DB2 via REST, IMS) were isolated into dedicated services. The frontend talks to a coordination layer that orchestrates calls across these services. The result was a clean set of containerized microservices on AWS Fargate — each independently deployable, scalable, and testable. The result: A modernization that didn’t just relocate the code, but restructured it to absorb the inevitable changes coming from adjacent migrations across the organization — without recompile-redeploy-retest pain. Pattern 4: Frontend Decoupling via S3 + CloudFront The original WAS application followed the classic tightly-coupled pattern: JSP pages rendered server-side, deployed alongside the backend, scaling and updating as one unit. We made an architectural break in the migration — the frontend became a fully independent single-page React application hosted on Amazon S3 and served via CloudFront. Three factors made this the right call. 1. Independent deployment cadence. Frontend and backend evolve at different speeds. UI tweaks — copy changes, validation logic, visual updates — are frequent and low-risk. Backend API changes are slower and require careful coordination with downstream service migrations. Decoupling them means UI changes can be deployed instantly through a separate UI pipeline (different Git repository, different infrastructure, different release cadence) without touching the backend microservices. A small label change no longer requires a full backend deployment. 2. Adopting an accessibility-first enterprise UI library. Alongside our migration, an internal innovation track was building a shared component library to unify UX patterns across the organization’s applications — consistent typography, controls, brand elements, and critically, accessibility as a first-class concern: full screen reader support, keyboard navigation, sufficient color contrast, and ARIA-compliant semantics. JSP-based legacy pages couldn’t meaningfully integrate this kind of library. By rebuilding the frontend as a React single-page application, we adopted the library fully — and incorporated rigorous accessibility testing into every release cycle. Users who rely on assistive technologies (screen readers, alternative input devices, magnification) get full application access. For an application processing PHI in a regulated industry, this proactive accessibility-first approach is itself a substantial improvement over the legacy app. 3. Global performance through edge caching. S3 alone would have served the static assets, but we layered CloudFront on top to push content to edge locations closer to users. Business partners access the application from different geographic regions; CloudFront cuts load times by serving cached assets from the nearest edge, not the S3 origin in a single AWS region. This is a substantial UX improvement that simply wasn’t possible with WAS-hosted JSPs. How the architecture flows. User requests hit CloudFront, which serves cached React bundles, HTML shells, and static assets from the nearest edge. The React application then makes authenticated REST API calls back to the backend microservices on AWS Fargate. The frontend has no awareness of which microservice serves any particular request — it talks to a coordination API layer that handles orchestration. The result: A UI architecture that’s faster (edge-cached), cheaper (no application servers for the frontend), easier to update (independent pipeline), more inclusive (accessibility-first), and aligned with the broader enterprise UX modernization effort. Pattern 5: Business Partner Real-Production Validation Cohort Pattern 1 described the deployment mechanism — a 15-day dark deployment exposing AWS production to a limited cohort. Pattern 5 is about who was in that cohort and why we deliberately chose real business partners over our QA team for production validation. Two factors shaped this decision. 1. Decades of muscle memory in the existing UX. Our business partners — long-tenured users of the application — had been using the legacy UI for 10–15 years. They knew every workflow, every shortcut, every quirk. The new React application introduced not just a new visual style but new patterns from the organization’s modern component library. Even with rigorous accessibility and usability testing in QA, a brand-new UI in front of users with a decade of habits guaranteed friction. The 15-day validation cycle gave those users time to acclimate to the new patterns and surface UX issues that only show up at the speed of real daily work — keyboard shortcuts they used unconsciously, screens they navigated to multiple times an hour, validation logic that affected their flow. QA testers, by definition, don’t have that muscle memory. 2. First-of-its-kind migration with concurrent change. This was the first WAS-to-AWS migration in the health division, and we’d simultaneously re-architected the UI, the API layer, and incorporated changes from downstream services that were also mid-migration. With that many concurrent changes, even thorough QA can’t realistically simulate the full combinatorial space of real production usage — real customer data, real edge cases, real integration timing, real load patterns. Putting real business partners on the actual AWS production environment for 15 days was our safety net: anything QA missed, the cohort would surface, and we could fix it before broad cutover. Beyond the cohort: maturing the delivery pipeline. A secondary benefit of running an extended validation window was that it gave the engineering team time to mature the CI/CD pipeline alongside the application. By the second application in the migration program, we’d evolved the cohort approach into a full blue/green deployment model on AWS — building organizational learning alongside the application portfolio. The validation pattern isn’t static; it strengthens with each subsequent migration. The result: a validation approach that combined deep domain familiarity (real business partners) with controlled exposure (limited cohort, real production) — catching the issues QA can’t, well before public cutover. Pattern 6: Defensive Tokenization for Sensitive Data in Free-Form Fields In regulated industries, the obvious sensitive data — SSN fields, credit card fields, account number fields — gets protected automatically. The dangerous category is the unstructured data: a free-form text field where a user can type anything. In our application, users entered "health notes" — narrative text describing customer interactions. The risk: nothing in the application schema prevents a user from typing an SSN, a credit card number, a driver’s license, or other regulated identifiers directly into that note. Once stored, that PHI/PII data is sitting in a free-text column with no encryption-at-rest tailored to it, no masking on display, no controlled access — and our compliance posture changes accordingly. We addressed this proactively by integrating an internal sensitive-data-masking service into the application’s write path. Before any free-form text reaches the data layer, the masking service scans the input, identifies regulated identifiers (SSN-pattern strings, credit card numbers via Luhn check, driver’s license formats), and applies tokenization — replacing the identifier with a non-reversible token or masked representation. The original value never lands in the database in plaintext. Three things made this a deliberate architectural pattern, not an afterthought: 1. It was incorporated before the formal risk assessment, not in response to it. Risk assessment was a new exercise for the team — none of us had been through one for AWS-hosted PHI before. Rather than wait for the assessment to flag the free-form field as a finding, we performed our own data classification first, identified the free-form notes as a regulated-data risk vector, and integrated the masking service pre-emptively. When the formal risk assessment ran, this control was already in place. 2. We reused an existing internal service, not built a new one. The masking service already existed in another WAS-hosted application within the broader life/health portfolio. Instead of re-implementing tokenization logic, we adopted the existing service — saving development time and inheriting the existing security review and operational maturity of that service. Migrations are a good moment to identify reusable internal capabilities rather than reinvent them. 3. It addresses a class of risk most compliance reviews don’t anticipate. Compliance checklists focus on declared sensitive fields ("the SSN field," "the account number field"). They rarely interrogate free-form text fields, because those fields aren’t supposed to hold sensitive data. But in practice, users type whatever they need to type — and what they type is what your application stores. Proactive defensive tokenization closes that gap. The result: free-form notes that look normal to users, but whose backend storage is sanitized of any regulated identifiers the user may inadvertently include. The application’s compliance posture is robust to user behavior, not just to user intent. Conclusion: The Through-Line Is Decoupling Looking back across the six patterns, the through-line isn’t any specific technology — it’s a posture: deliberate decoupling of risk vectors so that no single failure, regulatory finding, organizational hesitation, or user adoption gap can derail the whole migration. Pattern 1 (Strangler Fig with Dark Deployment) decouples cutover risk from broader rollout.Pattern 2 (Decouple App from Data) decouples application migration from the data-and-compliance timeline.Pattern 3 (EJB → Microservices) decouples downstream integration changes from our own deployment cadence.Pattern 4 (Frontend on S3/CloudFront) decouples UI release cadence from backend release cadence.Pattern 5 (Business Partner Validation Cohort) decouples real-world UX surprises from public rollout.Pattern 6 (Defensive Tokenization) decouples user behavior risk from data-layer compliance posture. None of these patterns are individually novel. What’s distinctive is choosing them together, as a coordinated set of risk-decoupling decisions in a first-of-its-kind regulated cloud migration. The result was a migration that didn’t surprise our compliance team, didn’t surprise our users, and didn’t surprise our auditors — which, in a regulated industry, is the kind of unsexy outcome that defines success. If you’re starting a similar program, the question isn’t which of these patterns to adopt. It’s: which risk vector are you decoupling, and is your team aligned on why?

There’s always more to our contributors than what you see in their author profiles. For our latest Member Spotlight, I sat down with Shamsher Khan to learn more about his newest project. What started as a frustrating Kubernetes troubleshooting problem has since grown into published research, a new way of thinking about operational evidence, and ongoing open-source work. What first got you interested in digging into complex infrastructure and systems problems? "I’ve always been interested in problems where the visible symptom is not necessarily the real cause. In infrastructure, especially distributed systems, a service can look healthy from one angle while something important is already failing underneath. Troubleshooting becomes less about finding one bad log line and more about understanding how the application, container, node, network, scheduler, and platform interacted over time. That is what made Kubernetes particularly interesting to me. It automates a lot of recovery, which is great operationally, but that also means the system can change very quickly while you are still trying to understand what happened. Over time, I found myself increasingly interested not just in fixing incidents, but in understanding what information engineers actually have available during and after those incidents, what disappears, and where existing tooling helps or still leaves gaps. That curiosity has shaped a lot of my writing and open-source work." Your DZone article, “When Kubernetes Forgets: The 90-Second Evidence Gap,” ended up becoming the starting point for Operational Memory Architecture (OMA). What were you seeing in Kubernetes that made you think, “There’s a bigger problem here”? It came from a very specific frustration during incidents. A pod would crash, Kubernetes would restart it, and by the time I got there to investigate, some of the information I wanted was already gone or had changed. One example is LastTerminationState. Kubernetes keeps information about a container’s most recent termination, but when that container fails again, the previous termination context is replaced. In a fast crash loop, that can happen repeatedly in a short period of time. You can arrive at a pod that has restarted thousands of times and still have only a very small window into how that sequence began. What made me think the problem was bigger was realizing that this was not really a Kubernetes bug. Kubernetes is primarily designed to maintain desired state and restore workloads. Preserving a complete forensic history is a different concern. Once I started looking more systematically, I saw similar boundaries elsewhere. Kubernetes Events have limited retention, short-lived workloads can exist entirely between monitoring samples, and some node- or runtime-level evidence can become difficult or impossible to reconstruct after the underlying state changes. There are already strong observability tools that help with logs, metrics, traces, and events, so the question was not, “Why doesn’t Kubernetes keep everything forever?” That would not be realistic or necessarily desirable. The question became more specific: are there predictable points after which certain diagnostic evidence can no longer be recovered, and can we reason about those points explicitly? I started calling those points evidence horizons. OMA grew from trying to characterize those horizons and explore what evidence may need to be captured before they are crossed." Now that the research is being published in IEEE Access, what do you hope people working with these systems take away from it? And where would you like to see OMA go from here? "The main thing I hope people take away is that recovery and diagnosis are related, but they are not the same problem. A platform can successfully restore an application while still losing some of the context that would have helped explain why it failed. I think many engineers have experienced this without necessarily having a name for it. If you have ever finished an incident review with, “We’re not completely sure what actually triggered this,” disappearing or short-lived evidence may be one reason. I also want to be careful not to suggest that OMA replaces existing observability platforms. Tools for logs, metrics, traces, events, and distributed tracing are already essential. OMA is better thought of as a way of reasoning about when different kinds of evidence remain available and when they may cross a point where recovery becomes difficult or impossible. There is also a practical side to this. Teams doing post-incident reviews, reliability analysis, or audit and compliance work may need to reconstruct what happened after the system has already recovered. Thinking explicitly about evidence retention and recovery boundaries can help teams decide what information is worth preserving. As for where OMA goes next, the research is still early. The work evolved in stages: I first published the foundational OMA idea on arXiv, then extended it with a broader evidence-horizon taxonomy and additional validation before developing it into the peer-reviewed IEEE Access paper. The implementation and experiments are public, and the most useful next step is independent validation in environments different from the ones I tested. There are also limitations in the current work. For example, some node-level evidence across kubelet or node restart boundaries requires deeper integration than the current architecture provides. I documented that rather than trying to claim the problem was solved. Some of these ideas have also influenced practical work I’m doing in OpsCart, an open-source Kubernetes operational triage project. OpsCart is not a replacement for OMA or for established observability tools. I use it more as an engineering testbed for exploring how incident context, workload history, and diagnostic evidence can be surfaced in a way that is useful during everyday Kubernetes troubleshooting. I would like to see other engineers test both the research assumptions and the practical tooling, challenge the model, and point out where it does not hold up. That kind of feedback is more valuable at this stage than claiming the architecture is complete." Research: https://ieeexplore.ieee.org/document/11656328OMA implementation: https://github.com/opscart/k8s-causal-memoryOpsCart: https://github.com/opscart/opscart-k8s-watcher After spending so much time thinking about Kubernetes, what’s your ideal way to completely unplug for a weekend? The first requirement is definitely no Kubernetes dashboards. I spend a lot of time during the week thinking about systems, debugging, writing, and experimenting, so on weekends I like doing almost the opposite: spending time with family, getting outside, going somewhere for the day, or just having time where I’m not trying to solve a technical problem. Infrastructure problems have a way of staying in your head even after you close the laptop, so sometimes the best reset is doing something that has absolutely nothing to do with technology. To see more of Shamsher's content, here's the link to his DZone profile.

A Temporal Workflow that appears stuck is rarely “stuck” in the conventional process sense. Temporal persists Workflow state through Event History and resumes execution through replay, so an open execution can remain healthy while waiting for a timer, Signal, Activity, or external condition. The operational problem is therefore not simply lack of completion; it is lack of expected progress. Effective diagnosis starts by establishing what event should have happened next, why it did not happen, and whether remediation can preserve the Workflow’s business invariants. Temporal’s history model makes that analysis unusually tractable because commands, task transitions, Activity attempts, failures, timers, and external interactions are durably represented as Events. Progress Is Visible in the Event History The first diagnostic artifact should be the execution description and raw history, not application logs. temporal workflow describe exposes current execution information and pending Activity state, while temporal workflow show --output json returns Event History in a form suitable for programmatic replay or analysis. A Workflow Query can additionally expose application-defined state without mutating the execution. Shell temporal workflow describe --workflow-id order-7814 temporal workflow show \ --workflow-id order-7814 \ --output json History should be read as a state-transition trace. A WorkflowTaskScheduled event with no corresponding start suggests that work is waiting for a Worker. A started Workflow Task that repeatedly times out can indicate blocked Workflow code, Worker instability, or excessive work inside a task. Repeated WorkflowTaskFailed events can indicate replay or deterministic-compatibility failures after code deployment. Workflow Task failures are retried by Temporal rather than governed by an Activity-style Retry Policy, so a Workflow can remain open while repeatedly failing to make application-level progress. Activity sequences reveal a different failure surface. ActivityTaskScheduled without ActivityTaskStarted points toward dispatch capacity, missing pollers, queue mismatch, or backlog. Temporal persists Workflow and Activity Tasks in Task Queues, and worker-health guidance identifies Schedule-to-Start latency and approximate backlog count as key signals when tasks wait for Workers. ActivityTaskStarted without completion requires inspection of Start-to-Close and Heartbeat behavior because Temporal relies on Start-to-Close timeout to detect a Worker crash after an Activity has started. Not every long pause is pathological. A timer that has not fired, a Workflow waiting for a Signal, or an Activity still inside a valid timeout window can represent correct durable waiting. Conversely, very large histories can become an operational risk. Temporal warns after 10,240 events or 10 MB and enforces a limit of 51,200 events or 50 MB; Continue-As-New creates a new run with a fresh history while carrying forward relevant state. Triage Works Best as Deterministic Evidence Before Model Judgment LangGraph is useful for automating this analysis, but the safest design keeps Temporal facts deterministic and uses an LLM only for classification, hypothesis ranking, and explanation. LangGraph explicitly supports graphs that mix deterministic nodes with model-driven nodes, while structured output can constrain routing decisions into a defined schema rather than free-form text. A compact analyzer can first reduce raw history into evidence that is difficult to hallucinate: the last completed Workflow Task, consecutive Workflow Task failures, pending Activity IDs, the latest Activity attempt, the timeout type, the last Signal, the last timer, the history size, the task queue, and deployment/version metadata. The model then receives that normalized evidence instead of thousands of raw events. Python def extract_facts(state): events = state["events"] return { "facts": temporal_fact_extractor(events), "tail": events[-60:], } def classify(state): result = triage_model.with_structured_output(TriageResult).invoke({ "facts": state["facts"], "tail": state["tail"], "allowed_causes": [ "worker_unavailable", "activity_retrying", "workflow_task_failure", "intentional_wait", "history_pressure", "unknown", ], }) return {"triage": result} That separation matters operationally. Event parsing can enforce hard rules such as “scheduled but never started,” while the model can correlate several weak signals and produce an explanation. Conditional edges can then route low-risk cases to observation, ambiguous cases to deeper diagnostics, and recovery candidates to an approval gate. LangGraph’s graph API supports conditional routing, and persistence stores checkpoints so triage state survives interruptions or process failures. Recovery Must Preserve Temporal and Business Semantics Diagnosis and remediation should remain separate graph stages. A model-generated recommendation must not directly issue cancellation, reset, or termination. LangGraph interrupts provide a natural control boundary because execution can pause with persisted state and resume only after external approval. Python def approval_gate(state): decision = interrupt({ "workflow_id": state["workflow_id"], "cause": state["triage"].cause, "action": state["triage"].recommended_action, "evidence": state["triage"].evidence, }) return {"approved": decision == "approve"} The remediation choice depends on the failure mode. A transient Worker outage usually requires restoring Worker capacity rather than mutating Workflow state because queued tasks persist until Workers can process them. An Activity repeatedly failing on a recoverable dependency can often be left to its Retry Policy, while permanent errors should be made non-retryable in application design to avoid pointless retries. Activity side effects should be idempotent because Activity attempts may execute more than once under retry and recovery behavior. Cancellation is the preferred stop mechanism when Workflow cleanup logic must run. Temporal records a cancellation request and schedules a Workflow Task so Workflow code can react. Termination is forceful: Workflow code does not receive a chance to clean up, and the terminated event closes the history. That makes termination an escalation path for executions that cannot process cancellation normally. Reset is more powerful and more dangerous. Temporal terminates the current execution and creates a new execution that copies history through a selected reset point, then replays forward using current Workflow code. Progress after the reset point is discarded. Reset is therefore appropriate only after the underlying cause has been corrected and after downstream side effects are reviewed for possible re-execution beyond the reset boundary. Shell temporal workflow reset \ --workflow-id order-7814 \ --event-id 42 \ --reason "Recovered after deterministic-compatibility fix" For history pressure rather than a fault, Continue-As-New is generally the safer lifecycle mechanism because it preserves logical continuity under the same Workflow ID while starting a fresh Event History with a new Run ID. It should be designed into long-lived or high-volume Workflow logic instead of used as an improvised emergency action. Safe Automation Requires an Explicit Remediation Envelope A production triage graph should treat remediation as a constrained transaction. The evidence snapshot, selected run ID, candidate reset event, intended action, reason, approval identity, and execution result should all be persisted before any mutation. The action node should re-read the Workflow immediately before execution and reject the operation if the run has changed or the observed condition no longer matches the diagnosis. This is an engineering safeguard rather than a Temporal requirement, but it reduces time-of-check/time-of-use errors when active Workflows continue progressing during investigation. LangGraph’s checkpoint model supports durable approval state, but resumed graph nodes can re-execute from checkpoint boundaries. Its documentation therefore recommends isolating side effects and designing them to be idempotent. A remediation executor should consequently use an operation ID, record completion externally, and refuse duplicate destructive actions. Recovery Without Guesswork Reliable recovery of a stuck Temporal Workflow is fundamentally an event-history problem, not a process-restart problem. The strongest diagnostic path reconstructs expected progress from Workflow Tasks, Activity attempts, timers, Signals, queue state, timeouts, and history growth before considering mutation. LangGraph can turn that evidence into a durable triage pipeline by combining deterministic extraction, constrained model reasoning, conditional routing, and interrupt-based approval. Safe remediation then follows Temporal semantics: restore Workers when dispatch is the issue, allow bounded retries for transient Activities, cancel when cleanup matters, terminate only as a last resort, reset only after the root cause is fixed, and use Continue-As-New to control long-running history growth. The result is automation that accelerates incident response without allowing probabilistic diagnosis to become an unchecked control plane.

By Akhil Madineni CORE

Open source projects dominated by a single vendor are a hallmark of "open source in name only." Rather than filling the traditional role of open source fostering innovation and decision-making from a diverse community, "open source in name only" projects are often used as marketing tools for proprietary platforms. These projects are also seen as riskier than community-driven projects because a single vendor is more apt to abruptly terminate long-term support, restrict contributions, or switch from an open-source license to a more restrictive one (forcing some previous contributors to pay for the project they helped build). In these projects, critics claim that investments are often lopsided and heavily skewed toward onboarding, marketing, and brand-related support. As a result, technical contributions are frequently less developed, opaque, undocumented, or lacking in real substance, often manifesting merely as a superficial "ease of entry and onboarding." Because of these underlying gaps in documentation and codebase depth, developers are routinely forced to reverse-engineer functionality simply to get the tools to work correctly. An evaluation of three leading open-source observability projects–OpenSearch, Prometheus, and OpenTelemetry (OTel)– by ReveCom was conducted to determine whether they fell under this vendor-dominated category or are truly vibrant community-led projects. According to Gartner research, these three projects are collectively important because together they provide a complete, vendor-neutral observability architecture covering all three fundamental telemetry signals—metrics, logs, and distributed traces — without locking an enterprise into proprietary agent formats or single-vendor cloud platforms. Gartner defines observability as the extent to which internal system states can be inferred from externally emitted data. By pairing OpenTelemetry as a universal collection and routing tier with Prometheus for real-time metric alerting and OpenSearch for high-volume log analytics and trace analysis, organizations gain end-to-end operational visibility, retain full ownership of their telemetry pipelines, and avoid runaway cloud ingestion or lock-in costs. To develop the framework, data from the ReveCom Observability Report 2026 was used, which includes metrics about contribution numbers and quality, including commit frequency, contributor growth, community expansion, and deployment patterns. Based on this data, authentic efforts were separated from perfunctory efforts. "Authentic" contributions were defined as those made to the computing code (i.e., the observability stack for logs, traces, and metrics) and its computational efficiency as measured in latency. The Controversial Fork AWS's controversial decision to monetize and then fork Elasticsearch to create OpenSearch in 2021 (when Elastic made its license more restrictive) is a case study of the risks associated with vendor-dominated projects. It also serves as an example of the issues associated with a vendor forking and heavily promoting a project it contributed minimally to. According to Elastic representatives, although a major beneficiary of Elastic through its managed service, AWS engineers contributed only a "handful" of commits to Elasticsearch from 2020 to 2021, Elasticsearch says. This disparity suggests that the successor project, OpenSearch, was born from a position of minimal technical familiarity with the core codebase. Elastic famously described this as "there is no compression algorithm for experience." For a technical leader, this lack of pre-fork familiarity suggests a significant "experience gap" that can impact the speed and stability of future feature releases. AWS made few fundamental changes to the Elasticsearch codebase it forked to create OpenSearch, largely just rebranding the existing observability tool. Comparing Three Observability Communities In 2024, Amazon donated OpenSearch to the Linux Foundation, bringing it under a governance structure and setting the stage for it to become a more decentralized project. Among other things, once a project is donated to the Linux Foundation, no single company can hold more than 25% of the seats on the technical oversight bodies. Decentralized governance is structured so that substantive, collaborative contributions from several competing observability vendors can better serve the broader community's needs. Amazon's donation set the stage for OpenSearch to become a much more community-driven effort, comparable to the community-led support of the Prometheus and OpenTelemetry projects. Prometheus and OpenTelemetry exemplify healthy, community-led open source standardization. This is how teams should evaluate open source: by the diversity of the entities with "skin in the game." Prometheus emerged from SoundCloud in 2012, where it was designed to track metrics and store them in a time-series database. Around 2014, Grafana and its glassy, visually appealing panels became part of the ecosystem. The combination of Prometheus and Grafana became an integral, de facto standard for monitoring and observability in Kubernetes deployments and infrastructure. Prometheus was donated to the CNCF in 2016 and graduated in 2018. Since then, it has evolved into a very diverse, community-led project, with multiple contributing companies. Grafana Labs remains one of the largest contributors, but the breakdown of substantive commits-excluding documentation-is wide and varied, reflecting the project's broad, collaborative nature. This wider contribution to the project's standardization ensures that engineering talent is portable and the stack remains interoperable. Separating Brand From Backbone A key open source health metric-perhaps the most substantial of all-is ranking substantive engineering contributions, such as code-level commits and pull requests or high-impact technical commits. These are described as commits that can lead to v1.0, v2.0, or v3.0 milestones, signifying production readiness and improvements. The number and frequency of technical contributions, as measured by commits, are markers for a project's community dynamics and value to end users. Looking at OpenSearch, AWS made significant technical contributions in 2025. As the data shows, Amazon contributes the majority of substantive commits (73%) to OpenSearch. Much of this can be attributed to a surge in contributions related to the AI aspects of observability, specifically "search-to-Al infrastructure" commits. IBM and Red Hat have also contributed AI-related work on RAG and vector database optimization. These are solid contributions, and they show that Amazon has moved beyond the early days, when it simply forked Elastic even though it had contributed relatively little to the project. Hopefully, OpenSearch will continue this shift toward increased community participation as new features are added. However, such a dominant share of commits from a single vendor means that one vendor effectively controls the roadmap. In this case, AWS is potentially prioritizing its managed services over users' infrastructure needs. Source: ReveCom Prometheus has a wide range of contributions from vendor organizations. Grafana is the leading technical contributor to Prometheus, largely based on its development of TSDB storage refactoring, Remote Write 2.0, and agent-mode contributions. Red Hat is the second-most frequent technical contributor to Prometheus, a position solidified by its acquisition of CoreOS. As the primary maintainer of the Prometheus Operator-a critical element for monitoring Kubernetes-Red Hat ensures seamless integration between the monitoring stack and the orchestration layer. While Red Hat provides deep engineering support, Prometheus remains a highly collaborative open-source project with contributions from across the industry. Source: ReveCom The OpenTelemetry project, under the leadership of Splunk, Microsoft, Elastic, Grafana Labs, and Google, provides a mature, stable, and innovative framework for the future of observability. By focusing on high-impact technical commits and "good faith" participation, the community helps ensure that observability data remains a standardized utility that empowers developers and platform engineers to navigate the complexities of the modern cloud landscape. Splunk remains the largest contributor of high-impact technical commits to OpenTelemetry. Grafana is a notable contributor at number four by providing Beyla eBPF instrumentation and Prometheus receiver stability improvements. Strategic Recommendations Organizations should adopt an open-source technical strategy that prioritizes authentic engineering and project diversity. The following recommendations are derived from scrutinizing vendor-dominated projects and analyzing high-impact technical commitments. The high-impact focus of companies like Grafana Labs, Splunk, Microsoft, Elastic, and hundreds of other contributor organizations means that OpenTelemetry and Prometheus should remain the foundation of observability for the next several years. When choosing an observability solution, organizations should prioritize vendors that are not only OTel-compliant but also OTel-contributing. This should also apply to Prometheus solutions, especially those for managing Kubernetes environments. ReveCom's findings indicate that the most valuable contributions are those that advance the core "engine" of observability. Procurement decisions should be based on a vendor's ability to demonstrate substantive engineering that solves real-world infrastructure problems rather than relying on superficial marketing claims. Ultimately, none of the three projects covered in this article can be fully characterized as "open source in name only." While OpenSearch arguably fell into that category immediately after it was forked from Elasticsearch, it has evolved since. OpenSearch remains an Amazon-dominated project, but it has seen an upward trend in contributions from the community and from third parties such as Uber, SAP, and Red Hat. For observability community support, as measured by substantive technical contributions that solve infrastructure problems, OpenTelemetry and Prometheus exemplify a healthy balance of governance and code contributions across hundreds of organizations (notably Grafana and Splunk). Led by Grafana and Splunk among the observability providers, these projects fall behind only Kubernetes itself.

By Chris Ward CORE

Most engineering teams working on healthtech applications reach a point where someone asks a question that sounds simple but isn't: How do we make sure a developer testing a new feature can't accidentally access production patient data? The answer determines whether the architecture that follows will be auditable or not. Teams that answer it with process — "we have policies about that" — spend the next 18 months patching access-control gaps that reopen every time a new engineer joins or a new service gets wired in. Teams that answer it architecturally spend a week setting up AWS Organizations correctly and then largely stop thinking about it. This article covers the multi-account architecture pattern for HIPAA-compliant infrastructure — specifically, the account structure decisions that either enforce PHI workload isolation or make it a permanent source of audit findings. Why Single-Account PHI Isolation Fails at the Seams A single AWS account running production, staging, and development workloads creates a specific problem that IAM policies alone cannot fully solve. The issue is not that IAM is insufficient as a technology. IAM policies enforced within an account are only as reliable as the discipline of the people who manage them. A policy that restricts a developer's access to production RDS today can be modified tomorrow by anyone with sufficient IAM permissions. Nothing in the account structure itself prevents the boundary from being crossed. In practice, the gaps show up in predictable ways. A pipeline service role gets broad permissions during a sprint because scoping them properly would have taken an extra hour. An engineer copies an IAM role from staging to production because it was faster than creating a new one. A debugging session in production happens under an account that was supposed to be read-only. None of these are malicious decisions. They are the natural result of putting access control boundaries inside an environment where the people who need to cross them also have the permissions to do so. The access control problem that surfaces during security reviews is almost always this one — not a missing encryption setting or an unpatched vulnerability, but access boundaries that exist on paper and drift in practice. The Multi-Account Model: Enforcement at the Boundary AWS Organizations with a properly structured multi-account hierarchy solves this problem by moving the enforcement point outside the accounts being protected. The boundary is no longer an IAM policy that someone with IAM permissions can modify. It is an account boundary that the engineers inside those accounts cannot cross, enforced by Service Control Policies applied at the organizational unit level. The recommended structure has four organizational units under the root: a Security OU containing a Log Archive account and a Security Tooling account, a Production OU containing only the Production account where PHI workloads run, a Non-Production OU containing Staging and Development accounts, and a Shared Services OU containing the account used for CI/CD pipelines, DNS, and shared tooling. The Production OU sits under its own organizational unit with SCPs that restrict what can happen inside it, regardless of what IAM policies exist within the production account itself. An engineer whose IAM role in the development account grants broad permissions has those permissions scoped to the development account. Crossing into production requires a separate role, in a separate account, with a separate set of credentials. The architectural boundary is the enforcement mechanism, not the IAM policy. The Log Archive account under the Security OU serves a specific purpose: it is the only account to which CloudTrail logs from all other accounts are delivered, and it is an account to which production engineers have no write access. This means the evidence trail for PHI access events cannot be modified by the accounts generating those events - which is exactly what auditors verify when they ask about log integrity. Service Control Policies: What to Enforce at the OU Level SCPs applied to the Production OU are where the architectural enforcement becomes concrete. The first policy prevents anyone inside the production account from disabling CloudTrail, including account administrators: JSON { "Effect": "Deny", "Action": [ "cloudtrail:StopLogging", "cloudtrail:DeleteTrail", "cloudtrail:UpdateTrail" ], "Resource": "*" } CloudTrail continuity across the full audit period is not something that should depend on engineering discipline. It should be architecturally enforced. An account that can leave the organization can escape every SCP applied to it. This policy closes that path: JSON { "Effect": "Deny", "Action": "organizations:LeaveOrganization", "Resource": "*" } PHI that moves outside defined regions may fall outside data residency commitments. This policy locks the production account to specific regions: JSON { "Effect": "Deny", "Action": "*", "Resource": "*", "Condition": { "StringNotEquals": { "aws:RequestedRegion": ["us-east-1", "eu-west-1"] } }, "NotAction": [ "iam:*", "organizations:*", "route53:*", "budgets:*", "waf:*", "cloudfront:*", "globalaccelerator:*", "importexport:*", "support:*", "trustedadvisor:*" ] } EBS encryption is not enforced by default in all account configurations. This policy makes an unencrypted volume impossible to create in the production account: JSON { "Effect": "Deny", "Action": "ec2:RunInstances", "Resource": "arn:aws:ec2:*:*:volume/*", "Condition": { "Bool": { "ec2:Encrypted": "false" } } } Cross-Account Access: The Pattern That Doesn't Create New Gaps Multi-account architecture introduces a problem engineers feel immediately: how does anything talk to anything else? A CI/CD pipeline in the Shared Services account needs to deploy to production. A developer needs read access to production logs during an incident. A monitoring service needs metrics from all accounts. The answer is cross-account IAM roles with tightly scoped trust policies. A role created in the production account with minimum required permissions defines a trust policy that allows only specific principals from specific accounts to assume it, and only under specific conditions like MFA or an external ID: JSON { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::SHARED-SERVICES-ACCOUNT-ID:role/DeploymentRole" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "deployment-pipeline-prod" } } } ] } The deployment role in the Shared Services account can assume the deployment role in production - but only that role, only from that account, and only with the correct external ID. A developer's personal IAM credentials cannot assume it. An engineer who compromises the development account cannot use that foothold to pivot into production. This pattern creates cross-account access without creating a backdoor through the account boundary. The boundary holds because the trust relationship is explicit, narrow, and auditable through CloudTrail - every role assumption generates a log entry in both accounts. What This Architecture Makes Provable The operational argument for multi-account PHI isolation often focuses on security. The architectural argument that matters more for engineering teams dealing with audits and enterprise security reviews is about provability. In a single-account setup, proving that a developer did not touch production PHI during a given period requires auditing IAM policies, CloudTrail logs, and access history, and then arguing that the policies were correctly configured and consistently enforced throughout the period. There is always a gap between what the policy said and what actually happened, and that gap is what auditors probe. In a multi-account setup, the same question has a simpler answer. The developer's credentials are scoped to the development account. The development account has no access to the production account's resources. Access to production PHI requires a separate role assumption that is logged, requires separate credentials, and would appear immediately in CloudTrail. You are not arguing that the configuration was correct. You are pointing to an architectural boundary that makes the question moot. This shift from arguable to verifiable is what separates teams that sail through security reviews from teams that spend three weeks responding to follow-up questions. The Operational Overhead Is Smaller Than It Looks The most common objection to multi-account architecture from engineering teams is overhead. More accounts means more IAM configuration, more billing to reconcile, more consoles to log into. In practice, this friction is front-loaded and largely disappears once the structure is in place. AWS Control Tower reduces the account provisioning overhead significantly - new accounts inherit the correct SCP structure, logging configuration, and security baseline automatically. Account Vending Machine patterns built on top of Service Catalog or Terraform can provision a correctly configured new account in minutes. After the initial setup, adding a new account is not significantly more work than adding a new VPC. The billing concern is resolved through AWS Organizations consolidated billing, where all accounts roll up to a single payment method with unified cost visibility. The console switching concern is resolved through IAM Identity Center, which provides a single sign-on entry point across all accounts in the organization. The overhead that remains is real but small. The alternative - treating IAM policies inside a single account as the primary PHI protection mechanism - creates ongoing operational overhead that grows with the team and never fully goes away. Final Thoughts PHI workload isolation is an architectural problem, not a policy problem. IAM policies enforced inside an account are only as reliable as the operational discipline of the team maintaining them. Account boundaries enforced by SCPs at the organizational level are reliable by construction — they hold regardless of what happens inside the accounts they protect. The multi-account structure described here is not a compliance checkbox. It is the architecture that makes the access control claims in a security review actually true rather than approximately true with caveats. When an auditor asks how you prevent developer access to production PHI, the strongest answer available on AWS is an account boundary that the developer's credentials cannot cross. Building that boundary is a week of work. Not building it is a permanent source of audit findings.

Once upon a time, site reliability engineering rested on a linear assumption: monitor more, detect early, and you’ll recover faster. The rise of alert fatigue makes modern SRE teams realize otherwise: Ramadass's (2025) paper, Building an AI-Powered Observability Pipeline for Modern System Reliability, cited research that discovered that: More than two-thirds (82%, actually) of institutions experience alert spikes constantly.Most traditional monitoring tools generate approximately 2,100 alerts daily, with about 70% of them unnecessary and safe to ignore.66% of SRE professionals stated that increased false alerts lead to fatigue, potentially causing them to miss serious issues. How Should We Describe This Situation? Vigilance or Noise? Collaborative systems such as SaaS, third-party APIs, and microservices enhance the degree of observability and notification within systems. Everything is monitored, and occasionally these dependencies may duplicate alerts. When systems request superhuman attention, on-call engineers become fatigued rather than lazy or sloppy. Instead of swift action, alerts are responded to with mistrust. Reliability vs. Experience vs. Metrics Traditional alerting metrics follow traditional reliability practices, that is, error rates, uptime percentages, latency, etc. Although these are essential, they are not actual mirrors of how operators or users experience reliability. Operators may expect reliable alerting to inform decisions, while users may simply define reliability as how well a system enables them to fulfill their intentions. If alerts do not clearly connect to the user experience, there is a gap between detection and action. Over time, the gaps lead to fatigue. On-call engineers begin to “reasonably” ignore these alerts. Why worry over alerts that are not logically related to user outcomes? They may assume. Over time, organizations may end up paying dearly for real issues because alerts were missed or delayed. An On-Call Engineer Experience Here is a typical example of a system design problem an on-call engineer or SRE team may face: 01:15 AM Alert: Latency spikes on a third-party API.01:16 AM Alert: Retry queues are filled.01:16 AM Alert: Timeout alert storms on three dependencies.01:17 AM Alert: Error-rate notification on unrelated endpoints.01:18 AM Alert: Memory and update alerts. And this sequence of alert storms continues, with the on-call engineer receiving more than 20 alerts in just four minutes. The system seems to pass standard observability SRE practice. But what about the long-run reliability suspicions that the bugging signals may create? In this case, the teams are not just grappling with response speed but also with the amplification of confusion when critical alerts are mixed with non-actionable ones. When Detection Outpaces Interpretation We can’t rule out the fact that monitoring in the past decades has taken an advanced leap. And we might be at its cloying stage, where system detection software is outpacing on-call engineers’ interpretation. Systems are “wonder-full” when it comes to identifying when something seems “off.” However, they rarely give explicit descriptions to aid SRE teams’ understanding. An alert can indicate that a queue has exceeded its depth, but may not categorically state whether the issue is temporary or actionable, or whether users are affected. This occurrence spans dozens of dependencies, each with its own signal. The on-call engineer is kept puzzled about the best action to take at the right time. Hence, a reliable response could be excessive caution or delay as the engineer seeks to clarify the situation. The users are negatively impacted. Although the system met technical observability SRE standards, it failed operationally due to its opacity. The Hidden Cost of Alert Overload We rarely see the outcome of alert fatigue overnight. Its effects build up. Delayed response time accumulates. The aftermath incident review loses credibility. Engineers are skeptical of alerts and hesitate to decide first whether they are real or false. The cultural cost of alert fatigue is that on-call roles become a burden SRE teams endure rather than enjoy with a sense of responsibility. In the long run, engineers may feel they have no control over issues due to the confusion that multiple alerts create. Ironically, the same reliability problems that alerts were designed to solve are what they quietly create. Are Alerts Creating a False Sense of Safety? Lots of alerts may seem like a good thing or a sign of strong monitoring at first glance. But here is the truth: alerts could be hiding actual risk. As every deviation is notified, critical and minor alerts blend in. Teams begin to feel alert fatigue and delay response. Then, real problems begin to breed behind the scenes. Remember how SLAs could paint an illusory picture of safety? Similarly, alert volume could do so. Therefore, your SRE team should bind these caveats as the core of their modus operandi. Alerts shouldn’t replace action.Alerts shouldn’t be unsorted (by machines or humans).Alerts shouldn't be discarded. Alerts are signs that our systems need attention, and we should never be tired of listening. SRE Teams Designing Systems that Alert Smartly High-quality systems respond efficiently when dependencies fail. Instead of creating panic, they automatically degrade. SRE teams could design circuit breakers that could inhibit alert storms before they explode. They could also install bulkheads to prevent a single failure from spreading. There could be alert limits and a summary of conditions that resolve the problem of spamming. Instead of relying on metrics, system engineers could set up composite alerts that describe system states. For instance, it’s clearer if a system alert indicates, “Checkout degraded because of latency in payment dependency.” This composite alert is better than 7 alerts that say “Checkout Timeout.” The former shows impact, cause, scope, and urgency. Clarity clears fatigue. Noise does the opposite. Redesigning SRE: Human Reliability That Quells Alert Fatigue We have seen that technical designs may be great, yet other aspects of SRE remain wanting. One such area that could resolve a system design problem is humaneness. To avoid alert fatigue, our design choices must acknowledge human limitations. Therefore, we should accept that some alerts may not require immediate response. Conversely, not every anomaly should trigger an alarm. Understood silence could sometimes be a golden sign that nothing critical is wrong. Advanced SRE teams do not focus on events (or every deviation) but on the states of the system or infrastructure. They are guided by the question: What conditions really impact users, business objectives, or the system's overall health? To achieve this, engineers need to balance product understanding with technical operations. Then they can give a human touch to their designs. Designing systems for human reliability requires a high level of discipline. Site reliability engineers have to continually review, refine, and repair alerts and their trigger commands. Systems are like living organisms that need constant feeding of updates. The evolving nature of alerts could make a helpful one-time alert redundant or harmful in six months. On-Call as a Reliability Interface of SRE No doubt, humans have a role to play in ensuring reliability, but system designs that depend on heroic actions are built not with resilience but with fragility. Reliability is truly achieved when on-call engineers are guided by predefined scripts, models, runbooks, signals, and interfaces. These reduce the tendency to resort to fallible improvisations when issues arise. On-call engineers often take the appellation of “last point of call.” A careful look at their roles shows that they are intermediaries among complex systems, user experience, and consequences. We can thus see that the role of on-call engineers extends beyond problem resolution to stewardship. Conclusion Alert fatigue is a design problem. It often arises when on-call engineers prioritize detection over interpretation, or technical workability over user experience. The dependencies of modern SRE teams make it necessary to align technical alerts with human capability. Alert storms could wear out hardworking engineers who need to take a break. So, system designs need to account for human limitations, recognize that runbooks are better than on-the-spot improvisation, and prioritize clarity over opacity. Designs that account for these factors reduce or eliminate fatigue and preserve the very essence of alerts. In summary, reliability goes beyond resolving many problems to responding to what matters most. When teams can always trust their alerts, they will be more likely to follow up on new cases.

Originally, back-end and front-end Site Reliability Engineering (SRE) were owned by teams. They code the programs, set up databases and infrastructure, and quickly spring to action at the beep of any anomaly. The advent of code vs no-code infrastructure, SaaS, API dependencies, third parties, and other modern systems seems to be eroding this authority. Mainstream and underdog companies now often leverage the significant advantages of outsourcing, collaboration, or delegation, which are usually accompanied by a silent clause: no or partial control. Unlike in previous systems, modern production is largely assembled rather than built from scratch. For example, a conventional SaaS product is built on interdependencies among payment processors, outsourced data infrastructure such as Amazon Web Services (AWS), messaging services, web hosting, design, AI inference APIs, authentication providers like Google, and more. These useful platforms and products are essentially outside teams' control stations, even though they critically impact users' experience. When they function effectively, you share the glory with the platforms. But when there is a system blackout, your users put you on your toes, even though you have no direct access to resolve the problem on time. Therefore, we shall be exposing SRE practices in platform-SaaS and API-dependent systems and how reliability is getting beyond the control of engineering teams and companies. Why Classical SRE Practices May Fail One major downside of SaaS and dependency on external platforms is that reliability control is often assumed to be in a team's hands, whereas it has been bargained. However, teams must reckon with the fact that the case is reversing. For example, traditional SRE models once alleged that: Service Level Indicators (SLIs) focus on availability or internal uptime and latency.Error budgets arise from changes teams make or deploy.Runbooks still suggest that teams can immediately reconfigure or directly work on faulty components. All these are becoming past cases, especially in platform-SaaS systems. You can have a system indicating 99.99% or even 100% uptime on the back end, while new users are struggling to sign up, probably because an authenticator provider is not fully functional. Dashboards and control panels may indicate green, but in reality, third-party payment APIs have been degraded. A New Definition of Reliability in Operating SRE Practices To resolve the new problem in site reliability engineering (SRE), there needs to be a conceptual shift from component health to an integrated, continuous user experience. Therefore, teams need to undergo a paradigm shift away from questions such as "Is our CPU working maximally?" “Is our API up?” “What are the error rates?” Instead, we should inquire: “Are users checking out seamlessly?” “How fast can they authenticate?” “Can they use the SaaS product to perform its key function?” These types of outcome-based questions span interdependent platforms beyond your full control. The login SLI needs to work with the identity provider; otherwise, its output is meaningless. If the checkout SLO skips payment authorization, then it's both fishy and unreliable. True, there may be some internal errors in a reliable system, but what really matters is an integrated multiplatform experience that the user enjoys. Error Budgets? An SRE Practice to Revisit How many teams would love error budgets to disappear when they give up control? But that’s not so. Instead, they are molecularized. When components of your systems are outsourced, the error budget doesn’t just fade away; it is instead transferred to the interdependent platforms. So, it’s better to plan for the fact that SaaS and API providers will consume some of your reliability budget. Doing so keeps you a few steps ahead and protects your business in the long run. Reliable SRE teams make decisions such as allocating part of their error budget to certain dependencies, setting acceptable parameters for degradation, and defining specific steps to take when a dependency exceeds the stipulated budgets. Here’s an example you can adapt: “We will accept payment authorization failure of 0.0% to 0.2% if it is caused by dependency instability. If it goes above that, we will turn on delayed capture or turn off promotions.” This SRE approach keeps you ready for downtime, as your systems automatically switch to planned or budgeted actions rather than relying solely on integrated platforms. What to Do When Failures Beyond Your Control Arise Actually, some failures may seem beyond your control. The more you attempt to resolve them, the more amplified they become. At this point, your team must adapt to the savvy absorption of such situations. Instead of focusing solely on retrial in an SRE approach, your team needs to design its processes and platforms. This could include failing selectively through circuit breakers, failing fast with timeouts, or failing visibly by keeping users informed. Some core settings should always remain non-negotiable and on standby. These could include the following: Read-only modes/cachesBulkheads that prevent a failure avalanche.Automated circuit breakersDeferred processing These reliable practices ensure there is some form of controlled uptime even when operations seem interrupted. Laser Observability That Proves Reliability In traditional SRE observability, the service boundary is usually the ultimate, but in most modern integrated SaaS platforms, this could be insufficient or worse, dangerous. Operators need to be aware of the actual dependency that is failing, how it is failing (e.g., errors or throttling), and how the failure affects the user experience. Accurate observability for platform-SaaS and API-dependent systems requires these four provisions: Specific dashboard and internal metrics for each vendor.SLI monitoring at the dependency level.Parallel tracing of all outbound calls.Simulation of real-time user experience and workflows. Essentially, whenever there is an emergency, operators should be able to promptly identify whether the source is internal or external. Accuracy and clarity facilitate swift response. Responding to Incidents Without Ownership Another distinct characteristic of modern SRE practice in platform-SaaS is how incidents are responded to. Without ownership, you often cannot debug on your own, roll back a bad deploy, or directly manage other issues. However, you can choose how your system responds by identifying when certain features are disabled, when signals to activate degraded modes are sent, when high traffic is redirected or shed, or when to notify users. To maintain reliability, incident response relies on runbooks to inform decisions. The following questions could help convert the technicality of runbooks to practical solutions: What is the impact on the customer?In what ways can we respond harmlessly?What can we reverse?What should we communicate externally? These questions help resolve incidents, mitigate losses, and intertwine reliability with sound judgment. Is Safety an Illusion in SLAs? SLA providers often readily contract for financial compensation when losses arise, but seldom give absolute reliability guarantees. You may not always expect vendors to consistently meet your availability goals or resolve an avalanche of outages. Safety is a critical consideration when building systems, because when users lose trust in a brand, compensation may not be able to redeem it. Therefore, advanced teams do not consider SLAs as safety nets but as risk pricing. They understand that contractual credits cannot replace trust, brand image, and some almost irredeemable damages. Human Factors in Platform-SaaS and API-Dependent Systems Dependency failures often escalate when cognitive load increases. There could be degraded performance, timeouts without error indicators, partial success, or inconsistent system behavior. Operators may not only focus on machines when dashboards lag or seem to lie. They examine the logs, failure history, or commands. Teams have to design systems with overrides and predictable degradation paths, and observability tools are beyond the failure systems. Reliability goes beyond the correct function of software; it's also about human operations. How Your SaaS and API Platforms Can Imbibe “Good” SRE Practice Effective SRE practices are modern. The following attributes know saas products and API-dependent platforms: Acknowledgment of lack of control very early.Ensuring reliability is embedded in the design.Measuring the outcomes of each SRE criterion or target, instead of just the components.Giving priority to clarity instead of trying to model or control everything because you do not own all the components.Making engineering and operations decisions and products as an integrated whole.Preparing for degradations as inevitable procedures when things fail. Your systems can be reliable if you anticipate failure and accept the reality. Conclusion Modern platform-as-a-service (SaaS) operates in a reliability-without-control manner, leading solid SRE teams to accept that they need to adapt when failures occur. It's simple logic: if you don't absolutely own everything end-to-end, then prepare for the worst: each dependency might fail. It's all about keeping the trust of your users and protecting your brand image.

Every few weeks, someone on my team, or in a client meeting, asks me the same question: "Which cloud should we use for our AI workloads?" I have been building enterprise integrations for over fourteen years now, and lately most of my time goes into RAG pipelines, vector databases, and agentic orchestration on top of these platforms. So I get this question a lot, and honestly, there is no single right answer. The right cloud depends on where your data already lives, what your compliance team will accept, and which models your architecture actually needs. In this article, I want to walk through the three big players, AWS Bedrock, Google Vertex AI, and Microsoft Azure AI Foundry, and share what I have learned working with these platforms in real enterprise settings, not just from reading marketing pages. AWS Bedrock Bedrock started as a model marketplace back in 2023, and it has grown into a full platform with Guardrails for content filtering, Knowledge Bases for RAG, and AgentCore for building agentic workflows. What I like most about Bedrock is the sheer breadth of models available behind a single API. You get Claude from Anthropic, Llama from Meta, Mistral, Cohere's Command models, and Amazon's own Nova family, all through one consistent interface. If your architecture needs to swap models without rewriting your integration layer, Bedrock makes that easier than the other two. Pros: Broadest model catalog of the three, so you are not locked into one vendor's models.Strong identity and governance story if you are already running on AWS, since it plugs directly into IAM, CloudTrail, and Macie.Bedrock is one of the few places where you get Claude with enterprise indemnification, which matters a lot when legal teams get involved.Provisioned throughput options give you predictable latency for production workloads that cannot tolerate spikes. Cons: If your organization is not already AWS-native, the onboarding curve is steeper than it looks.Cross-cloud portability is basically nonexistent. A model you fine-tune on Bedrock does not export cleanly to Vertex AI or Foundry. That is a real switching cost you should plan for on day one, not something to figure out later.Some of the newer agentic tooling is still maturing, so documentation gaps show up more than I would like. Google Vertex AI Vertex AI feels different from the other two because Google's DNA here is research first. If your team cares about multimodal capability, or you want access to Gemini models the moment they ship, Vertex AI tends to be ahead. It is also the strongest option if your data already lives in BigQuery, because the integration between Vertex and BigQuery for feature engineering and MLOps pipelines is genuinely smooth. Pros: Best fit for teams doing custom model training, not just calling a hosted API. AutoML and the broader MLOps tooling cut training time noticeably compared to the other two.Tight coupling with BigQuery is a huge advantage if your organization already runs its analytics there. You avoid a lot of data movement overhead.Gemini-first multimodal workflows, plus Google Search grounding for agents, which is something neither Bedrock nor Foundry offers natively.TPU support gives real throughput advantages for heavy batch processing. Cons: If your organization is not GCP-centric already, the value proposition weakens fast. You end up paying a data-gravity tax to move information into Google's ecosystem.Governance and compliance tooling, while solid, is not as battle-tested across regulated industries as AWS's certifications.The agent ecosystem, while improving, still trails Bedrock's AgentCore and Foundry's Azure AI Agents in terms of enterprise adoption stories I have personally seen. Azure AI Foundry Foundry, formerly Azure AI Services, is Microsoft's rebranded and expanded platform, and it is the one I have written about before because it is what my own recent client work has centered on. If your enterprise already lives inside Microsoft 365, Entra ID, and Azure infrastructure, Foundry removes almost all of the identity and governance friction you would otherwise deal with. That matters more than people expect once you are past the proof of concept stage and into actual production rollout with security review. Pros: Deep Microsoft 365 and Entra ID integration means your existing enterprise approvals and identity workflows extend naturally into your AI layer.Strong OpenAI-led model access, since Microsoft's partnership with OpenAI gives Foundry early and deep access to GPT-family models.Hybrid deployment options are genuinely better here than on the other two platforms, which matters if you have on-prem systems you are not ready to fully cloud-migrate.Roughly three-quarters of Fortune 500 companies already run on Microsoft's stack, so for a lot of enterprises Foundry is simply the path of least resistance. Cons: Model breadth is narrower than Bedrock's catalog, so if you need a specific non-OpenAI model family, you may find yourself stitching together a secondary platform anyway.Because it is tied so closely to Azure compute pricing, cost predictability requires more upfront modeling than teams expect.Some newer agentic and orchestration features are still catching up to what AWS has shipped with AgentCore. So Which One Should You Actually Pick? Here is the honest answer I give in client meetings: do not choose based on a benchmark screenshot or a features table. Choose based on where your data already lives and where your governance and compliance story already works. If you are AWS-first and want maximum model flexibility, go with Bedrock. If you are Microsoft-heavy and need your AI layer to inherit existing Entra ID and 365 approvals without a fight, Foundry is the path of least resistance. If your analytics already lives in BigQuery and multimodal Gemini capability is core to your roadmap, Vertex AI earns its place. What I am increasingly seeing among the teams I work with is a hybrid pattern. A primary cloud handles the bulk of regulated workloads, and a secondary cloud gets called in only when a specific model family is not well supported on the primary platform. It is not the cleanest architecture on paper, but it reflects how fast this space is still moving. None of these three platforms is standing still, and the leader on any given feature this quarter is not guaranteed to hold that spot by next year. My suggestion, whichever cloud you land on: build your RAG and orchestration layer with enough abstraction that swapping the underlying model provider is a configuration change, not a rewrite. That single decision will save you more pain than picking the "right" cloud ever will.

By Balaji Venkatasubramaniyar CORE

Senior data engineers are trained to be skeptical of proprietary platforms. When I entered a Palantir Foundry training bootcamp, I expected to find a slow, expensive alternative to the mature tools I know on AWS and Azure. What I found instead was a platform built for a radically different user, one who cannot write SQL but needs answers now. I want to write about what I actually observed honestly, including where I think the hype is justified and where I think it is not, because most Foundry content I have seen is either from Palantir's own marketing or from practitioners so embedded in the platform they have forgotten what it was like to come to it fresh. I am writing this while that perspective is still clear. The Speed Thing Is Real The surprise that hit me hardest was not a feature. It was pace. During the bootcamp we worked across a range of tasks: connecting data sources, building transformation pipelines, setting up workflows that business users could interact with directly. To make this concrete: building a pipeline that ingested data from multiple sources, applied transformations, and exposed the output to business users took only hours in Foundry. On a standard AWS or Snowflake stack with dbt and an orchestration layer, a comparable setup typically runs to a full sprint for a small team, not because of any single hard step, but because of the coordination overhead between tools. I want to be careful about what I am and am not claiming here. This was a structured training environment with guided examples, not production infrastructure with real enterprise complexity and legacy constraints. The comparison is not controlled. But the direction of the difference was clear enough that I took notice. Foundry's Pipeline Builder abstracts away a lot of the coordination work that consumes time in a more assembled stack. Whether that advantage holds at full scale is a question I cannot answer from a single bootcamp, but it is worth asking seriously. The honest counter-argument: speed in a training environment does not always translate to speed in production. A well-resourced engineering team that already knows Snowflake deeply can move fast too, without the overhead of learning a new paradigm. If your team is highly capable on your current stack, the productivity gain from switching may not justify the learning curve cost. "Tasks I would have planned for a full day on my normal stack were done in a couple of hours." Who Actually Benefits Most However, raw speed is not the platform's most disruptive feature. The more I used it, the more I realized that the real value of that speed is not for engineers. It is for the people who are usually waiting on us. The more I worked with Foundry during the training, the clearer it became that the people getting the most out of it in the room were not the engineers. They were the non-technical participants, the analysts, the operations people, the business users who in a traditional stack would be waiting for an engineer to build them something before they could interact with data at all. Foundry's ontology model, the way it creates a shared semantic layer that different types of users can navigate without writing code, is differentiated from what I work with on AWS, Azure, and Snowflake. On those platforms, self-service data access for non-engineers is possible, but it takes deliberate, often significant engineering effort to expose data in a way that non-technical people can actually use. In Foundry, it felt closer to the default. If I were advising an organization on whether to consider Foundry, the first question I would ask is: what percentage of the people who need to interact with your data can actually write SQL? In organizations where more than half of business analysts and operational users cannot write code, the engineering burden of building self-service access on a traditional stack becomes a recurring, compounding cost. That is the environment where Foundry's default self-service capabilities start to justify serious evaluation. The counter-argument here is worth stating directly: a strong, well-resourced data engineering team could build a better, more tailored self-service layer on Snowflake in the same time it takes to master Foundry's ontology. If your organization has that team and the patience to build the right abstractions, the open platform may serve you better in the long run. Foundry's self-service advantage is most compelling when you do not have that engineering capacity, or when the number of non-technical users is large enough that a custom-built solution would require constant maintenance. The Cost Reality Palantir does not publish list pricing for Foundry. Everything is negotiated. The platform uses a core-based licensing model, meaning you pay based on the computational capacity (server cores) allocated to the platform rather than by the number of users. Based on publicly available government procurement records, core-based licenses start at roughly 66,000 pounds per server core per year, with no additional per-user fees on top. Solution-based use case licenses, which bundle implementation and support, start at 250,000 pounds at entry level and scale significantly from there depending on data complexity, user base, and operational scope. What this means practically is that Foundry's cost is not a fixed number you can evaluate on a spreadsheet. It is a negotiation. According to procurement advisory analysis of Palantir Foundry negotiations conducted between 2024 and 2025, annual platform fees for comparable mid-size deployments varied by a factor of two to three depending purely on negotiation posture (Redress Compliance, 2025). The leverage comes primarily from having a credible, costed alternative, which for most organizations means Databricks or Snowflake with named engineering owners and a realistic build timeline. Organizations that enter Palantir conversations without that alternative built tend to pay significantly more for the same deployment than organizations that do. "The leverage in the Foundry cost negotiation comes primarily from having a credible, costed alternative built before you walk in." My honest assessment after the bootcamp is that the cost is hard to justify for smaller organizations or simpler use cases. If a well-designed Snowflake environment can meet your data engineering needs with dbt and a standard BI layer on top, Foundry is probably not the right answer, and the delta in platform cost will buy you a lot of engineering time on the stack you already know. The calculus changes for large enterprises with complex, multi-team data environments and a significant population of non-technical users who need meaningful data access. What I Would Tell a Data Engineering Leader A few things I would want another senior data engineer or engineering leader to know before evaluating Foundry: Do not evaluate Foundry on pipeline performance alone. That is not its primary differentiator. Compare it to Snowflake or Databricks on what it does for the non-engineer users in your organization, not on compute efficiency.Build your alternative cost model first. Whatever your current stack is, cost out what it would take to build the data product capabilities Foundry promises on that stack, with your own team. That number is your negotiating anchor.Take the learning curve seriously. Foundry has a broad ecosystem: the ontology model, Pipeline Builder, Code Repositories, AI integrations, and coming to it fresh from a traditional data engineering background takes real adjustment. The training helped, but it is not a platform you pick up in a day.Be specific about who your users are. Foundry earns its cost fastest in environments where non-technical users need to do more with data than your current stack allows. If your users are primarily technical, the value proposition narrows considerably.Negotiate the second contract inside the first. Procurement analysis consistently shows that organizations that lock in phase two pricing before signing the initial contract pay significantly less per added use case than those who do not. Treat the pilot as the deal. The Honest Summary I came to Palantir Foundry expecting to be underwhelmed. I was not. But understanding its value requires a paradigm shift for any engineer raised on AWS or Snowflake. Evaluate Foundry not as a faster pipeline tool, but as a platform for organizational data literacy. For enterprises drowning in data but starved of accessible insights, it is a compelling, if expensive, contender. For everyone else, the tools you already have remain the better investment. The challenge is being honest enough with yourself to know which bucket your organization falls into.