Cloud Architecture

Cloud-native systems now create far more machine identities than human ones. Security strategies built around workforce identity are no longer sufficient. Here's what engineering leaders should build instead. The Breach That Didn't Need a Password On August 8, 2025, a threat actor now tracked by Google's Threat Intelligence Group as UNC6395 began quietly moving through the Salesforce instances of hundreds of companies. No phishing email landed in an inbox that day. No password was cracked. No multi-factor prompt was bypassed with a fatigue attack. The attacker simply had something better than a password: a valid OAuth token, stolen months earlier from Salesloft's GitHub account, that let it impersonate the Drift chatbot integration and act with all the trust that integration had been granted. Over the following ten days, the group ran automated Salesforce Object Query Language searches against more than 700 organizations — Cloudflare, Zscaler, Palo Alto Networks, and PagerDuty among them — harvesting account records, support case text, and, crucially, the AWS keys and Snowflake tokens that customers had pasted into support tickets months earlier. Google's investigation later found the same stolen tokens had reached into Google Workspace mailboxes too. Cory Michal, CSO at AppOmni, put his finger on what made the campaign notable: it wasn't a single lucky break but a methodical operation against hundreds of tenants using nothing but credentials the tenants themselves had issued to a vendor they trusted. That's the detail worth sitting with. Every access control that companies had built around human identity — MFA, conditional access, session monitoring, SSO — was irrelevant to this attack, because no human ever logged in. The identity that mattered was a machine's, and almost nobody was watching it the way they watch people. This wasn't an isolated case. In the same twelve months, a compromised API key issued to a DOE staffer gave a stranger standing access to more than 50 large language models at xAI — and stayed active for days after the exposure was discovered, according to reporting from KrebsOnSecurity. A supply-chain attack against the widely used tj-actions/changed-files GitHub Action, relied on by over 23,000 repositories, scraped AWS keys, GitHub tokens, npm credentials, and private RSA keys directly out of CI/CD workflow logs. GitGuardian's 2026 State of Secrets Sprawl report counted 28.65 million new hardcoded secrets pushed to public GitHub repositories in 2025 alone — a 34% jump year over year — and found that AI-assisted commits leak secrets at roughly twice the baseline rate of human-written ones. None of these incidents required a zero-day. They required an organization to have created a machine identity, granted it access, and then stopped paying attention to it. That is now the default failure mode of cloud security — and it's a failure mode that identity programs built for humans were never designed to catch. Section 1: Identity Has Already Changed Underneath Us For most of the last two decades, "identity and access management" meant managing people: employees, contractors, customers. A person logged in, proved who they were, and was granted access based on their role. The infrastructure existed to serve human judgment. That model quietly stopped matching reality. In a modern cloud environment, the majority of authentication events aren't between a person and a system — they're between systems. A pod in a Kubernetes cluster calls another pod. A CI/CD pipeline authenticates to a cloud provider to deploy an artifact. A SaaS integration holds an OAuth token that lets it act on a company's behalf indefinitely. Each of these is an identity in every meaningful sense — it can be granted permissions, it can be revoked, it can be stolen — but almost none of them are managed with the rigor applied to a human employee's badge. The mechanisms behind this shift are now familiar to anyone running production infrastructure: Kubernetes service accounts that authenticate workloads to the API server, workload identity federation that lets a pod assume a cloud IAM role without a stored credential, SPIFFE and SPIRE issuing cryptographically verifiable identities to workloads at runtime, OAuth client-credential grants powering service-to-service calls, and service meshes like Istio wrapping every internal request in mutual TLS. Layer on top of that the identities created by CI/CD systems, and it becomes clear that a mid-sized cloud environment can easily contain ten or twenty machine identities for every human one. Security researchers at IDMWorks, reviewing the identity breaches of the last three years for their 2026 NHI Reality Report, described the pattern bluntly: these attacks succeeded through poor governance, not sophisticated malware. There was no payload to detect — just valid credentials doing exactly what valid credentials are allowed to do. That's a much harder thing to catch than a virus, because there's nothing anomalous about the code path. The only thing that's wrong is which entity is walking it. Section 2: Why the Existing Security Model Fails Here Identity and access management built for people assumes a handful of things that simply don't hold for machines. It assumes credentials are issued to a known, accountable owner. It assumes a login event is rare enough to be worth alerting on. It assumes a compromised credential will eventually show up in unusual behavior — an impossible-travel alert, an after-hours login, a new device. None of that transfers cleanly to a service account. IDMWorks' research is direct about the resulting blind spot: a service account that authenticates ten thousand times a day isn't behaving anomalously — that's just Tuesday. Detecting misuse requires knowing what a credential is supposed to be doing well enough to notice a deviation, and almost no organization has that baseline built for its non-human identities the way it does for its people. The ownership problem compounds this. Aembit's running catalog of non-human identity breaches documents a 2025 flaw in a major identity provider that let anyone holding a valid API key enumerate every OIDC application in a tenant and pull its client secrets — a bug that, if exploited, would have let an attacker impersonate entire applications and move laterally across an organization's stack. It was responsibly disclosed and patched, but it illustrates how identity providers themselves can become breach multipliers the moment a machine credential leaks. Then there's lifespan. GitGuardian's research, cited in Snyk's 2026 analysis of the secrets sprawl problem, found that private repositories are six times more likely to contain hardcoded secrets than public ones — largely because private repos get cloned, forked, and handed to contractors without anyone revisiting what's inside them. And because git's data model is append-only, a secret committed and later deleted in a follow-up commit should still be treated as exposed; it lives on in history whether or not it's still visible in the latest diff. The legal exposure is no longer theoretical, either. In United States v. Sullivan, Uber's former Chief Security Officer was criminally convicted of obstruction of justice for concealing a 2016 breach that began with hardcoded AWS credentials sitting in a GitHub repository — credentials that let attackers pull data on 57 million riders and drivers. The Ninth Circuit's 2025 ruling upheld that conviction, establishing that executives can face personal criminal liability for how they respond to a credential-based breach, not just for the breach itself. That should recalibrate how seriously engineering leadership treats "just another leaked API key." An Honest Name for the Problem: Machine Identity Debt Engineering teams already have a vocabulary for the gap between "shipped quickly" and "built correctly" — they call it technical debt. There's no equivalent term for the identical pattern happening in identity, so let me propose one: machine identity debt. Technical debt accumulates in code: shortcuts taken under deadline pressure that someone eventually has to pay down. Identity debt accumulates in trust: every API key issued and never revisited, every OAuth grant approved by someone who's since left the company, every IAM role created with "just give it admin, we'll fix it later" and never fixed. None of it shows up in a sprint retro. None of it fails a build. It just sits there, compounding, until an attacker finds it and collects the interest all at once — which is close to a literal description of what happened to the 700-plus organizations caught in the Salesloft Drift breach, where OAuth grants approved months or years earlier turned out to still carry far more reach than anyone had tracked. A rough way to think about what's accumulating: Plain Text Machine Identity Debt ≈ long-lived credentials with no expiration policy + service accounts no longer tied to an active workload + OAuth grants no one has reviewed since approval + secrets discovered in tickets, chat, and docs rather than a vault + IAM roles scoped broader than the task requires + any machine identity with no accountable human owner This isn't a precise formula you can drop into a dashboard query today — treat it as a checklist for a conversation, not a KPI. But naming each line item is useful, because each one is independently measurable, and most organizations have never measured any of them. When enough of this debt accumulates that nobody can produce an accurate answer to "what machine identities exist, who owns them, and what can they reach" — that's not an IAM maturity gap anymore. It's identity bankruptcy: the point where inventory, ownership, and trust have diverged so far from reality that incremental cleanup stops being realistic and the organization needs a forced reconciliation, usually triggered by an incident rather than a planning cycle. The mechanism that gets organizations there is worth naming too. Every new SaaS integration, every GitHub Action, every Terraform module, every AI agent granted API access mints a new unit of trust — a new thing the organization implicitly promises to govern. Nobody budgets for governing it; the integration just gets approved because it unblocks a project. Multiply that across a growing stack and you get something like trust inflation: the total quantity of trust an organization has extended growing faster than its ability to actually track or revoke any single unit of it. Eventually a credential's nominal access — what the ticket said it was for — and its real access — everything it can actually still reach — drift far enough apart that the gap itself becomes the attack surface. None of these terms are industry standard — I'm proposing them here because the pattern needed a name and didn't have one. Judge them by whether they make the problem easier to talk about, not by whether you've heard them before. Section 3: A New Boundary — Adaptive Machine Trust Architecture If the perimeter used to be defined by "who logged in," it now has to be defined by a different question: can this specific workload be trusted, right now, to do the specific thing it's asking to do? That's a shift from identity as a static credential to identity as a continuously re-evaluated claim. A workable framework for this — call it Adaptive Machine Trust Architecture, or AMTA — rests on a small number of principles that reinforce each other: Continuous verification. A workload's identity is checked at the moment of each request, not once at startup. Trust isn't a badge you're handed at the door; it's re-earned per transaction. Cryptographic workload identity. Instead of a static API key sitting in an environment variable, a workload is issued a short-lived, cryptographically verifiable identity document — the SPIFFE Verifiable Identity Document (SVID) model is the clearest existing implementation of this idea — that ties the identity to what the workload is, not to a secret it happens to be holding. Just-in-time authorization. Access is granted for the duration of a task and expires automatically, rather than being provisioned once during a rushed deployment and left in place indefinitely, which is precisely the pattern IDMWorks identified as the root cause of most CI/CD credential compromises. Policy-driven trust decisions. Authorization decisions are externalized to a policy engine that can evaluate context — the requesting workload's identity, its recent behavior, the sensitivity of the resource — rather than being baked into application code as a hardcoded allow-list. Identity lifecycle management. Every machine identity has a documented owner, a defined purpose, and an expiration path. The absence of exactly this — what IDMWorks calls "no ownership model" — is the single most commonly cited root cause across the non-human identity breaches of the last three years. Continuous attestation. The system periodically re-proves that a workload is still what it claims to be — still running the expected code, in the expected environment — rather than trusting a credential indefinitely once it's issued. None of these principles is exotic on its own. What's new is treating them as a single coherent architecture for machine trust, instead of a scattered collection of best practices that get implemented inconsistently across teams. Section 4: What Implementation Actually Looks Like The tooling to build this exists today, and it's more mature than most security teams realize. SPIFFE and its reference implementation, SPIRE, provide the identity layer: workloads receive short-lived X.509 or JWT SVIDs based on attested properties of the environment they're running in — the specific pod, the specific node, the specific Kubernetes namespace — rather than a secret baked into a config file. A workload requesting an SVID doesn't present a password; it presents proof of what it is, and SPIRE's server verifies that against a registration policy before issuing anything. In a service mesh like Istio, this identity layer can be paired with mutual TLS enforced at the sidecar proxy, so every service-to-service call is authenticated and encrypted without the application code needing to know anything about certificates. Authorization decisions can be externalized to Open Policy Agent, letting teams write access policy as code — reviewable, versioned, testable — instead of scattering if user.role == 'admin' checks through a codebase. For software supply chain integrity — relevant given that the tj-actions/changed-files compromise spread through a CI/CD pipeline — Sigstore's Cosign and Fulcio provide a way to sign build artifacts and verify their provenance using short-lived certificates tied to an OIDC identity, rather than a long-lived signing key that itself becomes another secret to protect. None of this is a rip-and-replace project. Teams typically start by identifying their highest-value machine credentials — the ones with production database access, the ones with broad cloud IAM permissions — and migrating those first to short-lived, attested identities, while instrumenting logging so that every machine identity's access can actually be reviewed rather than assumed. What This Looks Like When Someone Actually Ships It The architecture described above isn't hypothetical. Pinterest has publicly documented using SPIFFE alongside its internal secrets-management system, Knox, specifically to solve identity in a multi-tenant environment where workloads from different teams share infrastructure and can't be trusted by network location alone. Square presented its adoption of SPIFFE and SPIRE at a SPIFFE Community Day, describing how it used the framework to secure communication across a hybrid infrastructure — cloud and on-premises systems that previously had no consistent way to authenticate to each other. Uber's security team gave a KubeCon talk walking through why it built an internal workload identity platform on these same principles, and ByteDance has separately documented replacing a homegrown certificate system with SPIRE to get PKI-based authentication working at the scale TikTok's infrastructure requires. The common thread across all four is the same one this piece has argued from the incident side: none of them adopted workload identity because a compliance checkbox required it. They adopted it because operating at their scale made network-location-based trust and long-lived shared secrets genuinely unworkable — the same pressure that's now reaching far smaller organizations as their own machine identity counts climb. The trade-off they all had to work through in public is worth naming honestly: SPIRE introduces real operational overhead — a server and agent fleet to run, node and workload attestation to configure correctly for each hosting environment, and a learning curve for teams used to thinking about secrets rather than attested identity. None of the public talks describe it as a drop-in replacement. They describe it as an infrastructure investment that pays off once the number of services and the rate of change outgrow what static credentials can manage safely. Section 5: The Metrics That Actually Indicate Progress Security leaders asking for budget need numbers, not architecture diagrams. The ones worth tracking: Mean credential lifetime – how long, on average, does a machine credential remain valid before rotation or expiration? GitGuardian's finding that some leaked keys remained live for months is really a mean-lifetime failure.Percentage of workloads using attested workload identity versus static, long-lived secrets – this is the single clearest proxy for how exposed an environment is to the failure pattern behind the xAI and tj-actions incidents.Secret rotation frequency, measured against an actual policy rather than an aspirational one.Unauthorized service-to-service request rate – a signal that requires the behavioral baselining IDMWorks flagged as largely absent today.Credential exposure rate in code, tickets, and chat – Snyk's research found leaks occurring in Slack messages, Jira tickets, and Confluence pages at meaningful rates, not just in source code, so this metric has to look beyond the repository.Policy compliance rate for third-party OAuth integrations – the exact control gap that let the Salesloft Drift tokens retain broad, long-lived access to Salesforce, Google Workspace, and AWS simultaneously. The Reports Keep Saying the Same Thing Independently It's worth pausing on how many separate organizations, using separate datasets, landed on the same conclusion in the same twelve-month window. Verizon's 2025 Data Breach Investigations Report — built from 22,052 incidents across 139 countries, the kind of dataset no single vendor could assemble on its own — found 441,780 exposed secrets sitting in public code repositories, with a median remediation time of 94 days once discovered. Nearly half of those were high-privilege Google Cloud API keys tied to automated infrastructure, not human logins. GitGuardian's own 2026 research, working from a different pipeline entirely, arrived at the same order of magnitude: 28.65 million new hardcoded secrets added to public GitHub in 2025 alone. IDMWorks, analyzing three years of non-human identity incidents rather than scanning code, described the same underlying failure in different language: no ownership model, no rotation cadence, detection tooling built for human login patterns that generates nothing but noise against machine behavior. Snyk's research adds the vector most of these reports don't emphasize enough — over a quarter of credential incidents originate entirely outside source code, in Slack messages, Jira tickets, and Confluence pages. Different data sources, different methodologies, different commercial incentives — and all of them converge on the same sentence: non-human identities are growing faster than the governance built to manage them. That's not a marketing claim from any one vendor. It's what independent datasets keep saying when you line them up next to each other. What the Trajectory Actually Implies I won't pretend to know the machine-to-human identity ratio a cloud-native enterprise will have in 2030 — nobody has the longitudinal data to state that number with confidence, and treating a guess as a fact would undercut everything else in this piece. What can be said with more confidence is the direction and the reason. Every driver behind today's machine identity growth — CI/CD automation, service mesh adoption, multi-cloud workload identity, and now AI agents authenticating to APIs on an organization's behalf — is accelerating, not leveling off. AI agents in particular are a new category of machine identity, not just more volume in an existing one: an agent can be granted a credential, use it in ways its creator never explicitly authorized, and, in the case of the Common Crawl training-data exposure, potentially reproduce a credential it was never supposed to have seen in the first place. If the ratio of machine to human identities is already in the double digits at a typical mid-sized cloud shop today, as the SPIFFE/SPIRE and workload-identity adoption patterns suggest, then adding an autonomous-agent layer on top doesn't nudge that ratio — it compounds it. The honest prediction isn't a specific number for 2030. It's that any organization treating machine identity governance as a 2026 problem to revisit later is already behind a curve that isn't slowing down. A Rough Map of How Organizations Get Here Most organizations don't leap from careful to reckless. They drift through recognizable stages, usually without anyone deciding to: Plain Text Centralized human IAM ↓ Cloud IAM roles multiply per service ↓ Service accounts proliferate, ownership blurs ↓ Workload identity adopted for some, not all, systems ↓ AI agents added as a new identity class ↓ Identity sprawl outpaces any team's ability to inventory it ↓ Machine Identity Debt crosses into Identity Bankruptcy ↓ Incident forces the reconciliation that governance should have Most organizations reading this are somewhere between stage two and stage five. Very few have consciously decided which stage they're in — which is itself the point: nobody plans to reach identity bankruptcy; they just never stop to check how much debt they've taken on since the last audit. Section 6: Where This Goes Next A few developments will make machine trust an even sharper problem over the next few years rather than a solved one. Confidential computing — running workloads inside hardware-enforced trusted execution environments — is moving from research curiosity to something cloud providers offer as a standard instance type, which will let attestation extend down to the hardware layer rather than stopping at the software identity. AI agents that authenticate to APIs and take autonomous action on an organization's behalf are a new and rapidly growing category of machine identity, and the data-poisoning risk is already visible: Truffle Security's scan of Common Crawl's December 2024 archive, covering roughly 400 terabytes of public web data, found close to 12,000 live, working credentials embedded in text that's now part of the training data feeding future models. An AI system trained on that data can, in principle, reproduce or act on a credential it was never supposed to have. Post-quantum cryptography considerations will eventually reach workload identity systems, since the SVIDs and certificates underpinning frameworks like SPIFFE rely on cryptographic assumptions that are being reassessed industry-wide. And identity graphs — mapping which machine identities can reach which resources, and through which chains of trust — are becoming the tool that lets a security team answer the question that mattered most in the Salesloft Drift breach: not "was this OAuth token valid," but "what could this token reach, and did anyone actually decide it should be able to?" The Question Worth Asking The organizations that got hit in 2025 weren't running unpatched software or ignoring known vulnerabilities. Cloudflare, Palo Alto Networks, and Zscaler — security vendors with mature programs — were among the hundreds caught in the Salesloft Drift breach. The tokens that got them were valid. The access was, technically, authorized. That's what makes machine identity the harder problem: it doesn't fail loudly. The practical shift for engineering leadership is to stop asking "who is the user?" as the primary security question and start asking "can this workload be trusted right now, for this specific action?" That means building ownership records for every service account before an incident forces the question, migrating high-value credentials to short-lived attested identities before a leaked key becomes a header on KrebsOnSecurity, and treating third-party OAuth grants with the same scrutiny given to a new employee's laptop. None of this is speculative. Every incident cited here happened in the past eighteen months, to organizations with real security budgets. The architecture to prevent the next one already exists. What's missing, in most companies, is the decision to build it before the postmortem forces the issue. The pattern underneath every breach in this piece is the same one: a credential nobody was actively watching, doing exactly what it was built to do, for whoever happened to be holding it. Passwords get the attention because a stolen password is a story people understand — a human made a mistake, or got tricked. A stolen service-account token is a harder story to tell, because the mistake happened months earlier, in a decision nobody remembers making, and the debt just sat there accruing until someone else cashed it in. Paying that debt down before it's due is a less dramatic project than responding to a breach. It's also the only version of this problem that ends with a postmortem you never have to write. Sources Google Cloud / Google Threat Intelligence Group, "Widespread Data Theft Targets Salesforce Instances via Salesloft Drift," August 26, 2025The Hacker News, "Salesloft OAuth Breach via Drift AI Chat Agent Exposes Salesforce Customer Data," August 28, 2025Anomali, "Reviewing the Salesforce–Salesloft Drift OAuth Supply Chain Breach," December 2025Guardz, "The Salesloft Drift Breach and the Impact on Google Workspace," September 2025Defakto, "xAI API Key Leak by DOGE Staffer Reveals Cracks in API Security," December 2025Snyk, "Why 28 million credentials leaked on GitHub in 2025, and what to do about it," March 2026Aembit, "Real-Life Examples of Non-Human Identity Security Breaches," updated regularlyIDMWorks, "When Service Accounts Attack: How Identities are Weaponized," May 2026PointGuard AI, "AI Training Data Secret Leak 2025 | 12,000 API Keys Exposed," January 2026CybelAngel, "API Threat Report 2025: Key Findings for Security Teams," March 2026United States v. Sullivan, 9th Cir. 2025 (referenced via Snyk's legal-consequences analysis, above)Verizon, "2025 Data Breach Investigations Report" (18th edition; 22,052 incidents, 139 countries)GitGuardian, "The Secrets Sprawl is Worse Than You Think: Key Takeaways from the 2025 Verizon DBIR," April 2025SPIFFE Project, "Case Studies" (Pinterest, Square, Uber, ByteDance talks)

By Igboanugo David Ugochukwu CORE

In November 2025, I published a Bash script that analyzed Kubernetes clusters in about 60 seconds. It generated HTML reports, surfaced crash loops, orphaned resources, and other operational issues that were easy to overlook. The most interesting part wasn't the script — it was what happened after people started running it. Many told me they found problems they hadn't known existed. Looking back, the bash script wasn't really solving debugging. It was solving prioritization. I just didn't have the vocabulary for it yet. That script eventually became four different experiments, then a collection of small scanners, and eventually the dashboard shown in this article. Over the next eight months, that script evolved into OpsCart Watcher — an open-source operational triage dashboard for Kubernetes. This article is about what the journey taught me, and what I think is still missing from most Kubernetes environments. OpsCart Watcher — operational triage for Kubernetes (6 minutes) The Problem the Script Revealed The script did one thing well: it looked at an entire cluster and listed what was broken. Engineers who ran it kept telling me the same thing — "I had no idea this was there." That response was the important signal. These engineers had Grafana, Prometheus, and kubectl. Visibility was not their problem. The problem was that nothing told them to look at this specific namespace, this specific pod, this specific storage volume — before it became an incident. Consider a pod in CrashLoopBackOff for 19 days with 5,000+ restarts. To a metrics dashboard, that deployment looks healthy: replica count satisfied, a pod exists in Running state between crashes, CPU and memory flat because the container barely lives long enough to consume anything. The dashboard is answering the question it was built to answer — is the cluster meeting its SLOs? — and the answer is yes. The question nobody built tooling for: what deserves attention right now? LayerWhat It AnswersToolsMetricsIs the cluster meeting its SLOs?Prometheus, Grafana, DatadogPer-resource stateWhat is this specific pod doing?kubectl, k9s, LensOperational triageWhat deserves attention right now?Prioritizing operational work across cluster state What Triage Looks Like in Practice Overview page — Incident Score 41/100, KPI bar, Top 5, War Room panel The first time I ran the rebuilt dashboard against a cluster with real failures, the top of the screen didn't show me a CrashLoopBackOff pod. It showed me four CrashLoopBackOff pods spread across three namespaces, collapsed into a single operational problem: Plain Text 1. 4 pods crash-looping CRITICAL payments/fraud-detection (1810 restarts) → kubectl logs fraud-detection-... -n payments --previous That collapsing is the entire idea. Instead of inspecting every deployment individually, I was looking at a ranked list of operational problems — each with a severity, a location, and the exact kubectl command to start investigating. The full output for this environment: Plain Text Incident Score: 41/100 (Degraded) Top 5 Things to Fix: 1. 4 pods crash-looping CRITICAL 4 pods 2. 3 image_pull_backoff issues CRITICAL 3 items 3. 1 privileged_container issue CRITICAL 1 item 4. 1 namespace missing NetworkPolicy HIGH 1 ns 5. 3 orphaned PVCs wasting money MEDIUM 80 GB None of these had triggered an alert. All were present and accumulating before the scan. The Incident Score — a composite 0–100 across reliability, security, and waste — exists for one reason. Engineers fix incidents. Managers remember numbers. "We moved the Incident Score from 41 to 67" is a sentence that sticks. The crash loops and NetworkPolicies are the work behind it. The Step After Detection Finding problems was never the hard part. Knowing where to begin was. The most common feedback on the original bash script was some version of: "I found the problem, but I still didn't know what to do next." In March, I wrote about finding a container with 24,069 restarts that had been accumulating undetected. Finding it took sixty seconds. The next hour was the actual work: what do I run first? Is this configuration or code? Is it customer-facing? The investigation page is my answer to that hour. Investigation page — OpsCart Assessment, Evidence, Recommended Investigation One click from any triage finding opens a dedicated investigation view: Plain Text OpsCart Assessment This workload has restarted 1810 times over 6 days. The restart rate appears stable, suggesting a deterministic configuration or application failure rather than an intermittent infrastructure issue. No referenced ConfigMaps or Secrets were detected in the pod spec — missing configuration is unlikely to be the root cause. Investigation should begin with previous container logs. Estimated time: 5–10 minutes. Evidence [1810 Restarts] [CrashLoopBackOff] [6d] [Deployment/fraud-detection] Recommended Investigation HIGH CONFIDENCE Check previous container logs MEDIUM Verify ConfigMaps and Secrets exist LOW Check for OOMKill in events The assessment is rules-based — no AI. It reads restart count, failure pattern (stable vs accelerating), and referenced configuration objects, then produces a deterministic, auditable summary. The confidence levels reflect how a senior engineer actually reasons: previous logs are almost always the right first move for a crash loop; OOMKill is worth checking but less likely. This is the part kubectl doesn't give you. Neither does Lens, k9s, or Headlamp. From "What Is Broken?" to "What Changed?" The biggest architectural change came when the dashboard gained memory. The first version of the tool answered: "what is broken?" The current version — backed by a small embedded database recording every scan — answers "what changed?" That sounds like a minor distinction. Operationally, it changes everything. An incident that has existed for three days deserves different attention than one that appeared five minutes ago. A cluster whose Incident Score dropped eight points overnight is telling you something that no single scan can. War Room — critical issues with visual differentiation per type Every KPI now carries a trend arrow — critical issues up three since the last scan, waste down one — and the Incident Score shows a seven-point sparkline. Each incident is tracked with first-seen and last-seen timestamps and an active/resolved status, so "CrashLoopBackOff — first detected 6 days ago, still active" replaces "CrashLoopBackOff." Operational memory changed the tool from a scanner into something that remembers the history of a cluster. What This Is Not The triage pattern does not answer when an issue started at the metrics level, why an application is slow, or whether last Tuesday's deployment caused a regression. Prometheus, APM tooling, and deployment audit logs remain the right tools for those questions. The triage layer is not a replacement for observability. It is the layer that tells you which questions to ask of your observability stack. The Biggest Lesson When I started, I thought Kubernetes debugging was about collecting more information. It wasn't. Kubernetes already exposes almost everything an operator needs through its API. The difficult part is deciding what deserves attention first. Over eight months, I found myself spending less time searching for failures and more time ranking them. That is ultimately what OpsCart became — not another dashboard, but a prioritization engine for cluster operations. Why Open Source I considered keeping the dashboard private. Instead, I open-sourced it because operational patterns only become useful when they're tested across different clusters. Every environment fails differently, and I wanted the prioritization model to evolve from real-world feedback rather than a single infrastructure. The Remaining Gap The conclusion from my March article is still true: the question worth asking of your environment is not whether these conditions exist — they almost certainly do — but whether your current observability layer would surface them before they become incident preconditions. Eight months of building has only made that conclusion more specific. The gap is not data. The gap is attention: knowing which five things, out of hundreds of resources, deserve a human's time right now. Eight months ago I thought I was building a better debugging script. I wasn't. I was building something that helps operators decide where to spend the next ten minutes. About the environment: The scenarios shown in this article — CrashLoopBackOff pods, orphaned PVCs, missing NetworkPolicies, privileged containers — are representative of what OpsCart finds on real production clusters. The environment shown is a dedicated demonstration cluster configured with realistic failure scenarios. No production data was used. About the tool: OpsCart Watcher is open-source at github.com/opscart/opscart-k8s-watcher. It deploys as a single read-only container: Shell kubectl apply -f https://raw.githubusercontent.com/opscart/opscart-k8s-watcher/main/deploy/dashboard.yaml kubectl port-forward -n opscart-system svc/opscart-watcher 8080:80

By Shamsher Khan CORE

If you're building a data platform on Azure in 2026, you're going to be asked this question: Azure Databricks or Microsoft Fabric? Both run on Delta Lake, both integrate with ADLS Gen2, both have Spark, and both promise to be your unified data platform. The overlap is real, and the marketing doesn't help. This post is an honest breakdown of where each genuinely excels, where they overlap, and how to decide without getting lost in feature comparison tables. Architecture Comparison Decision Flow Detailed Capability Comparison CapabilityAzure DatabricksMicrosoft FabricWinnerSpark engineFull Spark, Photon, tunableSpark via Notebooks, less tunableDatabricksDelta LakeNative, full controlVia OneLake (Delta Parquet)TieMLflow / MLOpsNative, full MLflow stackBasic experiment trackingDatabricksModel servingDatabricks Model ServingAzure ML integrationDatabricksPower BI integrationDirectQuery via SQL WarehouseDirect Lake (zero-copy, faster)FabricSQL analyticsServerless SQL Warehouse + PhotonSQL Analytics EndpointTieData pipelinesDelta Live Tables, WorkflowsData Factory pipelines (mature)TieReal-time intelligenceSpark Streaming + KafkaEventstream + KQL DatabaseFabricSetup complexityMedium-highLow (SaaS)FabricFine-grained governanceUnity Catalog (mature)Purview integration (growing)DatabricksCost modelDBU + VMFabric capacity unitsComparableOpen format portabilityHigh (standard Delta/Parquet)Medium (OneLake but some lock-in)Databricks Step 1 — Reading Data from Fabric OneLake in Azure Databricks The good news: Fabric and Databricks can share data via OneLake, which speaks Delta format. You don't have to pick one and abandon the other. Python # Azure Databricks reading from Microsoft Fabric OneLake # OneLake exposes an ABFS-compatible endpoint # Authenticate using the workspace's Managed Identity or Service Principal tenant_id = dbutils.secrets.get("kv-scope", "sp-tenant-id") client_id = dbutils.secrets.get("kv-scope", "sp-client-id") client_secret = dbutils.secrets.get("kv-scope", "sp-client-secret") # OneLake uses the same ABFS protocol as ADLS Gen2 fabric_workspace_id = "your-fabric-workspace-guid" lakehouse_name = "your-lakehouse-name" onelake_host = "onelake.dfs.fabric.microsoft.com" spark.conf.set(f"fs.azure.account.auth.type.{onelake_host}", "OAuth") spark.conf.set(f"fs.azure.account.oauth.provider.type.{onelake_host}", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider") spark.conf.set(f"fs.azure.account.oauth2.client.id.{onelake_host}", client_id) spark.conf.set(f"fs.azure.account.oauth2.client.secret.{onelake_host}", client_secret) spark.conf.set(f"fs.azure.account.oauth2.client.endpoint.{onelake_host}", f"https://login.microsoftonline.com/{tenant_id}/oauth2/token") # Read a Delta table from Fabric Lakehouse fabric_path = f"abfss://{fabric_workspace_id}@{onelake_host}/{lakehouse_name}.Lakehouse/Tables/sales_gold" fabric_df = spark.read.format("delta").load(fabric_path) print(f"Rows from Fabric Lakehouse: {fabric_df.count()}") fabric_df.show(5) Step 2 — Writing Databricks Results Back to OneLake Run heavy ML feature engineering in Databricks, write results back to OneLake so Fabric Power BI can consume them via Direct Lake — zero-copy, sub-second dashboard refresh. Python from pyspark.sql.functions import current_timestamp, lit # Run your Databricks feature engineering / ML inference here result_df = spark.table("production.gold.churn_predictions") \ .withColumn("_computed_at", current_timestamp()) \ .withColumn("_source", lit("databricks-inference-job")) # Write back to Fabric OneLake as Delta output_path = f"abfss://{fabric_workspace_id}@{onelake_host}/{lakehouse_name}.Lakehouse/Tables/churn_predictions" result_df.write \ .format("delta") \ .mode("overwrite") \ .option("overwriteSchema", "true") \ .save(output_path) print(f"Written {result_df.count()} rows to Fabric OneLake.") print("Power BI Direct Lake will pick up changes automatically.") Step 3 — When to Use Fabric Notebooks vs Databricks Notebooks Not everything needs Databricks. Fabric Notebooks are good enough for lighter data prep that feeds Power BI reports. Python # This kind of transformation is fine in Fabric Notebooks # Use Fabric when: output goes directly to Power BI, team is analytics-focused, # no MLflow tracking needed, data volume < 100GB # Fabric Notebook (PySpark — same syntax as Databricks) from pyspark.sql.functions import col, sum as _sum, date_trunc df = spark.read.format("delta").load("Tables/sales_silver") summary = df \ .withColumn("month", date_trunc("month", col("sale_ts"))) \ .groupBy("month", "region", "product_category") \ .agg(_sum("revenue").alias("monthly_revenue")) \ .orderBy("month", "region") # Write to Lakehouse table — Power BI picks it up via Direct Lake summary.write.format("delta").mode("overwrite").saveAsTable("monthly_revenue_summary") # Use Databricks when: MLflow tracking needed, complex ML pipeline, # Unity Catalog governance required, data volume > 1TB, streaming workloads When to Use Which: Decision Framework Python # Use this as a mental checklist when deciding DATABRICKS_STRENGTHS = [ "Complex ML pipelines with MLflow experiment tracking", "Production model serving with A/B testing", "Fine-grained governance via Unity Catalog (row/column security)", "Spark Structured Streaming with Kafka / Event Hub", "Very large scale ETL (multi-TB, complex joins)", "Open-source tool integrations (dbt, Great Expectations, etc.)", "Multi-cloud or portability requirements", ] FABRIC_STRENGTHS = [ "Power BI as the primary consumption layer (Direct Lake = fastest)", "Analytics-focused teams without deep Spark expertise", "Microsoft 365 integration (Teams, SharePoint data sources)", "Real-time dashboards via Eventstream + KQL", "Fabric Data Factory for straightforward ELT pipelines", "Lower operational overhead — fully SaaS managed", "Already licensed via Microsoft 365 E5 / Fabric capacity", ] BOTH_TOGETHER = [ "Heavy ML/MLOps in Databricks, results published to OneLake for Power BI", "Fabric Data Factory for ingestion, Databricks for complex transformation", "Unity Catalog governing Databricks tables, Fabric consuming via shortcuts", ] Things to Watch in Production OneLake shortcuts are the integration bridge. Fabric Lakehouses support shortcuts that point to external Delta tables in ADLS Gen2 — the same storage Databricks writes to. This means Databricks writes once and Fabric reads without data movement. Set up shortcuts rather than copying data between platforms. Unity Catalog doesn't govern Fabric. Your row-level security and column masks in Unity Catalog do not apply when Fabric reads the same underlying Delta files directly. If governance is critical, either run everything through Databricks or replicate governance rules in Fabric's permission model. Fabric capacity units and Databricks DBUs are both usage-based but measure differently. Don't try to compare them directly. Run the same workload in both and compare wall-clock time and cost on your actual data sizes. Fabric ML is improving fast but isn't MLflow. As of early 2026, Fabric ML experiment tracking is functional but doesn't have the depth of MLflow's model registry, artifact storage, or model serving. If MLOps maturity matters, stay on Databricks for ML. Wrapping Up The honest answer is: most mature Azure data platforms in 2026 use both. Azure Databricks for ML, complex transformations, governance, and streaming. Microsoft Fabric for Power BI-first analytics, simpler pipelines, and teams that don't need the full Databricks stack. OneLake shortcuts and the shared Delta format make them composable rather than competitive. Pick based on your primary consumer: if it's Power BI dashboards, start with Fabric. If it's ML models and data products, start with Databricks. When you need both, they integrate cleanly. References Microsoft Fabric DocumentationOneLake — The OneDrive for DataFabric Lakehouse vs Azure DatabricksDirect Lake in Power BIOneLake ShortcutsAzure Databricks and Microsoft Fabric IntegrationUnity Catalog vs Fabric Data GovernanceFabric Eventstream — Real-Time Intelligence

By Jubin Abhishek Soni CORE

Production incidents are inevitable. No matter how much testing, automation, observability, or resilience engineering an organization invests in, complex distributed systems will eventually fail in unexpected ways. The real differentiator between high-performing engineering organizations and everyone else is not whether incidents occur — it is how effectively organizations learn from them. Unfortunately, many root cause analysis (RCA) processes fail to achieve this objective. Instead of uncovering systemic weaknesses, they often focus on identifying a single mistake, a specific engineer, or a single technical failure. The resulting report may satisfy a compliance requirement, but it rarely produces meaningful improvements in reliability. As cloud-native architectures become increasingly distributed and interconnected, organizations must evolve beyond traditional RCA practices and adopt blameless postmortems that focus on organizational learning and continuous improvement. The Traditional RCA Trap Most incident investigations begin with a simple question: "What caused the outage?" At first glance, this seems reasonable. However, the question itself often leads teams toward finding a single root cause. Common conclusions include: An engineer deployed an incorrect configuration.A database migration introduced an error.An operator executed the wrong command.A monitoring alert was ignored.A service exceeded capacity limits. While these statements may be factually correct, they often represent only the final event in a much larger chain of failures. Consider a scenario where a configuration change causes a critical service outage. A traditional RCA might conclude: The outage occurred because an engineer deployed an invalid configuration file. While technically true, this explanation leaves many important questions unanswered: Why was the invalid configuration allowed into production?Why did automated validation fail to detect the issue?Why did monitoring not identify the problem immediately?Why was the blast radius so large?Why was rollback difficult?Why did recovery take longer than expected? These questions often reveal the real opportunities for improvement. Modern Incidents Rarely Have a Single Root Cause One of the most important lessons from operating distributed systems is that incidents are almost never caused by a single failure. Modern cloud environments contain thousands of interacting components: Microservices, APIs, Databases, Service meshes, Kubernetes clusters, CI/CD pipelines, Infrastructure automation, Third-party dependencies A seemingly simple outage often emerges from a combination of factors. For example: Contributing FactorImpactIncomplete testingAllowed faulty configurationMissing safeguardsFailed to block deploymentWeak observabilityDelayed detectionDocumentation gapsSlowed troubleshootingComplex architectureIncreased blast radiusManual recovery processExtended outage duration No single factor caused the outage. Rather, the outage occurred because multiple layers of defense failed simultaneously. This is why mature organizations increasingly focus on contributing causes rather than searching for a single root cause. What Does "Blameless" Actually Mean? One of the most misunderstood concepts in incident management is the idea of a blameless postmortem. Some teams incorrectly assume that blameless means avoiding accountability. It does not. Blameless means recognizing that engineers make decisions based on the information available to them at a given moment. During an active incident: Information is incomplete.Time pressure is high.Monitoring signals may be conflicting.Customer impact is increasing.Stress levels are elevated. The objective of a postmortem is therefore not to judge whether an individual made a perfect decision. The objective is to understand: Why the decision seemed reasonable at the time.What information was available.What information was missing.What systemic conditions contributed to the outcome. When teams focus on learning instead of blame, they become far more willing to share details openly and honestly. Anatomy of an Effective Postmortem High-quality postmortems typically follow a structured approach. 1. Incident Summary Begin with a concise overview: What happened?When did it occur?How long did it last?Who was affected?What was the business impact? Example: "On March 12, Service X experienced elevated latency following a configuration deployment. Approximately 15% of customer requests failed for 42 minutes before service was fully restored." 2. Timeline Reconstruction The timeline is often the most valuable section of a postmortem. Document key events chronologically: TimeEvent09:00Deployment initiated09:05Error rate increased09:08Customer complaints received09:12Incident declared09:18Rollback initiated09:25Error rate returned to normal09:42Incident resolved A detailed timeline helps teams understand exactly how events unfolded. 3. Contributing Factors Analysis Rather than searching for a single root cause, identify all meaningful contributors. Examples include: Technical Contributors Configuration validation gapsCapacity limitationsMonitoring deficienciesDependency failuresArchitectural constraints Process Contributors Incomplete deployment reviewsMissing runbooksEscalation delaysLack of disaster recovery testing Organizational Contributors Knowledge silosStaffing limitationsUnclear ownership boundariesTraining gaps The goal is to build a complete picture of the incident. 4. Recovery Assessment Analyze the effectiveness of the response. Questions worth asking: Was detection timely?Were alerts actionable?Was ownership clear?Did responders have the necessary tools?Were runbooks useful?Could recovery have been automated? Many organizations discover that recovery challenges contribute more customer impact than the original failure itself. The Five Whys: Useful But Limited Many organizations use the "Five Whys" technique. Example: 1. Why did the outage occur? Because a configuration was invalid. 2. Why was it invalid? Because validation checks were incomplete. 3. Why were validation checks incomplete? Because a new deployment framework was introduced. 4. Why was the framework deployed without complete validation? Because release deadlines prioritized delivery. 5. Why were deadlines prioritized? Because organizational risk was underestimated. The Five Whys can uncover valuable insights. However, distributed systems are rarely linear. Multiple parallel factors often contribute simultaneously. Treat them as one investigative tool, not the entire analysis framework. Turning Findings Into Action A postmortem without action items is merely documentation. Every significant finding should produce a measurable improvement initiative. Examples include: FindingActionConfiguration errors reach productionAdd automated validationDetection delayed by 10 minutesImprove alert coverageRollback requires manual interventionImplement automated rollbackTroubleshooting knowledge unavailableCreate operational runbooksRecovery depends on expertsExpand team training Action items should be: specific, assigned, prioritized, and trackable. Without ownership, lessons learned quickly become lessons forgotten. Measuring Postmortem Effectiveness Many organizations measure success by counting completed postmortems. A more meaningful approach is measuring operational improvement. Consider tracking: Mean time to detect (MTTD)Mean time to recover (MTTR)Repeat incident frequencyAutomated recovery rateManual intervention reductionCustomer impact reduction The ultimate goal is not producing better reports. The goal is producing more resilient systems. The Future: AI-Assisted Incident Learning As incident management platforms evolve, AI is beginning to transform postmortem creation. Modern systems can automatically: Build incident timelinesCorrelate alertsSummarize communication channelsExtract remediation actionsIdentify recurring failure patternsGenerate draft postmortems This allows responders to spend less time gathering information and more time analyzing systemic weaknesses. However, AI should augment human investigation — not replace it. Understanding organizational context, operational tradeoffs, and architectural decisions still requires human expertise. Final Thoughts The most valuable outcome of an incident is not service restoration. It is learning. Organizations that focus solely on identifying who made a mistake often repeat the same failures. Organizations that focus on understanding how their systems allowed failures to occur continuously improve their resilience. Blameless postmortems shift the conversation from: "Who caused this incident?" to "What can we learn from this incident, and how can we make the system stronger?" That mindset is ultimately what transforms incident management from a reactive operational function into a strategic capability that improves reliability, resilience, and engineering excellence over time.

On December 2, 2024, a security vendor called BeyondTrust noticed something wrong inside its own AWS account. By the time the investigation closed, the story that emerged was almost absurdly simple for something with this much fallout: an attacker — later attributed to the Chinese state-sponsored group Silk Typhoon — had used a software flaw to reach into a BeyondTrust cloud account and pull out an API key. Not a password. Not a phishing victim's login. A string of characters that a piece of software used to talk to another piece of software. With that one key, the attacker walked straight into the U.S. Department of the Treasury, reset internal passwords, accessed workstations inside the Office of Foreign Assets Control, and read unclassified documents before anyone noticed. The Treasury disclosed it to Congress on December 30. The Department of Justice indicted the alleged operators in March 2025. If you've never worked in security, here's the plain-English version of what happened: somewhere inside the machinery that runs modern software, there's almost always a "key" — a credential one computer program shows another to prove it's allowed to be there. Humans log in with passwords and, increasingly, a second factor on their phone. Software mostly doesn't. It just holds a key, often for months or years at a time, and whoever holds that key gets treated as trustworthy, no questions asked. The Treasury breach happened because one of those keys ended up in the wrong hands and nothing else stood between that key and a federal agency's internal documents. Two months later, a different flavor of the same problem produced the largest theft of digital assets in history. $1.5 Billion, One Developer's Laptop In February 2025, the cryptocurrency exchange Bybit lost approximately $1.5 billion in Ethereum in a single operation. Palo Alto Networks' Unit 42 threat research team later tied the attack to Slow Pisces, a North Korean state-linked group also known as Lazarus or TraderTraitor, and traced the entry point back to a developer at a third-party vendor that managed Bybit's multi-signature wallet infrastructure. The attackers didn't break Ethereum's cryptography. They stole that developer's AWS session tokens — another form of machine credential — and used them to gain administrative access to cloud infrastructure that could authorize transactions, then quietly altered what a routine-looking transaction actually did before it executed. Unit 42 then found the same pattern at a second cryptocurrency exchange later in 2025, this time running through Kubernetes, the orchestration system that now runs much of the cloud-native world. The attackers phished a developer, used the access on the developer's machine to drop a malicious workload directly into the exchange's production Kubernetes cluster, and had that workload expose its own service account token — a credential Kubernetes automatically hands to every running pod so it can talk to the cluster's control plane. The stolen token happened to belong to a CI/CD management identity with sweeping permissions. From there, the intruders queried secrets across namespaces, planted a backdoor, and pivoted into the exchange's cloud-hosted backend, reaching the financial systems behind it. Unit 42's broader research found suspicious activity consistent with service-account-token theft in 22 percent of cloud environments analyzed in 2025, and recorded a 282 percent year-over-year jump in Kubernetes-directed attacks overall. Different industries, different attackers, same root cause: a non-human credential that was both long-lived and broader in scope than the task in front of it ever needed. Why This Keeps Happening Identity and access management, as a discipline, was built for people. People have managers, onboarding dates, performance reviews, and an HR system that flags them the day they leave. A workload has none of that. A microservice can spin up, do its job, and disappear thousands of times a day; a service account, by contrast, often gets created once and never revisited again. CyberArk's research has been blunt about the resulting imbalance: machine identities now outnumber human ones by more than 80 to 1 in the average enterprise, and the security architecture protecting most of them still assumes the old, human-shaped world — an org chart, not a fleet of ephemeral containers. That mismatch is exactly why static secrets sprawl the way they do. A developer hardcodes a key during a deadline crunch, intending to externalize it "later." A Terraform state file ends up holding plaintext cloud credentials because nobody flagged it in review. A default Kubernetes service account token, more permissive than anyone realized, gets mounted into a pod by default because turning that off requires deliberate configuration most teams never get around to. None of these are exotic mistakes. They're the ordinary residue of moving fast, and they accumulate the way unpaid debt does — quietly, until the day someone calls it in. The structural fix has a name by now, even if adoption is uneven: frameworks like SPIFFE and its production runtime SPIRE replace the static key with a short-lived, cryptographically attested identity — something closer to a backstage pass that's reissued before every single show rather than a master key cut once and handed out forever. A workload proves what it actually is — which Kubernetes service account launched it, which container image it's running — and receives an identity document valid for minutes, not months. Steal that, and an attacker is racing a clock that resets automatically rather than one that only resets when a human notices something is wrong. Cloud providers offer narrower versions of the same idea for their own platforms — AWS's IAM Roles for Service Accounts, Google's Workload Identity Federation — letting a workload trade a short-lived token for cloud access instead of carrying a standing key in the first place. But identity alone doesn't close the loop, and this is the part most "zero trust" conversations skip past. None of it matters if nothing in your pipeline actually enforces it. Security By Design Is a Promise. CI/CD Is Where You Find Out If It's Kept. Plenty of organizations will tell you, with complete sincerity, that they practice "security by design." Most of them mean it stopped at an architecture review months before the first line of code shipped. That's not a fix, it's a memory of one. Code that deploys daily — sometimes hourly — doesn't wait for an annual audit to catch a misconfigured token or an over-privileged service account, and by the time a quarterly review would have caught the BeyondTrust-style key or the Bybit-style session token, the damage in both real cases was already done. The only version of "security by design" that survives contact with a real production pipeline is the one written as code and enforced automatically, at every stage, by something that can actually say no. Picture the pipeline this way: Plain Text Developer commits code | v CI build triggers | +--> SAST (code flaws) + SCA (dependency CVEs) + secrets scan | | | fail? -----> build blocked, developer notified | | | pass v Generate SBOM + sign artifact (Cosign) + build provenance (SLSA) | v Policy-as-code gate (OPA / Kyverno) | +--> checks: image from approved registry? running as non-root? | signature valid? provenance matches expected builder? | service account scoped to least privilege? | | fail? -----> deployment rejected, logged, alert raised | pass v Deploy to production | v Runtime monitoring + short-lived workload identity (SPIFFE/SPIRE, IRSA) | v Continuous re-verification — nothing trusted indefinitely Every box in that chain is a place where the Treasury breach or the Bybit breach could have stopped instead of escalating. A policy-as-code rule using Open Policy Agent's Rego language, or Kyverno's Kubernetes-native YAML equivalent, can flatly refuse to schedule a pod requesting broader RBAC permissions than its declared task needs — which would have directly undercut the over-privileged CI/CD identity that the crypto-exchange attackers rode into the cluster. A signing and attestation step using Cosign, tied to SLSA provenance, means a deployed artifact has to prove which build system actually produced it before it runs at all — closing exactly the kind of trust gap that let a single compromised AWS asset cascade into a stolen infrastructure API key at BeyondTrust. None of this is theoretical tooling. Red Hat's own Enterprise Contract documentation describes signing as tying an image to a specific builder identity precisely so an attacker can't substitute a malicious binary without the signature itself breaking and announcing the tampering. The Uncomfortable Bottom Line I don't think either of this year's headline breaches happened because anyone involved was careless in some obvious, fireable way. They happened because the credential — not the firewall, not the encryption, not the cleverness of the malware — was the actual asset under attack the entire time, and almost nothing downstream of "the key worked" was built to ask a second question. Gartner named non-human identity management a top strategic security trend for exactly this reason in 2025, and OWASP followed with a dedicated Non-Human Identity Top 10 the same year, an overdue acknowledgment that the tooling built for human logins was never going to be enough. My honest prediction, watching this pattern repeat across a federal agency and two of the largest crypto exchanges on earth within twelve months of each other: the organizations that treat policy-as-code enforcement and short-lived machine identity as default infrastructure — not optional hardening bolted on after an incident — are the ones that won't end up writing the next version of this story. Everyone else is currently running on borrowed time, secured by a key that, statistically, is already older than it should be.

By Igboanugo David Ugochukwu CORE

The Fundamental Problem: Signal Is Infinitesimal Compared to Noise Modern cloud systems operate at a scale where traditional data processing assumptions begin to break down. In large distributed environments, telemetry pipelines routinely process tens of billions of events per minute across millions of users, accounts, and resources. At this scale, the objective of threat detection is often misunderstood. The goal is not to process data — it is to extract actionable signals for incident response. The critical observation is simple, but non-intuitive: By design, almost all data is benign. For any given key — such as a combination of user, account, action, and source — the overwhelming majority of activity falls within expected behavior. Only a tiny fraction represents anomalies such as abuse, credential compromise, or unintended automation. This imbalance introduces two fundamental challenges. First, key cardinality becomes extremely high. Most keys appear once, or only a handful of times, within any given time window. The system is forced to track a vast number of unique identifiers, the majority of which will never become relevant. Second, signals are sparsely distributed. The events that matter — the ones that indicate potential threats — are rare, but carry disproportionate importance. Missing them is not acceptable, but exhaustively processing everything to find them is not scalable. Under these conditions, traditional aggregation approaches begin to lose their advantage. Systems built on MapReduce-style paradigms rely on the assumption that data can be meaningfully reduced through grouping and aggregation. That assumption does not hold when most keys are unique, and most activity is noise. When the input size approaches the output size, there is no real reduction taking place. The system spends the majority of its resources maintaining state and processing events that will never contribute to a detection. At that point, the bottleneck is no longer compute capacity — it is the system’s inability to ignore what does not matter. Where MapReduce Works — and Where It Breaks Before discussing alternatives, it is important to acknowledge that MapReduce-style systems are highly effective — when applied to the right class of problems. At a high level, the model is simple. The map phase transforms incoming data into key-value pairs. The reduce phase aggregates values for each key, producing a condensed representation of the dataset. This approach works well when aggregation leads to a meaningful reduction. Consider counting page views per URL across billions of requests. While the input volume is massive, the number of unique URLs is relatively bounded. Millions or billions of events collapse into a much smaller set of aggregated counts. The system compresses the data as it processes it. In this setting, the overhead of grouping and shuffling data is justified because the output is significantly smaller than the input. Threat detection, however, does not behave this way. Security telemetry is typically keyed on combinations such as: Python (user_id, account_id, action, resource, source_ip) This key space is effectively unbounded. It evolves continuously with user behavior, infrastructure changes, and external interactions. Most keys appear once. Some appear a few times. Very few appear frequently enough to matter. When MapReduce is applied here, aggregation does not reduce the dataset — it merely reorganizes it. The system pays the full cost of aggregation without realizing its benefits. State must be maintained for an enormous number of keys. Data must be shuffled across nodes to ensure correct grouping. The output remains nearly as large as the input because almost every key contributes a single record. This is the fundamental mismatch. MapReduce assumes that aggregation compresses data. High-cardinality threat detection violates that assumption. A Concrete Example: When the System Starts to Hurt Consider a scenario where an IAM principal begins issuing a burst of mutating API calls—creating roles, attaching policies, and modifying permissions across multiple accounts. This pattern is often associated with credential compromise or automated abuse. A detection might look like: Alert if a single (principal, source_ip) pair performs more than 100 mutating IAM actions within 5 minutes. In a real environment, millions of principals are active at any given time. Most perform one or two operations and stop. Some perform legitimate bursts due to deployments or automation workflows. A very small fraction represents potential abuse. A naive system will track every (principal, source_ip) pair. It will maintain state, update counts, and evaluate thresholds for all of them. Meanwhile, the one key that actually matters — the compromised principal — moves through the same pipeline as everything else. The system is functioning correctly, but inefficiently. It spends most of its effort processing keys that will never cross the detection threshold. From Exactness to Selectivity The turning point comes when the problem is reframed. Instead of attempting to compute exact counts for every key, the system first determines which keys are worth tracking at all. Reduction must happen before aggregation. This introduces a form of selective attention into the pipeline. Early stages apply lightweight, probabilistic techniques to eliminate obvious noise. Keys that appear only once — or exhibit behavior unlikely to lead to a signal — are deprioritized immediately. Keys that show repeated activity are then tracked using approximate counting methods. These methods do not aim for exactness. They aim to answer a simpler question: Is this key becoming significant? Only a small subset of keys — those that appear to be trending toward anomalous behavior — are promoted to later stages for precise evaluation. This shifts the system from exhaustive computation to targeted analysis. Heuristics That Make This Possible Two classes of techniques are particularly effective in enabling this shift. The first is probabilistic membership filtering. Structures such as Bloom filters allow the system to quickly determine whether a key has been seen before. Single-occurrence keys, which dominate the dataset, can be filtered out early without maintaining explicit state for each one. The second is approximate frequency tracking. Structures such as Count-Min Sketch provide a compact way to estimate how often a key appears. While the counts are not exact, they are sufficient to identify keys whose activity is increasing and may cross a detection threshold. These techniques share an important property: their cost is independent of the number of unique keys. Memory usage and computation are bounded, even as cardinality grows. In practice, they allow the system to discard the majority of data before it reaches expensive stages of processing. Why This Model Scales This approach fundamentally changes how the system scales. Instead of maintaining state proportional to the number of unique keys, the system maintains state proportional to the size of its probabilistic structures. Communication between nodes is similarly bounded, as these structures can be merged without requiring per-key coordination. More importantly, expensive operations are applied selectively. Exact aggregation is no longer the default path for all data. It is reserved for a small subset of keys that have already demonstrated potentially anomalous behavior. This leads to a significant reduction in resource usage across the pipeline — compute, memory, and network. In practice, the majority of events are filtered or summarized early, allowing the system to focus its resources on the small fraction of activity that may represent a real threat. How This Looks in a Real Pipeline A practical system can be thought of as a staged funnel. Incoming events first pass through a lightweight filtering layer that removes one-off or low-signal keys. The remaining events are then processed by an approximate counting layer that identifies keys with increasing activity. Only those keys are promoted to an exact aggregation layer, where precise counts are computed, and detection logic is applied. Each stage reduces the volume of data that flows into the next. By the time the system reaches exact aggregation, it is operating on a very small subset of the original stream. Closing Observation The instinct in distributed systems is often to scale computation to match data volume. In threat detection, that instinct leads to diminishing returns. The more effective approach is to reduce the amount of data that requires precise computation. At a large scale, the challenge is not computing aggregates efficiently. It is deciding which aggregates are worth computing at all. Once that distinction is made, the system becomes both simpler and scalable.

Our first version was wrong 57% of the time. Not because the AI model couldn't identify Docker container failure scenarios—it usually could. The failures occurred at the decision boundary: determining when an automated action was appropriate, when escalation was required, and when no action should be taken. Over several weeks, we built and evaluated an AI-assisted remediation system on Docker MCP Gateway across four container failure scenarios, improving decision correctness from 43% to 100%. What we learned surprised us: the hard problem is not teaching the agent to act. The hard problem is defining and enforcing the boundary where the agent must stop acting. The project reinforced a broader lesson: production-safe AI is less about model intelligence and more about engineering explicit policies, validation mechanisms, and execution controls. This article covers what we built, what failed, and the engineering changes that improved correctness. The full code, audit logs, validation datasets, and analyzer scripts are all in the companion repository. Why Naive Auto-Remediation Is Dangerous The most common mistake in AI-driven operations is treating "AI can fix things" as the goal. It isn't. A remediation system that attempts to fix every incident automatically is often worse than having no automation at all. Consider the failure modes: An automatic restart of a CrashLoopBackOff container does not fix the underlying problem—it simply generates more alerts. The container will fail again because the code or configuration issue remains unchanged. The result is additional operational noise without any meaningful remediation. Automatically increasing memory limits for every OOM event can be equally problematic. The workload continues running, but the underlying memory leak remains hidden. Months later, teams may find themselves running multi-gigabyte containers that should have been consuming a fraction of those resources. Automated remediation without an audit trail creates a different problem: a lack of accountability. Without structured records, it becomes impossible to determine what actions were taken, what actions were considered, and why a particular remediation path was selected. "The AI fixed it" is not a useful postmortem entry. The safest remediation systems are not the ones that automate the most actions. They are the ones with clearly defined operational boundaries, explicit escalation rules, and auditable decision paths. The engineering challenge is not maximizing automation — it is determining where automation should stop. According to Mohammad-Ali A'râbi, Docker Captain: One of the most dangerous assumptions teams can make is treating a language model as if it were an experienced senior site reliability engineer. It is not. A language model may generate useful recommendations, but it has no operational accountability. It does not understand business context, service ownership, deployment history, or the downstream consequences of an action. Any system granted the ability to modify production infrastructure must therefore be treated as an untrusted component operating behind strict controls. The container ecosystem learned this lesson years ago through the principle of least privilege. We stopped running containers as root whenever possible. We reduced Linux capabilities to the minimum required set. We learned that mounting Docker sockets into containers for convenience often created unacceptable security risks. The common theme was simple: convenience should not bypass security boundaries. The same principle applies to operational automation. Granting unrestricted access to restart workloads, modify resource limits, or execute privileged actions without meaningful controls introduces unnecessary risk. The challenge is not improving the quality of recommendations. The challenge is ensuring that every action is constrained, observable, and reversible. This is where Docker MCP Gateway becomes valuable. Rather than allowing direct access to infrastructure operations, the Gateway places a controlled execution layer between the decision-making component and the underlying tools. Authentication, rate limiting, audit logging, input validation, and execution isolation are applied consistently before any action is performed. In our implementation, every tool invocation passed through HMAC authentication, Redis-backed rate limiting, structured audit logging, and containerized execution. These controls were not added as enhancements; they were treated as core design requirements. Production systems already rely on admission controllers, access controls, audit trails, and policy enforcement. Operational automation should be held to the same standard. Access to credentials should remain isolated from the decision-making layer. Direct access to host resources should be minimized. Every action should be traceable and reviewable. The more authority a system is given, the more important it becomes to enforce clear operational boundaries. Reliable automation depends less on unrestricted capability and more on well-defined constraints. What Docker MCP Gateway Gives You At a high level, Docker MCP Gateway acts as a secure control plane between AI agents and MCP tools, enforcing authentication, rate limits, audit logging, and execution isolation for every tool call. The Model Context Protocol (MCP) is an open standard introduced by Anthropic in late 2024 that gives AI applications a uniform interface for invoking external tools and services. It has since gained support across multiple vendors, including Anthropic, OpenAI, Google DeepMind, and AWS. MCP solves the protocol problem. It doesn't solve the production problem. Production systems require controls around tool execution, not just a standardized way to invoke tools Authenticated tool calls (not just "the agent has the API key in plaintext somewhere")Rate limiting (agents can spiral fast)Audit logging of every decisionContainerized tool isolation (so a misbehaving tool can't take down its host)Centralized policy enforcement (so adding a new server doesn't require reconfiguring every client) Docker MCP Gateway provides these operational controls. It sits between AI clients and MCP servers, routing every tool invocation through a centralized enforcement layer that handles authentication, policy enforcement, rate limiting, and execution isolation. For our work, we built a custom MCP server inside Docker that exposes three remediation tools: check_container_logs, restart_container, and update_container_resources. Every request passes through HMAC authentication, is rate-limited using Redis, and is recorded in a structured JSON audit log before execution.mc From Mohammad-Ali A'râbi, Docker Captain: Docker's AI tooling strategy is fundamentally about building a verifiable supply chain for reasoning engines. You cannot build secure AI on top of bloated, vulnerable foundations. The strategy begins with Docker Hardened Images (DHI), providing agents and MCP servers with minimal attack-surface base images backed by cryptographically signed SLSA Level 3 provenance. The Docker Hub MCP then acts as a discovery layer, allowing agents to find and navigate trusted container artifacts through natural-language interactions. From there, these components converge into Docker AI Governance, where MicroVM-based sandboxes apply strict, deny-by-default controls over filesystem access, network connectivity, and tool execution. Together, these capabilities represent a broader architectural shift from securing application code to securing an agent's entire operational blast radius. Recent supply-chain attacks such as Shai-Hulud 2.0 have shown that modern attackers increasingly target the automation layers that underpin software delivery. AI agents now operate inside those same environments, making blast-radius reduction a first-class architectural concern. A Decision Framework: When to Auto-Fix vs. Escalate Before implementing any automation, we documented the expected behavior for each failure mode. This was not a planning exercise—it became the specification the system had to satisfy and later served as the foundation for our validation framework. Failure Type Likely Cause Safe Action OOMKilled Resource exhaustion (often legitimate) Auto-fix: increase memory CrashLoopBackOff Code or configuration bug Escalate — never auto-restart Single Exit (code 1) Could be transient (network, DB) or persistent Try restart once, escalate if it persists HealthCheckFailure App stuck or deadlocked Auto-fix: restart The guiding principle was simple: transient and resource-related failures could be remediated automatically, while persistent application and configuration failures required escalation. Transient and resource-driven failures auto-fix. Persistent and code-driven failures escalate. Every decision is logged. This framing matters more than the implementation. It's the part you should keep even if you replace every other piece of the system. The agent's job isn't to be smart — it's to apply this rule consistently and visibly. We chose to encode this in the agent's system prompt rather than in code branching, which turned out to be one of our most important design decisions. More on that below. The Architecture in Practice The system has five logical layers running across three Docker Compose containers: Five-layer architecture: container failure triggers the AI agent, which routes every tool call through the Docker MCP Gateway security pipeline before reaching MCP Tools and the Docker API. The architecture separates concerns into five layers. The AutoGen agent (GPT-3.5-turbo, cost-optimized for this decision space) handles reasoning and decision-making. The Docker MCP Gateway sits in front of the tools as a security enforcement point — every tool call passes through HMAC authentication, Redis-backed rate limiting (100 requests/hour), input validation, and structured audit logging. The MCP Tools layer exposes three remediation actions: check_container_logs, restart_container, and update_container_resources. Below that, the Docker API performs the actual container operations. In our current implementation, the Gateway and Tools layers are colocated in a single Python service for simplicity — in a multi-tenant production setup you'd separate them into distinct services that scale independently. Every tool call generates an audit log entry like this: JSON { "timestamp": "2026-05-07T02:08:15.456Z", "incident_id": "inc-20260507-020815", "agent_id": "docker-ops-agent-001", "alert": { "description": "Docker container crashed with OOMKilled", "container_id": "nginx-oom-test", "status": "OOMKilled" }, "decision_chain": [ {"tool": "check_container_logs", "result": "..."}, {"tool": "update_container_resources", "result": "Memory limit updated to 200MB"} ], "resolved": true } That structured output is what makes the system auditable. It's also what makes our validation work possible. The Engineering Reality: 43% to 100% Across 7 development-phase incidents, our agent made the correct decision 43% of the time. Across 6 validation-phase incidents after applying our fixes, it was correct 100% of the time. Both datasets are committed in the repo's monitoring/analysis directory. Phase Runs Correct Avg Turns/Incident Before fixes 7 3/7 (43%) 22.7 After fixes 6 6/6 (100%) 11.7 A note on sample size: this is a small dataset. It's enough to show the expected behavior is reproducible across the four scenarios, but not enough to make claims about reliability under load or at scale. What changed between the two phases is documented as nine challenges in the lab README. Three of them drove most of the improvement. Here they are. Challenge A: The OOM That Couldn't Be Fixed In the early runs, the agent correctly diagnosed an OOMKilled container, called the memory-update tool, and got back this Docker error: Plain Text Memory limit should be smaller than already set memoryswap limit, update the memoryswap at the same time Then it correctly escalated, because it had no tool for updating memoryswap. Our analyzer marked this as wrong because the OOMKilled scenario expected AutoResolved, not Escalated. But the agent's logic was right. The bug wasn't in the agent — it was in our test container's --memory-swap configuration. Once we fixed that (set --memory-swap=-1 for unlimited swap), the agent's behavior didn't change at all. The same logic that escalated correctly before now succeeded correctly. The agent went from 0/2 to 2/2 correct. Lesson: When the agent makes the right decision but your tests say it's wrong, check the test setup before blaming the agent. We spent a few hours debugging the agent before realizing our own container configuration was the problem. Challenge B: The Over-Eager Restart In the first three CrashLoopBackOff runs, the agent restarted the container 2 out of 3 times. CrashLoopBackOff is exactly the failure mode where you should never restart — the container is crashing because of a code or config bug, not a transient state. Restarting just generates more crashes. We almost wrote a code branch for it: add a check, route CrashLoopBackOff to a different path. Before doing that, we tried tightening the system prompt instead: Plain Text For CrashLoopBackOff failures: ALWAYS escalate to a human operator. NEVER attempt to restart the container. Restarting will only cause the container to crash again. Your role is to diagnose and report, not to fix. That single change — no code, just words in the prompt — made the agent consistently escalate on every subsequent run. Lesson: If you want the agent to follow a rule, write the rule down in the system prompt. Don't leave it to the model to figure out. We spent more time arguing about whether to add code branching than the prompt change actually took. Challenge C: The Hallucinated Containers After resolving real incidents, the agent started making up alerts for containers that didn't exist — memory-hungry-app, app-crash-loop, none of which were ever in our system. It was inventing failures and then "responding" to them. Root cause: AutoGen's max_consecutive_auto_reply was set to 10. After the agent finished a real incident, the conversation framework kept giving it turns. Without a real prompt to respond to, it generated plausible-looking next incidents and walked itself through fake remediations. Fix: drop max_consecutive_auto_reply to 3. The agent gets exactly enough turns to diagnose, act, and report — then the conversation ends. Lesson: AutoGen and similar frameworks default to long conversations because they're built for chat use cases. For production, you want them to stop talking once the job is done. From Mohammad-Ali A'râbi, Docker Captain: The progression from 43% to 100% correctness reinforced a key lesson: production AI is often less a machine-learning problem; it is a systems engineering challenge. The initial failures were not the fault of the LLM; they were the result of implicit, undocumented policies and permissive execution environments. Production AI engineering requires moving past the "magic" of conversational models and returning to a rigorous, deterministic engineering discipline. It means treating the system prompt as an immutable policy file, writing explicit, boundary-defining rules that leave zero room for the model to improvise. It means enforcing aggressive Redis-backed rate limits to prevent hallucination loops, isolating execution tools to eliminate docker.sock vulnerabilities, and relying exclusively on structured JSON audit logs rather than plain text for forensic validation. The agent is merely a component. The surrounding infrastructure — the cryptographic constraints, the isolated execution environments, and the hardcoded fallbacks — is what actually makes the system safe. Building trust in AI demands the exact same rigor we apply to cluster security: trust nothing, verify everything, and strictly log the rest. Production Patterns We'd Recommend If you're building something similar with Docker MCP Gateway, here's what we'd carry over from our nine challenges: Authenticate every tool call, even in dev. We used HMAC signing on every request from agent to MCP server. The reason to do this early isn't just production security — it surfaces auth integration bugs during development, when they're cheaper to fix. Use structured JSON for audit logs, not text. The audit format we used (incident ID, agent ID, alert, decision chain, resolved flag) made it possible to write an analyzer that validates agent behavior automatically. Plain text logs would have made that impossible. Set rate limit low. We used Redis with 100 requests per hour per agent. Agents can make a lot of tool calls quickly — a single bug in the system prompt triggered thousands of calls in one of our early runs before we noticed. Default to escalation when uncertain. A false-positive escalation costs you a page that turns out to be nothing. A false-negative auto-fix can mask a real problem for weeks. The costs aren't symmetric, so the default shouldn't be either. Validate against expected behavior. Write down what you expect each failure mode to do, then write an analyzer that checks the audit log against that spec. We open-sourced ours — it's about 250 lines of Python, no external dependencies. You can adapt it to any agent that produces structured audit logs. Tighten conversation turn limits. max_consecutive_auto_reply=3 is a sane starting point for production. The agent should do its job and then the conversation should end. Frameworks default to longer because they're optimized for conversational AI demos, not production ops. What's Still Missing This article would be marketing if we didn't include this section. Honest engineering means owning what isn't built yet. No Docker Scout MCP server exists yet. Security-aware container discovery — "find the most secure nginx tag," "show me CVEs in this image" — isn't possible through MCP today. The Docker Hub MCP server has 13 tools, but none of them surface vulnerability data. This is a real gap in the ecosystem. No incident memory or pattern recognition. Our agent treats every incident as fresh. A production system would learn that this container OOMs every Tuesday at 4 pm and recommend a permanent memory increase rather than reactively bumping it each time. We've left this as future work. Sample sizes are small. Our 6 post-fix incidents prove the expected behavior is reproducible across the four scenarios. They don't prove reliability under production load, traffic spikes, or adversarial conditions. We'd need 100x more data and load testing to make those claims. MTTR is unmeasured. AutoGen records all decision-chain timestamps within microseconds of each other, so the per-incident duration data we collected isn't usable as a real mean-time-to-recovery metric. Capturing real MTTR would require external timing instrumentation around the agent. Gateway and tools are colocated. Our MCP server bundles the security pipeline (HMAC, rate limiting, audit) with the tool execution. In a true multi-tenant production setup, you'd separate these into distinct services so they can scale independently. Our current architecture is fine for a single team or environment; it would need refactoring before serving multiple agent populations. What This Means for AI Infrastructure The interesting part of building agentic infrastructure isn't getting the agent to act. It's getting it to not act when acting would make things worse. Docker MCP Gateway is one of the first production tools that takes this seriously — treating the infrastructure around the agent as the security layer, not the agent itself. The pattern we ended up with — a Gateway in front, scoped tools, decision boundaries written into the system prompt, structured audit logs — isn't novel. It's just what worked. We expect most production AI agents will end up looking similar, because this is what makes them debuggable when something goes wrong. The nine challenges we documented in the lab README are probably challenges you'll hit too. The analyzer script, the audit log format, and the validation patterns are all MIT-licensed in the companion repository. Use whatever's useful. This article was originally published on OpsCart.

Picture this: two features are being developed in parallel. One has already been tested in lower environments, but is still awaiting business approvalThe other is fully validated and ready to go live Naturally, you want to release the second feature to production. But you can’t, because your deployment model forces you to release everything together. If you’ve worked with Azure Data Factory (ADF), this situation probably sounds familiar. Azure Data Factory (ADF) is a cloud-based data integration service from Microsoft that helps you build and orchestrate data pipelines across systems. It works extremely well for managing data workflows — but when it comes to deployments at scale, things get tricky. As our ADF usage grew across multiple teams and environments, we started running into a recurring problem: We had control over development — but very little control over what actually got deployedA simple pipeline fix could unintentionally introduce unrelated changesParallel feature development became harder to manageProduction releases became riskier than they needed to be That’s when we realized: The issue wasn’t ADF itself — it was the deployment model we were relying on. The issue wasn’t ADF itself — it was the deployment model we were relying on. This article walks through how we addressed that challenge by implementing a selective deployment pattern, allowing us to promote only intended changes without impacting everything else. The Real Problem: Parallel Feature Releases in ADF Before diving into the solution, let’s look at a scenario that frequently occurs in real-world teams. What This Diagram Represents This diagram shows two features progressing across environments: Feature 100 Developed earlier, successfully deployed to Dev and TestCurrently in UAT (User Acceptance Testing)Still awaiting business approval before production Feature 200 Developed later, successfully completed across Dev → Test → UATFully validated and ready for production Expected Behavior At this stage, the expectation is straightforward: “Let’s release Feature 200 to production.” Feature 100 is still under testing, so it should remain in UAT. What Actually Happens in ADF Azure Data Factory follows a full-state deployment model. That means when you deploy, you are not deploying a feature; you are deploying the entire factory state. So when you attempt to release Feature 200: Feature 100 gets included automaticallyYou cannot isolate Feature 200You lose control over what reaches production Why This Becomes a Real Problem This isn’t an edge case; it becomes a recurring pattern in larger environments. You’ll encounter this when: Multiple teams are working in parallelFeatures move at different speedsUAT cycles varyProduction fixes need to be released quickly It becomes even more complex when: Existing production pipelines are modifiedPartial updates are requiredDependencies overlap across features The Core Limitation: ADF promotes state, not intent. It does not differentiate between what is ready for production and what is still under testing. Why We Had to Rethink Deployment This limitation introduced real risks: Accidental promotion of incomplete featuresDelayed production releasesIncreased coordination overheadHigher chances of breaking stable pipelines We needed a way to: Promote only Feature 200Keep Feature 100 in UATAvoid impacting unrelated artifactsReduce production risk Architecture Overview To address this challenge, we introduced a selective packaging layer between build and deployment. Flow Feature Branch → PR → Validate → Selective Packaging → ARM Export → Incremental Deploy → Trigger Control Key Idea: Instead of exporting ARM templates from the full ADF repository, we export from a filtered staging folder containing only the required artifacts. Understanding Default ADF Deployment Behavior Before implementing selective deployment, it’s important to understand how Azure Data Factory works by default. ADF follows a full-state deployment model. How Default ADF Deployment Works When you use ADF with Git integration: Developers work in a collaboration branch (typically main)Changes are committed and merged via pull requestsADF provides a Publish button in the UI When you click Publish, ADF generates ARM templates representing the entire factory state. These templates are stored in the adf_publish branch: In modern setups, instead of clicking Publish manually, teams often use @microsoft/azure-data-factory-utilities (npm-based export). This allows pipelines to validate ADF resources and export ARM templates programmatically. YAML - name: Validate ADF resources run: | set -euo pipefail FACTORY_ID="/subscriptions/${{ env.SUBSCRIPTION_ID }/resourceGroups/${{ env.RESOURCE_GROUP }/providers/Microsoft.DataFactory/factories/${{ env.SOURCE_FACTORY_NAME }" npm run build validate "${{ github.workspace }" "$FACTORY_ID" YAML - name: Export ARM templates (CI publish) run: | set -euo pipefail FACTORY_ID="/subscriptions/${{ env.SUBSCRIPTION_ID }/resourceGroups/${{ env.RESOURCE_GROUP }/providers/Microsoft.DataFactory/factories/${{ env.DEV_FACTORY_NAME }" npm run build export "${{ github.workspace }" "$FACTORY_ID" "${{ env.ARM_OUTPUT_DIR }" Whether you click Publish manually or use npm export in CI/CD, the outcome is the same: Full factory deploymentNo control over individual featuresAll changes get bundled together Selective Deployment Layer (Core Design) We can address this requirement and the associated challenges by introducing a workflow driven by a manifest to define the deployment scope, and a program to identify all necessary ADF dependencies for each manifest file. As a developer, I can now control which release is promoted to production, without worrying about releasing any other features that are not ready. The manifest controls which pipelines to deploy and which optional categories to include. Below is an example of a manifest file JSON { "pipelines": ["pl_ingest_population_selective"], "includeTriggers": false, "includeIntegrationRuntimes": false, "includeAllGlobalParameters": true, "includeLinkedServices": true, "validateLinkedServicesExist": true, "includeManagedVirtualNetwork": false, "includeManagedPrivateEndpoints": false } Workflow Explanation Let's understand the crux of the selective deployment workflow now. I am working in the release branch on my feature branch directly in ADF Studio. Since ADF Studio is integrated with Git, my development changes will be saved to my branch. Here are the steps I can take to promote my change to a higher environment. 1) Validation of ADF on PR validation This is an early validation step and a guardrail: if the PR fails, it's because objects are invalid and misaligned. This is equivalent to the "validation all" button in the ADF ui, here is this workflow Trigger: Pull requests targeting the branch selective_deployment. Purpose: Validate that the ADF JSON in the PR is valid in the context of the target factory. Main steps: CheckoutSet up Node.js 20npm installAzure login using OIDC (azure/login@v2)Validate with ADF Utilities: YAML FACTORY_ID="/subscriptions/${AZURE_SUBSCRIPTION_ID}/resourceGroups/${AZURE_RESOURCE_GROUP}/providers/Microsoft.DataFactory/factories/${DEV_FACTORY_NAME}" npm run build validate "$GITHUB_WORKSPACE" "$FACTORY_ID" 2) Release build + selective deploy to DEV adf-release-build-selective-deploy.yml Triggers: Push to selective_deploymentManual run (workflow_dispatch) with optional manifest inputDefault: deploy/manifests/release.json This workflow has two jobs: Job A: adf-build (staging + export + sanitize + artifacts) Checkout (full history)Azure login using OIDCSet up Node.js 20Install build dependencies inside build/ (npm install in build)Stage selective subset python scripts/select_adf_subset.py <manifest>, a code snippet below for the complete script, refer to the GitHub repository link given Python import json import re import shutil import sys from pathlib import Path from typing import Dict, Set, Tuple, List from collections import defaultdict # Your repo layout has pipeline/, dataset/, linkedService/ at ROOT. REPO_ROOT = Path(".") STAGE_ROOT = Path("build/adf_subset") RESOURCE_DIRS = { "pipeline": REPO_ROOT / "pipeline", "dataset": REPO_ROOT / "dataset", "linkedService": REPO_ROOT / "linkedService", "dataflow": REPO_ROOT / "dataflow", "trigger": REPO_ROOT / "trigger", "integrationRuntime": REPO_ROOT / "integrationRuntime", "credential": REPO_ROOT / "credential", "managedVirtualNetwork": REPO_ROOT / "managedVirtualNetwork", } # Copy these if present so ADF utilities behave the same on staged subset. ROOT_FILES_TO_COPY = [ "publish_config.json", "arm-template-parameters-definition.json", "arm_template_parameters-definition.json", "package.json", "package-lock.json", ] Produces: build/adf_subset/ (staged tree)build/adf_subset_report.json (dependency report)Refer to logs below (showing output of stage selective subset and debug to view output generated after select_adf_subset.py )Export ARM templates from the staged subset via ADF Utilities: npm --prefix build run build -- export "adf_subset" "$FACTORY_ID" "ArmTemplate"Produces: build/ArmTemplate/ARMTemplateForFactory.jsonbuild/ArmTemplate/ARMTemplateParametersForFactory.jsonStrip infra-owned resources scripts/strip_arm_resources.py to produce a safe template: build/ArmTemplate/ARMTemplateForFactory.safe.json⚠️ Note on Infrastructure Components (Refer to the “Future Work & Next Steps” section for follow-up topics in this series) The step above intentionally strips infrastructure-dependent components from the generated subset to avoid overwriting existing shared resources such as linked services. This implementation focuses on developer-owned artifacts (pipelines, datasets, and triggers) and assumes that infrastructure components — such as Integration Runtimes, managed private endpoints, and linked services — are pre-provisioned and managed outside of this deployment workflow.Upload artifacts: ARM templates (adf-arm)metadata (adf-release-meta)subset report (adf-subset-report) Job B: deploy_dev (deploy safe template) Download ARM artifactAzure login using OIDCEnsure az Data Factory extension is installedValidate JSON files exist/parseDeploy via azure/arm-deploy@v2(Incremental) to DEV RG/factory: Template: ARMTemplateForFactory.safe.jsonParameters: ARMTemplateParametersForFactory.json + factoryName=<DEV_FACTORY_NAME> Lesson Learned Setting up selective deployment in ADF was more than a technical task. It made us rethink our approach to deployments, ownership, and CI/CD design. Here are the main things we learned: 1. The Problem Is Not Tooling; It’s Deployment Granularity At first, we thought the limitation came from the tools we used, like UI publish or npm export. However, both methods yielded the same result: full factory templates. The real problem was that we couldn’t control the scope of deployments, not how the templates were made. 2. Dependency Awareness Is Critical Selective deployment only works when every dependency is found and included. We learned that: Pipelines often reference multiple datasets and linked services. Missing even one dependency results in deployment failure You must automate dependency discovery. 3. “Incremental” Is Often Misunderstood Incremental deployment is important, but it doesn’t work like a patch. It reapplies the full configuration for all included resources. This means: Your generated templates need to be complete for all the artifacts you include. If you use partial definitions, deployments can fail. 4. Separation of Concerns Is Key Not all ADF artifacts are the same. We began to separate them into different groups: Application-owned artifacts: pipelines, datasets, triggers Infrastructure-owned artifacts: linked service, managed virtual networks, managed private endpoints, and integration-runtime, among others. This separation proved crucial for safe, scalable deployments. 5. Selective Deployment Adds Complexity, But It’s Worth It It’s true that implementing this approach brings in additional scripts, manifest management, and CI/CD complexity. But in exchange, we gained precise control over releases, reduced production risk, and faster hotfix deployments. Future Work and Next Steps While selective deployment solved a major gap in ADF CI/CD, it also opened up new areas for improvement and standardization. 1. Defining Infrastructure vs Application Ownership One of the biggest follow-up areas is clearly defining ownership boundaries. In our experience: Application teams should own pipelines, datasets, and triggers Platform or infrastructure teams should own linked services, managed virtual networks, and managed private endpoints, among other things. Future work can focus on: Enforcing this separation in CI/CD. Preventing accidental deployment of infrastructure components Integrating Terraform or platform pipelines for infrastructure provisioning 2. Governance Around Linked Services Linked services are often shared across multiple pipelines and teams. Future improvements include: Centralizing linked service management Using Key Vault and Managed Identity consistently Preventing direct modifications through application pipelines

When I decided to move into AI infrastructure, nobody warned me that I had to relearn how to think about compute. I proceeded with the usual steps, such as spinning up VMs, configuring networking, and managing costs. But then a moment came, and I watched, slightly horrified. I misconfigured the inter-node networking. The result was that an eight-node GPU ran a training job at just 11% GPU utilization. It was a wake-up call for me. AI workloads aren’t just different in a marketing sense. They’re different where it counts, i.e., in the architecture — how you build and run things. The ML engineers on that project immediately assumed the model was the problem. They decided to redesign the model and spent a couple of days tweaking the architecture, like chasing a ghost. The real issue resurfaced only when someone checked the network telemetry — the cluster nodes were using standard Ethernet, not InfiniBand. The model had no issues. The infrastructure configuration was incorrect. After years of working with Azure and a period on AWS before that, I wish someone had given me a cheat sheet before starting that project. Compute: Breaking Down the Model Many cloud engineers assume that AI infrastructure requires larger VMs: more cores and more memory, and the workload will run. This approach is insufficient. While right-sizing CPUs remains relevant, it now accounts for only about 20% of considerations. The remaining 80% is driven by GPUs, which operate fundamentally differently from CPUs and significantly impact the infrastructure. A GPU isn’t just a faster CPU; it's a collection of thousands of smaller cores working together to handle large datasets. If any part of your system—such as storage speed, network bandwidth, or data preprocessing—can't keep up, the GPU remains idle, incurring huge unwanted costs. On Azure, idle GPUs cost as much as active ones. Usually, the main limitation in AI infrastructure isn't the GPU itself, but the upstream systems that supply data to it. When working with Azure, you'll mostly use two main GPU families. The NC-series gives you a single A100 per VM at about $3.60 per hour on demand, making it the go-to choice for fine-tuning and inference tasks. The ND-series has eight A100S that are connected through NVLink and InfiniBand, which is perfect for distributed training. If your cluster uses regular Ethernet instead of InfiniBand between nodes, inter-GPU bandwidth can drop by 60 to 70 percent, and Azure may not warn you about this. It’s smart to double-check that your cluster is set up with InfiniBand before starting a multi-node run and to make sure your GPU quota is ready ahead of time. Storage: Where Training Jobs Are Exhausted When you’re training a language model, expect to chew through the dataset over and over — think of it as laps around a track, not a sprint. If you try to pipe 500GB of text straight from regular Azure Blob Storage, you’ll quickly find yourself staring at a progress bar that barely budges. Each blob tops out at about 60 megabytes per second, but an A100 GPU can eat data for breakfast at several gigabytes per second. There’s a massive mismatch. If you want to keep your GPUs busy (and not just waiting around), you’ll need something beefier — Azure Managed Lustre fits the bill, since it can dish out data to your training jobs at speeds regular storage can’t dream of. I’ll admit, the first time I ran into this, I wasted hours on model tweaks before realizing the bottleneck was staring me in the face the whole time. Model checkpoints are a cost trap that is often overlooked. A single checkpoint for a 7B parameter model is around 28GB. Saving checkpoints every 30 minutes over 72 hours generates more than 4TB of data. Configure a Blob lifecycle policy before you start to avoid unexpected storage costs. Networking: Two Problems, One Person Responsible During training, each GPU shares gradient updates with the others in the cluster via AllReduce. The efficiency of the cluster is directly determined by the bandwidth and latency of this communication. If this communication is disrupted, GPU utilization drops. Machine Learning teams often attribute this to model architecture issues, such as an excessive number of parameters or an incorrect batch size, but the network is usually the cause. First, assess network performance and address any issues before the job runs to avoid unnecessary model design, as ML engineers may not consider this when monitoring loss curves. The second networking problem is well known among cloud engineers. Many enterprise clients in financial services and healthcare require AI services that avoid the public internet. Azure AI services, such as Azure OpenAI, Azure ML, and Azure AI Search, all support Private Link, and the configuration process is identical to that of other PaaS services. The key consideration is to integrate private endpoint DNS zones with existing private DNS or manage them manually. ML engineers may interpret a generic “connection refused” error caused by an incorrect DNS configuration as an API issue. Both inter-GPU bandwidth and private network isolation — critical infrastructure concerns — typically fall under the same person’s responsibility. The Azure AI Services Stack: Known Infrastructure, Unknown Branding Recent Azure services such as OpenAI Service, Machine Learning, and AKS with GPU node pools might sound new, but for most infrastructure teams, the actual work remains familiar. The phrase “managed service” sometimes suggests that everything is taken care of, but in reality, only the AI model is managed. Everyday responsibilities like network security, permissions, cost tracking, and system monitoring still rest with your team, no matter how polished the portal looks. Azure OpenAI Service works much like other managed API endpoints, supporting private connections, role-based access, managed identities, and API Management for controlling usage rates. The main distinction is its use of Provisioned Throughput Units (PTUs) — these reserve GPU resources to guarantee performance. If you see HTTP 429 errors, it’s almost always a sign of resource bottlenecks rather than issues in your code, although the latter is a common assumption. Azure Machine Learning sits on top of other infrastructure stacks, such as Blob Storage, ACR, Key Vault, and compute, which you already manage. The failure mode is unique to Azure ML: the compute cluster lifecycle. Ensure clusters auto-scale to zero when idle. Unfortunately, this is not the default setting. When a bill arrives with huge costs due to a cluster running overnight because of an unset idle timeout, everyone looks to the cloud engineer first. While it’s tempting to go with Azure Container Apps for their apparent simplicity, most real-world inference workloads ultimately end up on AKS with GPU node pools. The reason? Container Apps are easy—that is, until you’re hit with cold start lag during actual user traffic and realize spinning up a GPU container on the fly just isn’t fast enough to meet your SLA. With AKS, you get far more say over things like keeping node pools warm, tuning autoscaling, and controlling scheduling—options that simply aren’t available with Container Apps. Costs: Higher Stakes, Faster Exposure Eight GPUs on an ND-series cluster aren’t cheap — about $27 an hour adds up quickly. A few long training runs and you’re already close to $2,000, and if you’re running a batch of experiments, $20,000 can disappear before anything launches. The price tag often slips by until accounting points it out. When models underperform, it’s easy to blame the architecture, but I’ve learned to glance at GPU usage first. If you’re seeing less than 60% during distributed runs, chances are the bottleneck is in the infrastructure, not the model itself. If you want to slash costs, spot VMs can drop your bill by as much as 90%. The catch? Your training jobs must be able to handle abrupt interruptions—so regular checkpointing and clean restarts are a must. If that’s not in place, spot isn’t the way to go—sort it out with your ML team before finance starts asking questions. Reserving GPU resources is a whole different equation than CPUs: GPU supply changes from region to region, and with how quickly AI hardware evolves, locking in a three-year reservation on today’s gear is a real gamble. Security: Same Toolkit, New Attack Surface For AI projects, you still need the basics like private networks, Managed Identity, strong RBAC, and encryption. But now there’s a twist: prompt injection. It’s like the old trick with SQL injection, but for language models. Someone might simply ask a chatbot to show its system prompt. If you haven’t set up protections, it could actually answer. Firewalls won’t help here. Azure Content Safety can block some of these risky requests, but most teams don’t use it until after trouble starts. If you’re in a regulated industry, logging every inference is a must. In finance or healthcare, you need to record inputs, outputs, who did what, and when, so auditors have all the details they need. Decide on your schema and retention policy before going live, because adding it later, after compliance comes calling, is always a headache. The ML engineers on these teams know the models well. But when infrastructure acts up, causing higher costs, slowdowns, or new risks, they're often the last to spot the cause. Closing that gap is the real challenge. For cloud engineers, "architecturally different" isn’t a red flag; it’s a chance to improve.

Most infrastructure teams have a moment where someone says “we should build a platform.” The motivation is real: teams are duplicating work, the current setup is hard to use consistently, and a more structured approach would help. A few months later, the platform is a Terraform module collection, a GitLab CI template, a shared repository of scripts, and a README that several people have tried to keep current. That is a useful thing. It is not a platform. The distinction is worth being clear about, not to dismiss the work, but because the word “platform” creates expectations. When internal teams hear “we have a platform,” they assume stability, a usable interface, a versioning model, and some mechanism for raising problems when things break. A toolchain with documentation does not deliver those things by default. What Makes Something a Platform A platform is defined by its contract, not its technology. The contract describes what the consumer can expect: what they call, what parameters they provide, what outputs they receive, and what stability guarantees apply to that interface. A Terraform module with a published interface is closer to a platform primitive than a pipeline that provisions the same resources through environment variables, undocumented flags, and positional arguments. The module has a contract. The pipeline has a process. The contract does not have to be formal. It needs three things. A stable surface. Consumers should be able to call the same interface next month and receive the same type of result. Internal changes to how it works do not break consumers.A versioning model. When the interface changes, that change is communicated, and consumers are not silently broken. A git tag is enough to start with. Semantic versioning is better.A feedback path. Consumers can report when the contract is violated or the interface does not behave as documented. Someone is responsible for responding. A Terraform module with these three properties is a platform primitive. A set of modules with a shared versioning model, a stable registry entry, and a team responsible for maintaining the contract is starting to look like a platform. What Teams Actually Experience The gap between a toolchain and a platform shows up in how teams actually use it. With a toolchain, onboarding a new team means pointing them at the repository and telling them to read the README. Anything not in the README requires asking someone who has been around for a while. Changes to the toolchain break existing consumers silently because there is no versioning model. The team that maintains the toolchain treats every consumer as having kept up with the latest state of the repository. With a platform, onboarding means pointing teams at interface documentation with a working example. Changes go through a version increment. Consuming teams that pin to a version are not broken by changes they did not ask for. Plain Text # Consuming a module with a pinned version module "vm" { source = "registry.example.com/hybridops/vm/proxmox" version = "~> 2.1" name = "web-01" cores = 2 memory = 4096 } This looks like a small detail. For teams consuming infrastructure modules across a growing estate, it is the difference between a managed dependency and a shared folder everyone is afraid to touch. When a Toolchain Is the Right Call Not every infrastructure system needs to be a platform. A toolchain is appropriate when the team is small and holds the full mental model, the surface area is limited, and the rate of change is low enough that everyone stays current without a formal versioning model. When those conditions hold, the overhead of maintaining a platform contract is not justified. The problem is not having a toolchain. The problem is calling it a platform when it is not, and then finding that the expectations it created are not being met. Teams told they have a stable platform, then hit with a broken workflow from an unannounced change, lose confidence quickly. That confidence is hard to rebuild. HybridOps has been working in this space: publishing Terraform modules to a registry, versioning releases, and treating module interfaces as contracts. It is not a finished platform. It is a direction, and being explicit about that direction changes how the work gets done. A Simple Test If a consuming team pins to the current version of your toolchain today, will it still work in three months without any changes on their side? If you cannot answer yes with confidence, you have a toolchain, not a platform. Both are useful. Only one creates the kind of trust that makes a growing engineering organisation move faster rather than slower. Knowing which one you have is the first step toward building the right one.