DevOps and CI/CD Resources

DZone's Featured DevOps and CI/CD Resources

Docker Containers Don’t Know Your Model Is Still Loading

By Pruthvi Raj Seknametla

It was a Friday at 4:50 pm, the worst possible time for anything to go sideways when marketing flipped on a new AI summarization feature for the whole user base instead of the 5% rollout we'd agreed on. Traffic to our LLM service doubled in about four minutes. The autoscaler did exactly what it was told: it spun up three new replicas. What it didn't account for is that each replica needed almost three minutes just to pull a 14GB checkpoint and warm up CUDA kernels before it could answer a single request. The load balancer, seeing new pods report as running, immediately started routing traffic to them. For three minutes, a chunk of our users got 504s while perfectly healthy-looking pods sat there loading a model into memory. Nobody on the infra side had touched Docker that day. The incident wasn't a Docker bug. We assumed that container orchestration designed for web services would function the same way for processes that take minutes to become useful, rather than those that operate in milliseconds. Why LLM Containers Break the Usual Assumptions Packaging an LLM serving stack in Docker still makes sense for the same reason it always has; CUDA versions, driver compatibility, and Python ABI mismatches are miserable to manage across a fleet without a frozen artifact. But an LLM container carries baggage that a typical inference service doesn't. The weights are tens of gigabytes, not a few hundred megabytes. GPU memory is a single shared pool that one greedy container can quietly exhaust for everyone else on the box. And “ready” doesn't mean “process started”; it means the model is resident in VRAM and the CUDA graph is warmed, which can take minutes on a cold node pulling weights from object storage over the network. The Mistakes, in Order Our first version baked the model weights directly into the image, because it felt simpler: one artifact, one pull, done. In practice, it meant a 16GB image, painfully slow CI pushes, and a registry bill nobody wanted to look at. Worse, every time we bumped into a new fine-tuned checkpoint, we rebuilt and repushed the entire layer regardless of caching, because the COPY step touching gigabytes of weight files invalidates everything below it. Unlike a typical ML inference image, there's no meaningful caching win here at all; the layer is simply too big to ever be a cache hit across versions. We moved weights out to a mounted volume, fetched at container start from object storage, and never looked back. Second mistake, and this one actually cost us a production incident: we ran the container with Docker's default shared memory size. vLLM, which we used for serving, spins up worker processes that talk to each other over shared memory even on a single GPU. With the default 64MB /dev/shm, those workers would crash with cryptic bus errors under any real concurrency. The fix was almost embarrassingly small: Shell docker run --gpus all \ --shm-size=2g \ -e MODEL=mistralai/Mistral-7B-Instruct-v0.2 \ -e GPU_MEMORY_UTILIZATION=0.85 \ -e MAX_MODEL_LEN=8192 \ -p 8000:8000 \ llm-serve:latest The third mistake was more subtle and took longer to diagnose. vLLM's continuous batching reserves a large slice of GPU memory upfront for the KV cache, controlled by gpu_memory_utilization. We'd set that fraction high to maximize throughput, then bin-packed two replicas onto the same GPU to save cost. Under normal traffic, fine. During a burst of unusually long-context requests, such as someone summarizing a 6,000-word document instead of a tweet, the KV cache for that single batch ballooned, causing the container to run out of memory (OOM) mid-generation and taking down every other in-flight request in the same batch. This failure mode is more severe than a typical web service OOM because it not only drops the new request but also terminates queries that were already halfway through generating answers for paying customers. What We Actually Changed The readiness adjustment turned out to matter more than any Docker flag. We split liveness from readiness: liveness just checks that the process hasn't died; readiness fires a real, tiny generation request through the local API and only flips to healthy once that round trip succeeds. That alone killed the cold-start routing problem because the load balancer stopped trusting a merely alive process. We also gave up on bin-packing two replicas per GPU. In hindsight, treating GPU memory like it's as elastic as CPU or RAM was the actual root cause, not any single Docker setting. We implemented a model that uses one GPU, sets a conservative memory utilization ceiling, and enforces a request-level token limit at the proxy in front of the container, rather than inside it, because it is too late to make adjustments once the batch is already running. On the orchestration side, we stopped trying to scale-to-zero or scale aggressively off CPU-style metrics. Scale-to-zero is effective for web apps but doesn’t fit GPU-bound LLM serving, where cold starts can outlast traffic spikes. We kept a warm floor of replicas sized to baseline traffic and let a request queue absorb bursts instead of expecting new pods to materialize in time. It's less elegant than the autoscaling story everyone likes to tell, and it costs more in idle GPU time, but it's honest about what the hardware can actually do. What We Rejected, and Why We seriously considered dropping self-hosting altogether and routing through a managed inference API. For a side project, that's probably the right call — less to own, no GPU bin-packing headaches. We rejected it due to data residency requirements that prohibited sending raw text to a third party, and at our volume, managed pricing would quickly exceed our GPU costs. We also looked at Ray Serve and Triton early on, and they solve some of the issues more natively, but the team's Docker and Kubernetes muscle memory was strong enough that rebuilding on a new serving framework felt like trading one set of unknowns for another, at least for the first version. Key Takeaways Never bake multi-gigabyte model weights into the image — there's no caching benefit at that size, only slower pushes and bigger registry bills.Set shared memory explicitly; vLLM and similar multiprocess servers will fail under load with Docker's tiny default.Treat GPU memory utilization conservatively and avoid bin-packing replicas onto a single GPU unless you can guarantee a strict ceiling per container.Build a readiness assessment that performs a real generation, not just a process check; cold model loading will otherwise receive routed live traffic.Don't expect autoscaling to save you on cold-start timescales measured in minutes; a warm floor plus a queue is more honest than reactive scaling. Closing Thought None of these issues was really a Docker failure; the container did exactly what we told it to do. The failure was treating a multi-gigabyte, GPU-bound, slow-to-warm process like it was just another stateless web container that happens to need a GPU flag. I suspect that many teams will learn this lesson in the same way we did, during an incident on a Friday afternoon. Is it the right move to keep stretching Docker and Kubernetes to fit LLM serving, or is this the workload that finally pushes most teams toward purpose-built serving layers? More

Why LLM Pipelines Fail in Production and How Temporal and Kafka Fix Them

By Akhil Madineni

A production LLM pipeline is rarely just a prompt and a response. It typically combines retrieval, prompt rendering, model inference, output shaping, validation, persistence, and downstream actions. That broader shape is why many systems look stable in a demo and then become fragile under live traffic. The model call is only one component; the operational problem is the workflow around it. Provider APIs impose rate limits, structured outputs still need application-level checks, and external calls introduce failure ambiguity that ordinary request-response code does not handle well. Where the Breakage Starts Most production failures happen between steps, not inside the prompt. A request enters an API, context is loaded, a model call is sent, the response is parsed, a downstream action is triggered, and a record is written. If the provider generated output but the network dropped before the caller saw it, the system no longer has a clean answer to whether the operation should be retried or treated as complete. Kafka’s default delivery model is at least once, and Temporal’s documentation is explicit that activities may be retried and therefore should be idempotent. That combination makes duplicate side effects the default risk unless the pipeline is designed around durable state and idempotent writes. Duration creates the second breakage pattern. Ingestion may fan out across thousands of chunks, while a risky action may need approval hours later. Temporal workflows can receive external write events through Signals, and durable timers persist across worker and service downtime, so a workflow can pause without collapsing into callback code and scheduled cleanups. Temporal also requires workflow logic to remain deterministic during replay and provides versioning methods so new executions can adopt new code while long-running executions remain on compatible paths. Those concerns are not edge cases in LLM systems; they are normal once the pipeline extends beyond a single synchronous call. Output shape is another common source of confusion. OpenAI’s Structured Outputs guide exists because unconstrained text is not a reliable contract; the feature is designed to enforce a supplied JSON Schema and avoid missing required keys or invalid enum values. But schema compliance is only the first gate. A response can be structurally valid and still be semantically wrong, stale, or unsafe to automate. Production failures happen when formatting success is mistaken for business correctness. Why Kafka solves only part of it Kafka is a strong fit at the ingestion boundary because it turns synchronous pressure into a durable stream of work. Kafka topics are partitioned, ordering is guaranteed within a partition, and each partition is consumed by exactly one consumer in a consumer group at a given time. Consumers also control offsets and can rewind to replay records. That combination is well suited to bursty LLM demand, key-based ordering, and reprocessing after a model or prompt change. Java public void submitRequest(LlmRequest request) { LlmRequestEvent event = new LlmRequestEvent( request.requestId(), request.tenantId(), request.documentId(), request.templateId() ); kafkaTemplate.send("llm.requests", request.requestId(), event); } This pattern keeps the API narrow. The service records intent by publishing an event keyed by requestId; Kafka’s default partitioning uses the key hash, so related records land on the same partition and preserve that partition’s order. Kafka’s producer is also optimized for batching, and its pull-based consumer model lets downstream services fall behind and catch up instead of being overwhelmed by broker-driven push traffic. That is useful when inference latency varies, and demand arrives in bursts. But Kafka only states that work was published and later consumed. It does not know whether retrieval already succeeded, whether a model provider timed out after actually producing output, or whether persistence ran before a crash. Offsets capture consumption position, not business completion. Kafka is excellent for transport, buffering, replay, and fan-out, but insufficient as the sole control plane for a multi-step inference process. Why Temporal Changes the Outcome Temporal addresses the state problem directly. Its model is durable execution: workflows advance through an event history stored by the Temporal service, and that history is what allows an execution to recover from a crash and continue making progress. Worker crashes, network interruptions, and infrastructure outages are handled differently from ordinary application failures because the workflow state is not reconstructed from logs after the fact; it is already part of the execution record. Java @KafkaListener(topics = "llm.requests") public void onRequest(LlmRequestEvent event, Acknowledgment ack) { InferenceWorkflow workflow = workflowClient.newWorkflowStub( InferenceWorkflow.class, WorkflowOptions.newBuilder() .setWorkflowId(event.requestId()) .setTaskQueue("llm-inference") .build() ); try { WorkflowClient.start(workflow::run, event); } catch (WorkflowExecutionAlreadyStarted ex) { log.info("Workflow already started for {}", event.requestId()); } ack.acknowledge(); } The important detail is the workflow identifier. Temporal guarantees workflow ID uniqueness within a namespace and prevents another open workflow with the same ID from starting, which turns duplicate Kafka deliveries into a safe re-entry case instead of parallel duplicate execution. The Kafka listener acknowledges the record after the workflow start is accepted, not after the entire inference path finishes. Kafka remains the transport layer; Temporal becomes the durable execution layer for the request. Inside that workflow, each external operation should sit in an activity with explicit timeout and retry policy rather than inside scattered retry loops. Temporal’s retry model is declarative; activities retry by default, and the platform documentation recommends making activities idempotent and granular because a retry re-executes the whole activity. That fits LLM systems unusually well, since retrieval, prompt rendering, model invocation, validation, and persistence rarely fail for the same reason. Java private final LlmActivities activities = Workflow.newActivityStub( LlmActivities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(45)) .setRetryOptions(RetryOptions.newBuilder() .setMaximumAttempts(5) .setInitialInterval(Duration.ofSeconds(2)) .build()) .build() ); public InferenceResult run(LlmRequestEvent event) { Context context = activities.loadContext(event.documentId()); Prompt prompt = activities.renderPrompt(event.templateId(), context); ModelResponse response = activities.callModel(prompt, event.requestId()); return activities.validateAndPersist(event.requestId(), response); } That separation also makes the final validation step honest. Structured output may guarantee a schema, but business rules still need to decide whether the result is usable, and persistence still needs an idempotent write keyed by requestId so a retry cannot create duplicate approvals, tickets, or rows. In practice, this is the difference between a pipeline that merely retries and a pipeline that resumes safely. Why the Handoff Works in Production The pattern that holds up is a clean handoff. Kafka should own ingress, buffering, replay, and downstream fan-out. Temporal should own the lifecycle of one logical request. When a workflow completes, it can publish a result event for indexing, notifications, analytics, or billing, and each downstream system can consume that event independently. Kafka moves facts through the platform. Temporal ensures the process that produced those facts reaches a correct terminal state. This split also improves the cases that usually appear after launch. Human approval can arrive as a Temporal Signal without keeping compute hot. A workflow can pause for days using a durable timer and continue after downtime. Workflow code can be versioned so new executions take a new path while long-running executions stay on compatible logic, which is important because replay depends on deterministic workflow behavior. The reason LLM pipelines fail in production is not that language models are impossible to operationalize. The reason is that they are often deployed as request handlers when they are actually distributed workflows with uncertain latency, replay risk, and side effects. Kafka fixes the transport problem by providing durable, replayable streams with partition ordering and load decoupling. Temporal fixes the execution problem by persisting progress, surviving worker failure, and making retries, timeouts, and human pauses part of the design instead of a patch applied after incidents. When those two responsibilities are separated cleanly, an LLM pipeline stops behaving like an experimental chain of API calls and starts behaving like a production system. More

Practical QA Workflow Showing How Teams Integrate LLM Testing into Real CI/CD Pipelines

By Minkle Kalra

Understanding Agentic SDLC: The Future of Software Engineering

By Pavan Belagatti

CORE

Containerizing and Testing a Python Backtesting System With Docker and GitHub Actions

By Gillian Lu

Engineering Production Agentic Systems: Part 3: The Topology

Human-in-the-Loop Topology for Production Agentic Systems — Loop Bounding, Approval Gates, and Termination Discipline This is Part 3 of a three-part field manual on engineering production agentic systems. Part 1 took context engineering. Part 2 took guardrails. This part takes human-in-the-loop topology and closes the series. The conviction underneath all three: production agentic systems are won on these three architectural disciplines — not on model choice. The Opening Claim Almost every conversation I have with teams designing agentic systems eventually arrives at the same wrong question: “Should we have a human in the loop?” The answer is yes, always. That is not the design question. The design question is “At which joints, with what context, and with what default behavior on timeout?” The right HIL topology is what separates an agentic system that can run autonomously under regulated-industry scrutiny from one that produces impressive demos and falls apart on the third Tuesday in production. The wrong HIL topology is one of two failure modes: humans approving every action (which kills the leverage agents are supposed to provide), or humans approving nothing (which surfaces every failure mode of unbounded agent execution). The agent-loop divergence failure mode I named in Part 1 is what HIL topology exists to prevent. The agent reasoning loop, left unbounded, generates plausible-but-wrong subgoals that compound across turns. Turn three produces a subgoal that should have been rejected at turn two. Turn five calls a tool whose preconditions don’t actually hold. Turn nine returns a confident output that’s structurally broken. The fix is not “give the agent better instructions.” The fix is explicit loop topology: agent reasoning bounded by deterministic logic, with human approval at the joints, and three independent termination conditions. This part is about how the topology gets built. Five HIL joint types, each with its purpose and default. Three termination conditions, each independent. One closing feedback loop that turns approvals into a training signal. And the multi-agent extension where humans become handoff points between agents as well as gates over actions. The Loop Bounding Problem The cheapest failure mode in agentic systems is also the most common: an agent loop that doesn’t know when to stop. The agent generates a plan, executes a step, evaluates the result, generates a new plan, executes a step, and so on. Each iteration looks locally plausible. The aggregate trajectory drifts. Three things produce loop divergence. First, compounding confidence error — the agent rates its own intermediate outputs as confident, and that confidence does not decay across turns even when the cumulative evidence should reduce it. Second, subgoal generation pressure — when no progress is being made on the primary goal, the agent invents plausible-sounding sub-problems to work on, none of which actually advance the task. Third, tool-call habit — if a tool is on the surface, the agent will eventually find a reason to call it, even when the right action is to stop and ask. There are three bounding techniques that work in production, applied together rather than in isolation. The first is explicit max-turn budgets — every agentic task has a maximum number of loop iterations defined at task initiation, often shorter than teams initially think (a typical resolution-planning task should resolve in five to seven turns, not twenty). The second is deterministic intermediate validators — after every plan-step output, a non-LLM check verifies the plan still aligns with the original task; if it doesn’t, the loop terminates and escalates. The third is joint-by-joint state machines — the loop doesn’t free-run from start to end; it traverses an explicit state graph where each transition is approved either deterministically or by a human. The first technique is cheap. The second is mid-cost. The third is the most expensive to design but also the most defensible — and the one that makes the HIL joints from the next section actually work. HIL Joints as a Design Vocabulary A joint is a position in the agent loop where a human can intervene. Not “could intervene if a flag is set” — will be involved under the conditions the joint specifies. Five canonical joint types, each with its own purpose, context shape, and default behavior on timeout. The Agent Loop With HIL Joints Joint 1 — Planning Approval Trigger: a high-stakes task that requires committing to a strategy before any tool is called. The agent produces a plan; the human approves, edits, or rejects the plan before the agent is allowed to act on it. Context shape: the full proposed plan, the alternative plans the agent considered, and the reasoning trace for the chosen path. Default on timeout: hold; expire after a configurable window and queue for batch review. Joint 2 — Parameter Approval Trigger: a tool with bounded but tunable parameters where the agent’s proposed value falls outside the “soft” range (the range that wouldn’t require approval) but inside the “hard” range (the range that’s allowed at all). Context shape: the tool name, the proposed parameters, and the policy band. Default on timeout: hold; the human edits parameters into the soft range or rejects the call. Joint 3 — Action Approval Trigger: any cost-reversible or irreversible action, per the reversibility matrix from Part 2. This is the most expensive joint, the one the matrix from Part 2 governs in detail. Context shape: the proposed action, its reversibility class, its expected impact, and the precise tool contract being invoked. Default on timeout: hard block — no fallback action runs; the loop pauses indefinitely until a human responds. This is the only joint where indefinite pause is the correct default. Joint 4 — Post-Hoc Review Trigger: a sample of actions of risk-class X, or all actions in audit-required regimes. Context shape: the full event from the audit trail. Default behavior: the action ran; review is queued; if rejected, a rollback or remediation event is emitted. This is where compliance pull-review lives. Joint 5 — Escalation Trigger: the agent’s self-assessed confidence drops below the floor (typical floor is ~0.45 on the calibrated scale), or the same subgoal fails twice in a row, or a deterministic validator flags a plan-step misalignment. Context shape: the full trajectory so far, the trigger reason, and the current state. Default: halt; request human takeover; the agent does not retry without explicit human re-engagement. A snippet showing the joint registration: Python # Pattern — joint registration at task initialization task = AgentTask( goal="resolve_supply_chain_exception", joints={ "plan_approval": Joint(trigger="every plan with risk≥medium", context="plan + alternatives", default="hold"), "param_approval": Joint(trigger="param outside soft-band", context="tool + params + band", default="hold"), "action_approval": Joint(trigger="risk×reversibility matrix", context="action + impact", default="HARD_BLOCK"), "escalation": Joint(trigger="confidence < 0.45 or subgoal_repeat", context="full trajectory", default="HALT"), }, termination=TerminationPolicy(max_turns=12, cost_ceiling="$5.00", confidence_floor=0.45), ) The discipline here is that joints are registered at task initialization, not improvised at runtime. The agent doesn’t get to decide when a human gets involved; the topology decides. Context for the Human The human at a joint is themselves a context consumer. This is the point where the Role Adapter from Part 1 reappears: the Supply Chain Manager at the planning approval joint needs different context than the Logistics Analyst at the assessment joint. The pipeline doesn’t just produce context for the agent; it produces context for the humans who gate the agent. Supply Chain Exception — State Machine with HIL Gates This is the architectural payoff for treating context as a managed pipeline. The same Role Adapter that produces the agent’s view of a hierarchical context tree also produces the human’s view. Different role, different subtree, same underlying data model. The Supply Chain Manager at Joint 3 — action approval for a reroute_container call — sees the proposed action, the cost impact, the customer impact, the SLA implications, and a short rationale chain. They do not see the agent’s full reasoning trace, the alternative plans considered, or the retrieval scores — that’s too much context for the decision the human is being asked to make. The right context for a HIL decision is the minimum context that supports a confident yes-or-no. Not the full trace; not the executive summary; the slice that’s specific to the joint type and the role. Teams that get this wrong tend to default to one of two failure modes: too much context (the human becomes a copy-edit pass over the agent’s full reasoning, which kills the leverage), or too little (the human is asked to approve an action they can’t actually evaluate, so they approve everything, which kills the safety). A practical rule: the context delivered at a HIL joint should be readable in under thirty seconds and decidable in under sixty. If the human at the joint can’t make a confident decision in that envelope, the joint is wrong — either the joint should be earlier (so less context is needed), or the agent should be doing more upstream work to make the decision tractable. The escalation joint (Joint 5) is the exception. When confidence drops or subgoals repeat, the human takes over the whole problem; they need the full trajectory. The expected human action there is not approve-or-reject; it is resume-and-redirect. Context shape is correspondingly richer. Termination Conditions The cheapest defense against agent-loop divergence is three independent termination conditions composed in AND-not-OR. Any single one of them breached, the loop terminates immediately. Termination Conditions — Three Independent Floors Cost Ceiling A budget on the total token spend, API call count, or dollar cost the loop is permitted to incur. Set at task initialization based on the task’s expected value. A routine status update should not exceed five turns’ worth of cost; a complex multi-stage resolution might cost the equivalent of fifteen turns. The ceiling is the budget, not the expected spend; breach it, and the loop terminates regardless of state. Turn Limit A hard maximum on loop iterations. This is the simplest and most underused failsafe. A typical task should resolve in five to seven turns; setting the limit at twelve gives the agent room to recover from one wrong path without removing the bound. Setting it at fifty defeats the purpose. The turn limit should be tight enough that hitting it surfaces a real problem, not lax enough that it’s a theoretical bound the agent never reaches. Confidence Floor The agent’s self-assessed certainty in its current trajectory, evaluated each turn. If the floor is breached, the loop terminates and the escalation joint fires. The floor’s calibration matters more than its absolute value — a confidence score of 0.7 from a well-calibrated agent means something different than 0.7 from a poorly calibrated one. The calibration is itself an artifact of the Evolution Layer’s feedback ingestion (next section). Python # Pattern — per-turn termination check def should_terminate(self, state: AgentState) -> Optional[str]: if state.cost_used >= state.cost_ceiling: return "cost_ceiling_breached" if state.turn_count >= state.max_turns: return "turn_limit_breached" if state.confidence < state.confidence_floor: return "confidence_floor_breached" return None The composition matters. Any breach terminates the loop — three independent failsafes, not three redundant ones. A budget overrun without a turn overrun still terminates. A turn overrun with high confidence still terminates. The AND-not-OR composition is the discipline that makes the bound defensible. Teams that compose termination as “must breach all three” produce systems where one weak failsafe defeats the other two. Feedback as the Closing Loop HIL approvals are not just gates. They are a training signal. Every joint decision — approved, edited, rejected — is data the system can use to retune itself. The Evolution Layer from Part 1 is the architectural seam where this loop closes. A snippet from the repo’s evolution module showing the feedback ingestion path: Python # From context_evolution.py — feedback ingestion (abbreviated) def collect_feedback(self, exception_id, process_step, feedback_type, feedback_value, metadata): feedback = { "exception_id": exception_id, "process_step": process_step, "feedback_type": feedback_type, # approval | rejection | edit "feedback_value": feedback_value, # signal magnitude in [0, 1] "timestamp": datetime.utcnow().isoformat(), "metadata": metadata, # role, joint type, edit diff } self.store.append(feedback) # Asynchronously: update fusion alpha, landmark selection, # role-adapter pruning, and termination thresholds. self.adapt_strategies(feedback) Three things to defend about this design. Approvals are first-class feedback, not just events. When a human approves a plan unchanged, that is a positive signal — the plan was on the right path; the agent’s strategy is working. When a human edits parameters before approving, that is a corrective signal — the strategy is approximately right, but the parameter selection needs tuning. When a human rejects an action, that is a strong negative signal — the strategy itself is wrong. The Evolution Layer reads these three as different update magnitudes on different strategy parameters. Feedback adjusts the pipeline, not just the model. The retuning targets are pipeline-layer parameters: the fusion alpha in Gather, the landmark selection in Compress, the role-adapter pruning in Enrich, the termination thresholds in Topology. Models do not get retrained inline; pipelines do. This is what makes the loop affordable to run continuously. Feedback runs asynchronously to the action path. The agent acts; the human approves; the action commits. The feedback ingestion runs after commitment, on a separate path, so retuning never blocks live execution. This is the same separation that lets financial trading systems retrain risk models without slowing the order-placement path. The closing loop runs continuously. Every HIL approval, every rejection, every edit becomes signal. The pipeline’s strategy parameters drift toward the choices humans make. Over time, the joints fire less often because the agent’s first-pass plans match what humans would have edited toward. This is what “the system learns” actually means in production agentic systems — not the model getting smarter, but the pipeline’s strategy parameters getting better calibrated to the workflow’s actual decision distribution. Multi-Agent HIL Topology When two or more agents collaborate to resolve a task, joints multiply. Inter-agent handoffs become their own joint type — sometimes deterministic, sometimes human-gated. The Cross-Process Memory Manager component from the Evolution Layer is the substrate that lets state pass cleanly across agent boundaries. Two patterns I have seen work in production. The first is one-human-many-agents: a single human approver acts as the joint for multiple specialized agents (a research agent, a planning agent, an execution agent). The human sees the handoff context as the agents pass work between themselves; they only intervene when a handoff itself is high-stakes. The second is agent-as-reviewer: a senior agent (typically with stricter validation prompts and access to ground-truth verification) reviews the junior agent’s output before it reaches a human. This is the agentic-AI equivalent of a code review — one agent gates another before a human gates either. Handoff Opacity — and How the Audit Trail Closes It The trap in multi-agent HIL topology is handoff opacity. When work passes between two agents and the human doesn’t see the handoff, the human cannot identify which agent’s reasoning produced the eventual error. The fix extends the audit event schema from Part 2 with three handoff-specific fields: JSON { "event_id": "evt_2026_05_12_h3a9c", "stage": "orchestration.agent_handoff", "actor": { "kind": "agent", "role": "planning_agent" }, "handoff": { "from_agent": "research_agent", "to_agent": "planning_agent", "handoff_context_hash": "sha256:c3a8...e91f", "handoff_reason": "research_complete" }, "prev_event_id": "evt_2026_05_12_h3a9b" } The handoff_context_hash is the key field. It hashes the context payload passed between agents at the moment of handoff, which means a post-hoc reviewer can verify what the receiving agent actually saw — not what the system claims it saw. Multi-agent systems without this discipline produce post-mortems where engineers spend hours reconstructing which agent had what context at what time; with the hash, the question is decidable in seconds. Worked Example — Three-Agent Supply Chain Exception In MoJoCo’s supply chain handling, a single exception traverses three specialized agents. The research agent gathers customer impact data, SLA terms, and alternative routing options; it has read-only tool scope. It hands off to the planning agent, which drafts two-to-three resolution options ranked by cost-and-customer-impact; it has read-only tool scope plus access to cost-modeling tools. Planning hands off to the execution agent, which has the operationally consequential tools — including reroute_container and notify_customer. The handoffs are joints. Research → planning is deterministic (driven by research_complete signal). Planning → execution is human-gated — the Supply Chain Manager reviews the planning agent’s recommendation in the context the planning agent compiled, approves the selected option, and only then does the execution agent receive its task scope. This is also the seam where the reversibility matrix from Part 2 actually fires — the Supply Chain Manager’s approval at this joint is the dual-approve gate that the matrix requires for irreversible action. Implicit Human Availability — The Scheduling Discipline The second multi-agent trap is implicit human availability. Multi-agent systems often assume a human is always reachable for the rare joints that require approval; in regulated industries with global workflows, this is wrong by construction. The HIL topology has to account for explicit human shift coverage (which roles are on call at which hours, in which regions), escalation paths when the first-line approver is unavailable, and the default-on-timeout behavior at each joint. The discipline that works: every HIL joint registers a coverage policy alongside its trigger — primary approver, secondary approver, tertiary approver, and a default action if all three are unreachable within the joint’s timeout window. Follow-the-sun coverage rotates the primary approver role through regional teams as the business day moves. None of this is glamorous; all of it is the difference between a multi-agent system that holds up at 03:00 on a Saturday and one that doesn’t. The topology design has to encode the scheduling from day one — adding it after the fact means retrofitting every joint’s default behavior, which is the kind of change that almost never gets fully completed. Closer — The Moat Is Architecture This is the closer of the series, not just of Part 3. The full argument is now defended. Production agentic systems are won on context engineering, guardrails, and human-in-the-loop topology — not on model choice. Across the three parts, the engineering specifics that defend that claim: The pipeline is five runtime stages (Gather, Enrich, Verify, Compress, Inject) composed with four architectural layers (Acquisition, Refinement, Distribution, Evolution) plus Orchestration. The pipeline produces role-calibrated, token-efficient context. It prevents context-window collapse. The guardrails are five-field tool contracts (name, schema, scope, risk class, reversibility) enforced by the Function Identifier at injection time, an audit event schema emitted as a first-class artifact, and a reversibility-by-risk matrix that determines the HIL gate depth for every action. The guardrails prevent tool-authorization sprawl and audit-trail opacity. The topology is five canonical HIL joint types (planning approval, parameter approval, action approval, post-hoc review, escalation), three independent termination conditions (cost ceiling, turn limit, confidence floor), and a closing feedback loop where approvals retune the pipeline’s strategy parameters. The topology prevents agent-loop divergence. Models are commoditizing. The frontier-model gap on agentic benchmarks is shrinking quarter over quarter. The moat for enterprise agentic systems is not which model you call — it is the architectural disciplines at the context layer, the tool surface, and the HIL topology. These decisions determine the production outcome quality almost entirely. The model decision determines very little. This is the pattern that underpins MoJoCo, the agentic modernization platform I have been designing hands-on for eighteen months. The deterministic reverse-engineering tools (ARC, MAM, CAST) provide the action surface. The pipeline produces reasoning-grade context above them. The guardrails filter and audit every action. The topology decides when the agent acts and when a human takes over. Four disciplines composed into one architecture. The same pattern recurs in the Digital Fitness Index (agentic tech-debt scoring) and AskProcurement (AI-native procurement intelligence). Different surface; same substrate. If you are designing agentic systems in production right now, the question is not which model you should use. The question is which of these architectural disciplines you have not yet engineered. Most teams have not engineered any of the three. That is the opportunity. That is the moat. References Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., & Lowe, R. (2022). “Training Language Models to Follow Instructions with Human Feedback.” OpenAI. NeurIPS 2022. The InstructGPT paper; anchors the feedback-as-training-signal claim and the discipline of treating approvals/rejections/edits as different magnitudes of corrective signal. arXiv:2203.02155 Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., … Kaplan, J. (2022). “Constitutional AI: Harmlessness from AI Feedback.” Anthropic. Closest published source for the agent-as-reviewer pattern referenced in the multi-agent HIL section; provides the discipline for one agent gating another before human review. arXiv:2212.08073 Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., Liu, J., Awadallah, A. H., White, R. W., Burger, D., & Wang, C. (2023). “AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework.” Microsoft. Foundation for the multi-agent HIL extension; defines the handoff patterns that the topology section builds the human-gating layer on top of. arXiv:2308.08155 Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). “On Calibration of Modern Neural Networks.” ICML 2017. Foundation for the confidence-floor termination condition; explains why a confidence score of 0.7 from a well-calibrated agent means something different than 0.7 from a poorly calibrated one. arXiv:1706.04599 This concludes the three-part field manual. Part 1 took the pipeline. Part 2 took the guardrails. Part 3 took the topology. Production agentic systems are won here. A consolidated Part 4 — Further Reading & Research Lineage — collects the references across all three parts plus a cross-cutting reference and a note on what is deliberately not cited. Ram Ravishankar is an IBM Distinguished Engineer serving as Chief Engineer. Four patents, member of the IBM Academy of Technology, $1B+ cumulative business impact across 200+ engagements on six continents. He has spent the last eighteen months hands-on designing agentic systems for enterprise modernization in regulated industries — including MoJoCo (an agentic modernization platform wrapping deterministic reverse-engineering tools — ARC, MAM, CAST), the Digital Fitness Index for agentic tech-debt scoring, and AskProcurement for AI-native procurement intelligence.

By Ram Ravishankar

Deploying a Spring Boot Microservice on AWS Fargate: Lessons From the Outage That Forced Me to Get It Right

My first attempt to deploy a Spring Boot microservice on AWS Fargate didn’t fail loudly. It failed quietly — in a loop. ECS kept launching tasks, the Application Load Balancer kept marking them unhealthy, and the service never stabilized. The logs looked fine, the container looked fine, but the ALB replaced every task within seconds. The root cause was painfully simple: Spring Boot needed 45 seconds to start, and my ALB health‑check timeout was 5 seconds. The tasks never had a chance. That night changed how I build and deploy microservices. It forced me to rethink startup behavior, JVM sizing, networking, task definitions, and the entire CI/CD pipeline. This article is the guide I wish I had before that incident — a practitioner’s walkthrough of deploying a production‑ready Spring Boot service on AWS Fargate, with real artifacts and the details that matter when things go wrong. The Architecture That Finally Worked Once the health‑check issue was fixed, the architecture settled into a predictable, cloud‑native flow: Developers push code to GitHubGitHub Actions builds the JARDocker image is built and pushed to Amazon ECRECS service runs AWS Fargate tasksTraffic enters through an Application Load BalancerTasks run in private subnetsConfiguration comes from Parameter Store and Secrets ManagerLogs and metrics flow to CloudWatch It’s the standard modern microservice pipeline — but the difference between “standard” and “production‑ready” is in the details. The Spring Boot Service The microservice itself was simple — a REST API with a few endpoints. The real complexity wasn’t the controller logic; it was everything around it: startup time, health checks, configuration management, and container behavior under load. A Dockerfile Built for Production My first Dockerfile looked like the one many tutorials start with: a single‑stage build running as root with no JVM tuning. It worked locally but failed under real load. Fargate tasks with default JVM heap sizing inside a 2GB container are a classic OOM story. Here’s the hardened version that finally stabilized deployments: Dockerfile FROM eclipse-temurin:21-jre # Create non-root user RUN useradd -u 1001 springuser WORKDIR /app # Layer extraction for faster builds COPY target/*.jar app.jar # JVM tuning for Fargate ENV JAVA_OPTS="\ -XX:MaxRAMPercentage=75 \ -XX:+UseContainerSupport \ -XX:+ExitOnOutOfMemoryError \ " USER springuser ENTRYPOINT ["sh", "-c", "java $JAVA_OPTS -jar app.jar"] This eliminated the OOMKilled events I saw on 2GB tasks and made startup time predictable. Pushing to Amazon ECR With Real Commands The first time I wrote down my ECR commands, they were placeholders. In production, they need to be exact: C aws ecr get-login-password --region us-east-1 \ | docker login --username AWS --password-stdin <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com docker build -t employee-service:1.0.3 . docker tag employee-service:1.0.3 \ <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/employee-service:1.0.3 docker push \ <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/employee-service:1.0.3 Immutable semantic version tags make rollbacks predictable and prevent “latest‑tag roulette.” The ECS Task Definition That Actually Runs in Production A real Fargate deployment lives or dies by its task definition. Here’s the JSON I use today — including secrets pulled from Parameter Store and Secrets Manager: JSON { "family": "employee-service", "networkMode": "awsvpc", "requiresCompatibilities": ["FARGATE"], "cpu": "512", "memory": "1024", "executionRoleArn": "arn:aws:iam::<ACCOUNT_ID>:role/ecsTaskExecutionRole", "taskRoleArn": "arn:aws:iam::<ACCOUNT_ID>:role/employeeServiceRole", "containerDefinitions": [ { "name": "employee-service", "image": "<ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/employee-service:1.0.3", "portMappings": [ { "containerPort": 8080, "protocol": "tcp" } ], "secrets": [ { "name": "DB_PASSWORD", "valueFrom": "arn:aws:ssm:us-east-1:<ACCOUNT_ID>:parameter/db/password" }, { "name": "API_KEY", "valueFrom": "arn:aws:secretsmanager:us-east-1:<ACCOUNT_ID>:secret:thirdparty/api" } ], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/employee-service", "awslogs-region": "us-east-1", "awslogs-stream-prefix": "ecs" } } } ] } The ALB Health Check That Stopped the Outage My outage happened because the ALB was impatient. Here’s the configuration that finally stabilized deployments: settingvalue Path /actuator/health Interval 20 seconds Timeout 10 seconds Healthy threshold 3 Unhealthy threshold 3 Spring Boot startup time + ALB patience = stable deployments. Why Fargate Tasks Belong in Private Subnets Early on, I deployed tasks in public subnets because it felt simpler. It wasn’t. Public IPs meant the containers were directly reachable from the internet — port scans, bot traffic, and noisy logs. Moving tasks to private subnets solved several problems at once: Reduced Attack Surface No public IPs. No direct inbound traffic. Only the ALB can reach the tasks. A Single Secure Entry Point The ALB handles TLS termination, redirects HTTP→HTTPS, performs health checks, and integrates with WAF. Clients never bypass it. Cleaner Security Groups ALB SG: inbound 443 from the internetTask SG: inbound only from ALB SG Nothing else touches the containers. Compliance Alignment PCI, SOC 2, HIPAA — all prefer minimizing public exposure. Controlled Outbound Access Tasks use a NAT Gateway for outbound calls (updates, third‑party APIs) without exposing themselves. Better Scalability ALB target groups automatically track tasks across AZs as ECS scales. The architecture becomes simple and predictable: Internet → ALB (public subnets) → Fargate tasks (private subnets) It’s quieter, safer, and easier to operate. The GitHub Actions Workflow That Deploys Automatically Here’s the pipeline that builds, tests, pushes, and deploys the service: YAML name: Deploy to Fargate on: push: branches: ["main"] jobs: build-deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Set up JDK uses: actions/setup-java@v4 with: java-version: "21" - name: Build JAR run: mvn -B clean package - name: Login to ECR uses: aws-actions/amazon-ecr-login@v2 - name: Build and Push Image run: | docker build -t employee-service:1.0.3 . docker tag employee-service:1.0.3 ${{ env.ECR_REGISTRY }/employee-service:1.0.3 docker push ${{ env.ECR_REGISTRY }/employee-service:1.0.3 - name: Deploy ECS Service uses: aws-actions/amazon-ecs-deploy-task-definition@v2 with: task-definition: ecs-task.json service: employee-service cluster: prod-cluster Auto Scaling With Real Target Tracking JSON Target tracking is the simplest and most reliable scaling strategy for Fargate: JSON { "TargetValue": 50.0, "PredefinedMetricSpecification": { "PredefinedMetricType": "ECSServiceAverageCPUUtilization" }, "ScaleOutCooldown": 30, "ScaleInCooldown": 60 } I use 50% as the target because it balances cost and responsiveness. What I Learned Every failure taught me something: ALB timeouts taught me to respect startup timeOOMKilled tasks taught me to tune the JVMPublic subnets taught me to isolate workloadsManual deployments taught me to automate everything AWS Fargate really does deliver on its promise — no servers to manage, automatic scaling, and clean integration with ECS — but only after you learn the hard parts. If you’re deploying Spring Boot on Fargate, I hope you learn those lessons from this article instead of from your own outage.

By Vishal Rameshchandra Shah

Retrieval Augmented Generation With Spring AI 2.0, Claude, and PGvector

Language models become much more useful when they can answer questions about information they were never trained on, including your internal documentation, product manuals, policies, and other proprietary data. Prompting alone cannot solve this, because the model simply does not have access to that knowledge. Retrieval-Augmented Generation, or RAG, is the most common way to bridge that gap. Spring AI comes with solid support for building RAG systems. It has been almost three years since Spring AI showed up, and in that time it has grown from an experimental member of the Spring portfolio into a mature layer over chat models, embedding models, vector stores, and the plumbing that sits between them, which happen to be exactly the pieces a RAG system needs. In this article, we build a small but complete RAG service with Spring AI 2.0. The application reads a set of documents into a PostgreSQL vector store, retrieves the fragments that are relevant to a user question, and lets Anthropic's Claude put together the answer based on those fragments. Everything runs from a standard Spring Boot project, and every step can be reproduced on macOS, Windows, or Linux. The full project is available on GitHub. If you just want to see the finished result, or you would rather skip the step-by-step build below, you can clone the repository and run it as it is. Everyone else can follow along and generate this project from scratch. The prompts themselves are kept deliberately simple. You can tune retrieval and prompts forever; here we care about the architecture and how the pieces fit together in Spring. Approach RAG is not really a single feature. It is more of a small pipeline, and the code below makes a lot more sense once its parts have names. Embedding: a vector of numbers that captures the meaning of a piece of text. Texts that mean similar things end up with vectors that are close to each other.Embedding model: the model that computes these embeddings. It is a different model from the chat model, and it has a different job.Vector store: a database that keeps text fragments together with their embeddings and can answer the question, "which stored fragments are closest in meaning to this query?"Chunking: documents are too large to embed and retrieve as a whole, so we split them into smaller fragments (chunks) before storing them.Similarity search: we embed the user question and fetch the top-k closest chunks from the store.Augmentation: we append the retrieved chunks to the user question before sending it to the chat model, so the model answers from the context we provided instead of from its training data. One thing here is worth calling out, because it shapes the whole setup of the project: the LLM model used in chat and the embedding model are two separate choices. As of today, Anthropic offers LLM models but no embedding API, so a Claude-based RAG system always has to pair Claude with an embedding model from somewhere else. Rather than bringing in a second cloud provider and a second API key, this project computes embeddings locally (inside the JVM), using Spring AI's ONNX transformers module and the well-known all-MiniLM-L6-v2 sentence transformer. It is free and fast enough for this, and it keeps everything on one API key. In our scenario, the service is an internal assistant for a fictional company called Nimbusfield Systems, and it answers employee questions based on the company handbook. The company and the handbook are fictional on purpose. Claude cannot possibly know about it, which makes it easy to verify that the answers really come from our documents and not from the model's own memory. We build this in three steps: Expose a /ask endpoint backed by Claude, with no retrieval, and show that the model cannot answer handbook questions.Ingest the handbook into PGvector at application startup: read, chunk, embed, and store.Attach Spring AI's QuestionAnswerAdvisor to the same ChatClient and ask again. Prerequisites Java 21Maven 3.9.x (the Maven wrapper included in generated projects works too)Spring Boot 4.0.xSpring AI 2.0.0Docker Desktop (macOS/Windows) or Docker Engine (Linux), used only to run PostgreSQL. A project skeleton can be generated at start.spring.io by selecting Web, Anthropic Claude, PGvector Vector Store, and Docker Compose Support. The remaining Spring AI modules are added manually below. The Claude API Key Sign in (or sign up) at the Anthropic Console, open Settings, then API Keys, and create a new key. New accounts may need a small prepaid credit before the API accepts requests, but the runs in this article cost only a few cents. The key is shown only once, so store it right away as an environment variable. If you would rather not spend anything at all, you can still follow along and read through the steps without running the calls yourself. macOS/Linux: export ANTHROPIC_API_KEY=sk-ant-... Windows (PowerShell, persists across sessions after reopening the terminal): setx ANTHROPIC_API_KEY "sk-ant-..." Solution Dependencies With the Spring AI BOM in place, there is no need to repeat versions on the individual artifacts. Initializr expresses the BOM's own version as a property rather than a hardcoded literal, so there is a single place to bump it later: XML <properties> <java.version>21</java.version> <spring-ai.version>2.0.0</spring-ai.version> </properties> <dependencyManagement> <dependencies> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-bom</artifactId> <version>${spring-ai.version}</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> A common source of confusion is that start.spring.io has no dependency literally named "Spring AI." Each provider- or store-specific starter (Anthropic Claude, PGvector Vector Database, and so on) is itself a Spring AI module, and picking one transitively pulls in the framework's core classes. (like ChatClient, VectorStore, etc.) Selecting any one of them is also what makes Initializr add the spring-ai-bom as shown above to the generated pom.xml for you. The BOM itself is never a separate item you tick on the Initializr dependency screen. The application needs six Spring AI modules on top of the web starter, each one with a single responsibility. XML <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-webmvc</artifactId> </dependency>  <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-starter-model-anthropic</artifactId> </dependency>  <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-starter-model-transformers</artifactId> </dependency>  <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-starter-vector-store-pgvector</artifactId> </dependency>  <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-vector-store-advisor</artifactId> </dependency>  <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-tika-document-reader</artifactId> </dependency>  <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-docker-compose</artifactId> <scope>runtime</scope> <optional>true</optional> </dependency>  <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-spring-boot-docker-compose</artifactId> <scope>runtime</scope> <optional>true</optional> </dependency> </dependencies> Two models are referenced from the code here. One is the chat model, Claude, which is served from the Anthropic API. The other is the embedding model, which runs locally, right inside the application. We will look at that local embedding model in the next section. The Embedding Model By default, the transformers starter fetches tokenizer.json and model.onnx from Spring AI's own GitHub repository the first time the application starts and then caches them locally. In practice, this default setup is a bit fragile. raw.githubusercontent.com may rate-limit unauthenticated requests, and model.onnx (which is roughly 90 MB) is stored via Git LFS, whose bandwidth quota can run out independently of the ordinary rate limit. When that happens, the endpoint serves the small LFS pointer stub instead of the binary, with a normal-looking HTTP 200, and the failure only shows up later as a cryptic ONNX Runtime protobuf-parsing error rather than a clear download error. The fix is to bundle both files with the application instead of fetching them at startup. So we download them once: Shell mkdir -p src/main/resources/onnx/all-MiniLM-L6-v2 curl -fL -o src/main/resources/onnx/all-MiniLM-L6-v2/tokenizer.json \ https://raw.githubusercontent.com/spring-projects/spring-ai/main/models/spring-ai-transformers/src/main/resources/onnx/all-MiniLM-L6-v2/tokenizer.json curl -fL --http1.1 -o src/main/resources/onnx/all-MiniLM-L6-v2/model.onnx \ https://media.githubusercontent.com/media/spring-projects/spring-ai/main/models/spring-ai-transformers/src/main/resources/onnx/all-MiniLM-L6-v2/model.onnx Then we point the embedding model at these local files in our application.properties, overriding the GitHub-backed defaults: Properties files spring.ai.embedding.transformer.onnx.model-uri=classpath:/onnx/all-MiniLM-L6-v2/model.onnx spring.ai.embedding.transformer.tokenizer.uri=classpath:/onnx/all-MiniLM-L6-v2/tokenizer.json With these two properties set, the application never touches the network for the embedding model, neither on the first run nor on any run after it. The Database The pgvector team publishes a PostgreSQL image with the extension already installed. A compose.yaml in the project root is all we need: YAML services: pgvector: image: "pgvector/pgvector:pg17" environment: - "POSTGRES_DB=nimbusfield" - "POSTGRES_USER=nimbusfield" - "POSTGRES_PASSWORD=nimbusfield" labels: - "org.springframework.boot.service-connection=postgres" ports: - "5432" The labels entry is important. Spring Boot's Docker Compose support auto-detects connection details by matching the image name against a list of well-known images. Plain Postgres is on that list, but pgvector is not, since it is a third-party image. The label tells Spring Boot to treat this container as if it were the official Postgres image, and that is what actually makes the automatic connection wiring work. If we omit it, the container still starts, but Spring Boot never creates a ConnectionDetails bean for it, so the run fails with a connection error rather than falling back gracefully. Because spring-boot-docker-compose is on the classpath, running the application starts the container automatically and injects the connection details. This works the same way on macOS and Windows, as long as Docker Desktop is running. Anyone who prefers to manage the container manually can run the same image with docker run -p 5432:5432 .. and set the datasource properties explicitly. Configuration The complete application.properties, now including the embedding model overrides shown earlier: Properties files spring.ai.anthropic.api-key=${ANTHROPIC_API_KEY} spring.ai.anthropic.chat.model=claude-sonnet-5 spring.ai.anthropic.chat.max-tokens=1024 spring.ai.embedding.transformer.onnx.model-uri=classpath:/onnx/all-MiniLM-L6-v2/model.onnx spring.ai.embedding.transformer.tokenizer.uri=classpath:/onnx/all-MiniLM-L6-v2/tokenizer.json spring.ai.vectorstore.pgvector.initialize-schema=true spring.ai.vectorstore.pgvector.dimensions=384 spring.ai.vectorstore.pgvector.index-type=HNSW spring.ai.vectorstore.pgvector.distance-type=COSINE_DISTANCE logging.level.org.springframework.ai.chat.client.advisor=DEBUG Four details matter here. First, max-tokens is mandatory for the Anthropic API, which caps every response explicitly. Spring AI provides a default, but it is better stated than left implied. Second, the two spring.ai.embedding.transformer.* properties point the embedding model at the local files we bundled in the previous section, instead of Spring AI's own GitHub-backed defaults. See "The Embedding Model" above for why this matters. Third, initialize-schema=true enables the automatic creation of the vector-store table and the required extensions. (Since Spring AI 1.0, this no longer happens silently by default.) Fourth, dimensions=384 must match the embedding model. all-MiniLM-L6-v2 produces 384-dimensional vectors. If the embedding model changes later, the table has to be recreated, because the column type is vector(384). The Documents Two short Markdown files under src/main/resources/docs play the role of the company handbook. remote-work-policy.md Markdown # Nimbusfield Systems Remote Work Policy Employees may work remotely up to three days per week. Remote days must be registered in the portal by Thursday of the preceding week. Working from abroad is permitted for a maximum of 30 calendar days per year and requires prior approval from both the line manager and the People team. travel-expenses.md: Markdown # Nimbusfield Systems Travel and Expenses The daily meal allowance for business trips is 65 EUR in Europe and 80 USD elsewhere. Taxi rides are reimbursed only between airports, hotels, and client sites. Flights longer than six hours may be booked in premium economy. All expense reports are due within 15 working days after the trip via the portal. Thanks to the Tika reader used below, dropping PDFs or Word documents into the same folder works without any code changes. Step 1: Chat Without Retrieval We start with a service that wraps a ChatClient, built once from the auto-configured builder: Java @Service public class AssistantService { private final ChatClient chatClient; public AssistantService(ChatClient.Builder builder) { this.chatClient = builder .defaultSystem(""" You are the internal assistant of Nimbusfield Systems. Answer employee questions precisely and briefly. If you do not know the answer, say so. """) .build(); } public String ask(String question) { return chatClient.prompt() .user(question) .call() .content(); } } And a controller associated with it: Java @RestController public class AssistantController { private final AssistantService assistantService; public AssistantController(AssistantService assistantService) { this.assistantService = assistantService; } @GetMapping("/ask") public ResponseEntity<String> ask(@RequestParam("question") String question) { return ResponseEntity.ok(assistantService.ask(question)); } } Start the application (./mvnw spring-boot:run on macOS/Linux, mvnw.cmd spring-boot:run on Windows) and ask it a handbook question: http://localhost:8080/ask?question=What is the daily meal allowance for business trips in Europe? The response, as we might expect, is: I don't have that information in my available knowledge base. Nimbusfield Systems' specific travel and expense policy—including per diem rates for European business trips—isn't something I can confirm accurately. To get the correct figure, please check: The company's Travel & Expense Policy document (likely on the intranet/HR portal)Your Finance or HR department directlyYour manager, if travel budgets are pre-approved per trip Would you like help with anything else I can assist with more reliably? This gives us a baseline. The model behaves correctly given what it knows, which is nothing at all about this company. Step 2: The Ingestion Pipeline Ingestion follows Spring AI's extract, transform, load structure: a DocumentReader extracts the text, a TextSplitter chunks it, and the VectorStore embeds and stores the chunks. The embedding call happens implicitly inside vectorStore.add() call. The auto-configured TransformersEmbeddingModel is wired into the PgVectorStore and each chunk is embedded into the table. Java @Component public class HandbookIngestion implements ApplicationRunner { private static final Logger log = LoggerFactory.getLogger(HandbookIngestion.class); private final VectorStore vectorStore; private final JdbcTemplate jdbcTemplate; private final Resource[] handbook; public HandbookIngestion(VectorStore vectorStore, JdbcTemplate jdbcTemplate, @Value("classpath:docs/*.md") Resource[] handbook) { this.vectorStore = vectorStore; this.jdbcTemplate = jdbcTemplate; this.handbook = handbook; } @Override public void run(ApplicationArguments args) { Integer count = jdbcTemplate.queryForObject( "select count(*) from vector_store", Integer.class); if (count != null && count > 0) { log.info("Vector store already contains {} chunks, skipping ingestion", count); return; } TokenTextSplitter splitter = TokenTextSplitter.builder() .withChunkSize(300) .build(); for (Resource resource : handbook) { List<Document> documents = new TikaDocumentReader(resource).get(); documents.forEach(doc -> doc.getMetadata().put("source", resource.getFilename())); List<Document> chunks = splitter.apply(documents); vectorStore.add(chunks); log.info("Ingested {} chunks from {}", chunks.size(), resource.getFilename()); } } } The count check makes ingestion idempotent, so restarting the application does not duplicate every chunk. And the source metadata attached to each chunk enables filtered searches later, for instance restricting retrieval to a single document. That same idempotency check has a practical downside worth pointing out. Once the vector store has data, restarting the application will not pick up edits to the handbook files, since the count check short-circuits before the splitter ever runs. To force a clean re-ingestion, for instance after changing a handbook document, tear down the container together with its data volume, not just the container: docker compose down -v The chunk size of 300 tokens is generous for documents this small. The splitter's default of 800 is aimed at larger, real-world content. Chunking is the least exciting and yet the most important knob in a RAG system: chunks that are too large dilute the similarity signals, while chunks that are too small lose their context. It is worth experimenting here: try a few different chunk sizes and see how the system behaves. Just remember to run docker compose down -v between runs, so the vector store is rebuilt from scratch each time. Step 3: Attaching the Retrieval Advisor Now we come back to the plain AssistantService from Step 1 and upgrade it, rather than writing something new. The ChatClient wiring we built earlier stays and what changes is what gets attached to it. Spring AI models the cross-cutting concerns around a chat call as "advisors", which are conceptually close to interceptors. The QuestionAnswerAdvisor embeds the incoming user question, runs a similarity search against the vector store, and appends the retrieved chunks to the prompt before it reaches Claude. Enabling RAG is therefore a change to how the ChatClient is constructed, not to how the request is handled: Java public AssistantService(ChatClient.Builder builder, VectorStore vectorStore) { this.chatClient = builder .defaultSystem(""" You are the internal assistant of Nimbusfield Systems. Answer employee questions precisely and briefly. If you do not know the answer, say so. """) .defaultAdvisors( QuestionAnswerAdvisor.builder(vectorStore) .searchRequest(SearchRequest.builder() .topK(4) .similarityThreshold(0.5) .build()) .build(), new SimpleLoggerAdvisor()) .build(); } topK(4) retrieves at most four chunks per question, and similarityThreshold(0.5) discards weak matches, so an entirely unrelated question augments the prompt with nothing rather than with noise. The SimpleLoggerAdvisor, combined with the DEBUG logging property we set earlier, prints the fully augmented prompt. This is the single most useful debugging tool while tuning retrieval, because it shows exactly what Claude was given. We restart and repeat the same request: http://localhost:8080/ask?question=What is the daily meal allowance for business trips in Europe? The daily meal allowance for business trips in Europe is 65 EUR. Same model, same question, and this time a precise answer grounded in the retrieved handbook chunk instead of a generic deflection. The debug log confirms what is going on behind the scenes: the user question arrives at Claude wrapped in a prompt that contains the retrieved handbook fragments as context. Going Further The default behavior of QuestionAnswerAdvisor is usable, but there are two refinements worth implementing if you want to take this pattern further. The first one concerns grounding. Even with retrieved context, the model may fall back on its general knowledge when the context does not actually contain the answer. The advisor accepts a custom PromptTemplate that controls how the question and the context are merged, and this is the place to enforce stricter behavior. The template must contain the query and question_answer_context placeholders: Java PromptTemplate strictTemplate = PromptTemplate.builder() .template(""" {query} Answer strictly based on the context below. If the context does not contain the answer, reply exactly: "This is not covered by the handbook." --------------------- {question_answer_context} --------------------- """) .build(); QuestionAnswerAdvisor advisor = QuestionAnswerAdvisor.builder(vectorStore) .promptTemplate(strictTemplate) .build(); Asking about, say, the parental leave policy (which is absent from our two files) now produces the fixed refusal instead of an invention. If people are going to rely on it, you want this on. The second refinement could be structured output, and it composes cleanly with retrieval. Declaring a record and calling .entity() instead of .content() gives back a typed object, with Spring AI instructing the model to respond in the matching JSON schema: Java public record HandbookAnswer(String answer, String sourceHint, boolean coveredByHandbook) { } public HandbookAnswer askStructured(String question) { return chatClient.prompt() .user(question) .call() .entity(HandbookAnswer.class); } A last note on the embedding choice. A local MiniLM model is not the strongest embedding model available, and for a large multilingual corpus a hosted embedding API or a bigger ONNX model would retrieve better. This choice is easy to reverse: EmbeddingModel is an interface, swapping the implementation is a matter of a dependency and a property, and the only hard constraint is the one mentioned earlier: the vector dimensions in PGvector have to match whatever the embedding model produces. Conclusion In this article, we built the RAG flow step by step. We started with a plain chat endpoint that could not answer anything about the Nimbusfield handbook, because Claude had never seen it. We then ingested that handbook into PGvector, embedding each chunk locally, and attached Spring AI's QuestionAnswerAdvisor to the same client. That single change was enough to turn a generic model into a service that answers from your own documents. After that, we talked about how we can tighten the grounding, so the model says it does not know when the context has no answer, and pulled the response straight into a typed Java record. If you want to take it further, clone the project, point it at your own documents, apply further the techniques we discussed in the Going Further section, play with different chunk sizes, retrieval settings, and prompts to see how the answers change. The Spring AI documentation goes deeper into advisors, vector stores, and retrieval configuration. The complete, runnable project is available on GitHub.

By Murat Balkan

CORE

What Nobody Tells You About Running AI Models in Docker

It was 2:14 in the morning when the pager went off. Our recommendation model's inference service had started returning 503s under a traffic spike that, frankly, wasn't even that big. Maybe three times the normal load. By the time I'd opened my laptop, the container had been OOM-killed four times in ten minutes, and Kubernetes was cheerfully restarting it into the same wall every ninety seconds. The image was 14GB. Cold start took eighty seconds. Nobody on the team had looked closely at any of that until it started costing us actual money in lost requests. That night is the reason I now have strong opinions about Docker and AI infrastructure. Why This Keeps Happening Containers became the default way to ship machine learning models because they solve a real problem: a model trained with CUDA 11.8, PyTorch 2.1, and a very specific glibc version doesn't reliably run on a colleague's machine, let alone a fleet of GPU nodes spread across three cloud regions. “It works on my machine” isn't a joke in ML infra; it's a recurring incident report. Docker gives you a way to freeze that dependency tree and ship it as one artifact, and that part genuinely works. What doesn't get discussed enough is the many issues that Docker does not resolve, along with the subtle ways teams exacerbate problems by forcing AI workloads into a packaging model that was originally designed for stateless web services. What We Tried First (and Why It Blew Up) Our first version of the inference image was, in hindsight, a small crime. We started from nvidia/cuda:12.2.0-devel-ubuntu22.04 because someone had seen it in a tutorial, installed the full CUDA toolkit, pip-installed every dependency without pinning, and copied in not just the model weights but three checkpoint versions “just in case.” Fourteen gigabytes. Every deployment pulled the entire image onto a fresh node, and during autoscale events, we experienced over a minute of waiting just for the image to be pulled before the container started loading the model into GPU memory. The first fix everyone reaches for is a multi-stage build, and yes, it helps — but it's not the silver bullet people pitch it as. Splitting a devel build stage from a runtime stage cut us from 14GB to roughly 6GB: Dockerfile FROM nvidia/cuda:12.2.0-devel-ubuntu22.04 AS builder WORKDIR /build COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt --target=/deps FROM nvidia/cuda:12.2.0-runtime-ubuntu22.04 COPY --from=builder /deps /usr/local/lib/python3.10/site-packages COPY model/ /app/model/ COPY serve.py /app/ WORKDIR /app CMD ["python3", "serve.py"] That's better, but the real lie in that Dockerfile is the COPY model/ line. We baked multi-gigabyte weights into an image layer, causing a full re-push of the weights with every code change, even minor ones, since Docker re-hashes the entire build context. We moved weights to an external volume pulled from object storage at container start, with a checksum cache to skip redundant downloads. That alone cut most of our deployment time. The trade-off is a slightly more complex startup script and a new dependency on storage being reachable at boot, which is its failure mode. There's no free lunch here; you're just choosing which problem is going to page you at 2 am. The Detour: Skipping Containers Entirely We also tried, briefly, running everything on bare metal with conda environments and skipping containers altogether, mostly because one engineer was convinced Docker added overhead with no real benefit for GPU workloads. It's not a crazy position; Docker's GPU story is genuinely leaky. <nvidia-container-toolkit exposes the host driver directly to the container, leading to potential mismatches that application-level packaging cannot resolve, so there is no real isolation>. However, we reverted that experiment within a sprint because as soon as more than two people interact with the training pipeline, environment drift reoccurs immediately. “Works on my conda env” is just “works on my machine” wearing a hat. What Actually Held Up in Production The architecture that worked was less about clever Docker tricks and more about admitting that an inference container and a training container have almost nothing in common and shouldn't share a Dockerfile, a registry strategy, or a deployment pattern. For serving, we kept images lean, stateless weights externalized, and a health assessment that actually runs a tiny dummy inference instead of just pinging an HTTP port. A server can report itself as “up” while a model failed to load correctly, and that gap has burned us more than once. Locally, a docker-compose GPU reservation block mirrored how production scheduled GPUs, so dev environments stopped lying about resource contention: YAML services: inference: image: registry.internal/rec-model:latest deploy: resources: reservations: devices: - capabilities: [gpu] count: 1 healthcheck: test: ["CMD", "python3", "healthcheck.py"] interval: 15s timeout: 5s retries: 3 For training, we accepted slower builds because training jobs run for hours, and a ninety-second image pull is negligible compared to that, even though the images are larger. Spending engineering time shrinking training images was effort we'd burned for no real payoff. A contrarian point I'll happily defend: not every container needs to be small, only the ones sitting in your hot path. The other decision that mattered more than any Dockerfile tweak was plain layer-caching discipline, putting "pip install before COPY." It sounds obvious when written down, but I've reviewed more than one ML team's Dockerfile that copies the whole repo first “for simplicity” and invalidates every cached layer on a README change. Key Takeaways Externalize model weights from the image; baking them in wrecks your caching and your deploy speed.Multi-stage builds help, but they don't resolve a runtime image that's still hauling around a full CUDA devel toolkit.GPU isolation via containers is partial; complete driver and toolkit mismatches between host and container remain entirely your problem.Training and serving images deserve different optimization priorities; don't apply the same size obsession to both.A health check that only confirms the process is running, without verifying that the model has loaded correctly, will mislead you at the most critical moments. Closing Thought Docker didn't fail us that night; rather, it was our incorrect assumptions about its free services that let us down. It's easy to treat containers as a solved problem because the tooling is so mature for web services and then be surprised when AI workloads expose every shortcut you took. I still think GPU container isolation lacks a clean solution, and I’m curious if upcoming tools will address it or if we'll continue improving our health-check scripts. What's the worst 2 am lesson your infrastructure has taught you?

By Pruthvi Raj Seknametla

Build Your Own Local AI QA Engineer With Docker, Ollama, LibreChat, and Playwright MCP

Artificial intelligence is rapidly transforming software testing by enabling QA engineers to generate test cases and test plans, automate browser interactions, analyze and debug failures, and execute complex testing workflows using simple natural-language prompts. While cloud-based AI assistants offer impressive capabilities, they often require subscriptions and sharing potentially sensitive application data with third-party services. Running an AI-powered testing assistant locally addresses these concerns by providing better privacy, lower operating costs, and complete control over the testing environment. In this tutorial, we’ll learn how to build our own local AI QA engineer using Docker, Ollama, Qwen3:8b, LibreChat, and Playwright MCP. It will allow us to perform browser automation and interact with web applications using natural language, all without relying on cloud-based AI services. Understanding the Architecture Every interaction begins with the user. For example, a user enters a prompt in LibreChat, such as “Open the Playwright website and click the ‘Get Started’ button.” LibreChat serves as the conversational interface through which users interact with the AI assistant. Rather than processing the request itself, it forwards the prompt to a locally hosted large language model, Qwen3:8b, running via Ollama. After receiving the prompt, Qwen3:8b interprets the user’s intent and generates a step-by-step execution plan. Instead of interacting with the browser directly, the model determines which tools are required and communicates those instructions using the Model Context Protocol (MCP). These MCP requests are handled by the Playwright MCP Server, which acts as the bridge between the language model and the browser. It translates the AI-generated instructions into executable Playwright commands. The Playwright MCP Server then launches a Chrome browser and performs the requested actions. Depending on the prompt, it can navigate to websites, click buttons, complete forms, extract text from web pages, capture screenshots, and execute a wide range of browser automation tasks. Once the browser completes the requested operations, the execution results are returned to Qwen3:8b. The language model analyzes the browser output and transforms the technical details into a clear, human-readable response. LibreChat then presents this response to the user. Instead of displaying raw Playwright logs, it provides a concise summary such as: “Navigation completed successfully. The Playwright website was opened, and the Get Started button was clicked successfully.” This architecture enables browser automation through natural language while ensuring that every component runs locally. As a result, we benefit from enhanced privacy, greater security, and complete control over the entire AI-powered automation workflow. Prerequisites Before getting started, ensure that the following software is installed on your machine: DockerNode.js 20 or higher versionGitOllama We’ll use Docker Desktop to run LibreChat, Node.js to install and run the Playwright MCP Server, Git to clone the required repositories, and Ollama to download and serve the local large language model. Having these tools installed beforehand will make the setup process smooth and straightforward. System Requirements Running a local AI-powered browser automation stack requires a reasonably capable machine. A system with 16 GB of RAM or more is recommended to run Docker containers and the language model efficiently. We’ll also need 20–25 GB of available disk space, preferably on an SSD, to accommodate Docker images and downloaded models. While a dedicated GPU can significantly improve model inference speed, it is entirely optional, and the setup works well on modern CPUs. For this tutorial, I’m using the following configuration: Operating system: macOS (M2 Pro)Memory: 16 GB RAM We can have the same setup on Windows and Linux, with only minor platform-specific differences in the installation steps. Setting Up the Environment for the Local AI QA Engineer Docker, Node.js, and Git are widely used development tools, and detailed installation guides for each are readily available online. Installing Ollama To install Ollama, either download the installer from the official website or use the installation command provided for your operating system. For macOS, it can also be installed using the following Homebrew command: Plain Text brew install ollama Once the installation is complete, it can be verified by running the following command in the terminal: Plain Text ollama --version Installing Qwen3:8b Qwen3:8b is chosen for this setup because it offers a strong balance of reasoning, code generation, and performance, making it ideal for Playwright TypeScript test generation, AI agents, MCP integration, and modern QA automation workflows while running efficiently on a local machine. However, other higher models can also be chosen if you know a better one. Another factor in choosing this model was the available system memory. Since my machine has 16 GB of RAM, some memory also needs to be reserved for other tools used in this setup, such as Docker, LibreChat, and Playwright. We need to start Ollama first by running the following command from the terminal. (It should be kept running in the background): Plain Text ollama serve Open a new terminal and run the following command to pull the Qwen3:8b model: Plain Text ollama pull qwen3:8b It should take some time to complete the pull, as the model is around 5.2GB. Once the download completes, we can check the model by running the command: Plain Text ollama list It should list the model downloaded. Next, we can quickly verify by running the model using the command: Plain Text ollama run qwen3:8b Once the model starts, it will prompt you to enter a query. To verify that everything is working correctly, try a simple prompt such as “What is 2 + 2?”. Observe how the model processes the request and generates its response. If the setup is successful, it should return the correct answer, 4, confirming that the model has been downloaded, installed, and is functioning properly. To stop the model, type “/bye” in the prompt, and it should exit. Qwen3:8b provides a good balance between performance and resource usage, making it a suitable choice for this hardware configuration. If more RAM is available, you can opt for larger LLMs that offer stronger reasoning and coding capabilities. Installing LibreChat With Docker LibreChat is an open-source AI platform that provides a unified and customizable interface for interacting with multiple AI models. It enables us to manage all our AI conversations from a single application while supporting features such as AI agents, Model Context Protocol (MCP) servers, custom tools, and integrations with both local and cloud-based LLMs. LibreChat acts as the front-end chat interface that communicates with the locally running Qwen3:8b model through Ollama. It allows us to execute AI-powered browser automation workflows entirely on our local machine. Follow the steps below to install LibreChat: Step 1: Clone the LibreChat GitHub Repository The repository can be cloned by running the following command: Plain Text git clone https://github.com/danny-avila/LibreChat After cloning the repository, navigate to the LibreChat folder, copy the .env.example file, and create a new .env file from it. Plain Text cd LibreChat cp .env.example .env Let's keep the .env file as it is, using the default values. Step 2: Connect Ollama to LibreChat Ollama can be connected to LibreChat by updating its configuration in the “librechat.yaml” file. The example file is already available in the cloned repo. Run the following command to copy librechat.example.yaml and create librechat.yaml. Plain Text cp librechat.example.yaml librechat.yaml Update the following configuration in the file to connect Ollama to LibreChat: YAML endpoints: custom: - name: "Ollama" apiKey: "ollama" baseURL: "http://host.docker.internal:11434/v1" models: default: - "qwen3:8b" fetch: true titleConvo: true titleModel: "current_model" summarize: false summaryModel: "current_model" modelDisplayLabel: "Ollama" Make sure that this configuration is added to the “custom” block, which falls under the “endpoints” block. This configuration adds Ollama as a custom AI endpoint in LibreChat. The baseURL tells LibreChat where to connect to the Ollama API, while the default model specifies that Qwen3:8b should be used by default. Since LibreChat is running inside a Docker container while Ollama is running directly on the host machine, we use http://host.docker.internal:11434/v1 instead of localhost. The special hostname host.docker.internal allows the Docker container to access services running on the host system, enabling LibreChat to connect to the locally running Qwen3:8b model through Ollama. Setting fetch: true allows LibreChat to automatically detect and display all models available in Ollama. The remaining options configure the user interface by generating conversation titles using the current model, disabling conversation summarization, and displaying the endpoint with the label Ollama in the LibreChat interface. Step 3: Mount the Configuration in the docker-compose-override.yml The docker-compose-override.yml can be copied and created in the same way as we did “librechat.example.yaml”. Plain Text cp docker-compose.override.yml.example docker-compose.override.yml The following block should be updated in the docker-compose.override.yml file. YAML services: api: volumes: - ./librechat.yaml:/app/librechat.yaml This file mounts the custom “librechat.yaml” configuration file into the LibreChat container. By mapping ./librechat.yaml to /app/librechat.yaml, Docker ensures that LibreChat uses the custom configuration each time the container starts. This approach allows us to modify settings, such as custom endpoints and AI models, without rebuilding the Docker image. Step 4: Start the LibreChat Application Using Docker Compose The LibreChat application can be started using the following command: Plain Text docker compose up -d It will take some time for the Docker images to download, and containers will start. Run the following command from the terminal to check the Container status: Plain Text docker ps -a This command displays the status of all Docker containers. If any container is unhealthy or encounters an issue, its status will be clearly indicated in the output. In case any container is unhealthy or encounters an issue, the following command can be run to check its logs: Plain Text docker logs <container name> Once all the containers are started successfully, open a new browser and navigate to http://localhost:3080 to start LibreChat. Since we are accessing LibreChat for the first time, we will be prompted to register and create a new user account. After completing the registration process, we can sign in and start using the application. Step 5: Selecting Ollama > Qwen3:8b Model By default, the gpt-5.5 model is selected. To select the Qwen3:8b model: Click on the gpt-5.5 modelSelect Ollama > Qwen3:8b Once the Qwen3:8b model is selected, we can verify if it is working by sending a simple prompt such as “What is 2+2?” Make sure the command “ollama serve” is already running in the terminal in the background, else the model Qwen3:8b won't work on LibreChat. Once we receive a successful response from the model, we can confirm that the Qwen3:8b model has been configured and integrated successfully with LibreChat. Install Playwright MCP Server Playwright MCP can be installed by running the following command in the terminal: Plain Text npx @playwright/mcp@latest \ --host 0.0.0.0 \ --allowed-hosts "*" \ --port 8931 \ By default, Playwright MCP listens only on localhost, which means applications running inside Docker (like LibreChat) cannot connect to it. Using --host 0.0.0.0 makes the server accessible from Docker containers, while --allowed-hosys "*" allows requests from host.docker.internal instead of restricting access to localhost. Once the Playwright MCP server is started, we can leave it running in the terminal. After the Playwright MCP server starts, it shows the following message at the bottom: “For legacy SSE transport support, you can use the /sse endpoint instead”. We will configure the Playwright MCP server using the SSE (Server-Sent Events) transport. Although Playwright MCP also supports the Streamable HTTP transport, LibreChat currently does not support connecting to it via the /mcp endpoint. Therefore, the SSE transport is used to establish a reliable connection between LibreChat and the Playwright MCP server. Configure Playwright MCP Server in LibreChat Playwright MCP server can be added to LibreChat by updating the following configuration in the “librechat.yaml” file. YAML mcpServers: playwright: type: sse url: http://host.docker.internal:8931/sse timeout: 120000 This configuration registers the Playwright MCP server with LibreChat. The type: sse setting specifies that the connection uses the Server-Sent Events (SSE) transport, while the url points to the Playwright MCP server running on the host machine. The hostname host.docker.internal allows the LibreChat Docker container to communicate with services running outside the container. The timeout: 120000 sets the request timeout to 120 seconds, giving the AI agent sufficient time to complete browser automation tasks before the connection expires. However, the timeout can be extended to 15–20 minutes or more, as there is no harm in doing that. YAML mcpSettings: allowedDomains: - 'host.docker.internal:8931' - 'localhost:8931' The mcpSettings configuration also needs to be added under the ‘actions’ block in the “librechat.yaml” file. The mcpSettings.allowedDomains section defines the list of trusted MCP server endpoints that LibreChat is allowed to connect to. By including both host.docker.internal:8931 and localhost:8931, LibreChat can establish a secure connection to the Playwright MCP server, whether it is accessed from within the Docker container (host.docker.internal) or directly from the host machine (localhost). Any MCP server not included in this list will be blocked, providing an additional layer of security. Restart the LibreChat app so it reads the newly configured Playwright MCP server: Plain Text docker compose restart That, or we can also shut down the already running LibreChat and start it again by using the commands below: 1. To shut down LibreChat: Plain Text docker compose down 2. To start it again: Plain Text docker compose up -d After restarting LibreChat, log in and navigate to the home page, and follow the steps below: Click on the MCP Settings menu on the left-hand menu panel.In the MCP Settings window, click on the “+” button to add MCP. Fill in the details for adding the Playwright MCP server; make sure to add the following settings: MCP server URL: http://host.docker.internal:8931/sseTransport: SSEAuthentication: NoneTick the “I trust this application” checkbox. Click on the “Create” button to save the details. Make sure that the Playwright MCP server is started and running on the terminal as discussed in the earlier section Click Connect for the newly created MCP server to establish the connection and begin using it. If everything is fine, a message should be displayed on successful connection. Understanding Model Context Protocol (MCP) By itself, a large language model (LLM) is limited to generating text. It can answer questions, explain concepts, write code, or summarize information, but it cannot directly interact with external systems or perform real-world actions. Model Context Protocol (MCP) changes this by enabling AI models to communicate with external tools and services through a standardized interface. Instead of simply providing suggestions, an AI model can execute tasks such as interacting with browsers, reading files, querying databases, or creating pull requests. Think of MCP as USB for AI A simple way to understand MCP is by comparing it to the USB standard. Before USB became the universal standard, every hardware manufacturer used its own proprietary connector. Printers, keyboards, cameras, and other peripherals all required different cables and custom software integrations. This made connecting devices unnecessarily complicated. USB solved this problem by introducing a common communication standard. Once both the computer and the device supported USB, they could communicate regardless of the device type. Whether you connected a keyboard, webcam, microphone, or external hard drive, the same protocol handled the communication. MCP brings the same level of standardization to AI systems. Without MCP, every AI application requires building and maintaining custom integrations for every external tool it wants to use. If we switch to a different AI application, those integrations often need to be recreated from scratch, resulting in duplicated effort and increased maintenance. A collection of awesome servers for the Model Context Protocol can be found at mcpservers.org. With MCP, tools expose a common interface that any MCP-compatible AI application can use. The AI model only needs to understand the MCP protocol, while the implementation details are handled by the individual MCP servers. Why MCP Matters for QA Automation For QA Automation Engineers, MCP unlocks the ability to automate complete testing workflows rather than isolated tasks. Consider the following request: “Read the Jira story, generate Playwright tests, execute them, analyze any failures, and create a GitHub pull request.” With MCP, the AI agent can coordinate multiple tools to complete the entire workflow. For example, it can: Read the user story from JiraAccess the application’s source code from GitHubGenerate Playwright TypeScript testsExecute the tests in a real browserCapture screenshots, logs, and execution reportsCommit the generated tests to GitHubUpdate the Jira ticket with the test results Each of these actions may be handled by a different MCP server, such as a Jira MCP server, GitHub MCP server, and Playwright MCP server. From the AI model’s perspective, however, every server is accessed using the same standardized MCP protocol. This standardization is what makes MCP so powerful. Rather than building custom integrations for every tool, AI systems communicate through a single, consistent protocol. As a result, MCP servers for Playwright, GitHub, databases, and many other services can be integrated and used in a uniform, scalable manner, significantly simplifying the development of AI-powered automation workflows. Creating an AI Agent With Playwright MCP Server in LibreChat for Automation Testing Let’s create a new AI Agent for browser automation testing with Playwright MCP using the steps below: Step 1: Click on the Agent Builder menu on the left-hand menu panel. Step 2: Enter the following mandatory details to create a new agent: Name: Provide a meaningful name to the agent.Category: Provide a category to the agent.Model: Select Qwen3:8bMCP Servers: Click on the Add MCP Server Tools button > Select the Playwright MCP Server that we created in the earlier section.Click on the Save button. Step 3: Update the model parameters. Clicking on the Model field, which has Qwen3:8b selected, should open the Model Parameters page. The following parameters can be set using this page: Provider: OllamaModel: Qwen3:8bTemperature: 0.2Top P: 0.85Frequency Penalty: 0.00Presence Penalty: 0.00Reasoning Effort: MediumReasoning Summary: Auto Click on the Save button to set the parameters. Step 4: Setting the instructions for the AI agent. The Following instructions can be pasted into the Instructions field in the Agent Builder window, or a “SKILL.MD” file can be created and uploaded using the Skills section of this agent. Markdown # Skills for the Local AI Agent for automation testing You are an expert QA Automation Engineer controlling a browser through Playwright MCP. Your goal is to execute browser actions safely and reliably. ## Tool Usage Rules - Do not run all MCP tools at the same time - Use only one Playwright MCP tool at a time. - Wait for the result of each tool before deciding the next action. - Never assume the page state. - Inspect the current page before interacting. - Do not start the next MCP tool unless the first one is complete ## Navigation Rules Treat the following actions as navigation-triggering actions: - Clicking Login, Submit, Continue, Save, Next, Checkout, etc. - Clicking any hyperlink. - Form submission. - Any action that changes the URL or reloads the page. - Wait until the page is fully loaded before making another tool call. After any navigation-triggering action: 1. Do not call any DOM inspection tool immediately. 2. Wait until the page has completely loaded. 3. Wait for the URL to stabilize if it changes. 5. Continue only after the new page is available. 6. Never inspect the previous page after navigation. ## Rules for locating web elements - Take a fresh snapshot to inspect the current page - Do not use XPath locator strategy - Use the same field name to locate elements, do not hallucinate and add prefix or suffix to field names - Use Semantic locator strategy: getByRole, getByText, getByLabel, getByPlaceHolder, getByAltText, getByTitle, getByTestId - Never use brittle CSS selectors such as .btn-primary, .container > div:nth-child(2), #content div span, or auto-generated classes. - Avoid nth() unless there is no unique locator. ## Interaction Rules - Verify and confirm that an element exists before interacting. ## Error Recovery If any Playwright tool fails: - Stop issuing new actions. - Inspect the current page. - Check Interaction Rules - Determine whether navigation has occurred. - Retry only if the page state confirms it is safe. - Do not repeat the same action more than once without confirming that the page state has not changed. Never repeat the same click more than once without checking the current page. ## Important If a click causes navigation, always assume the previous execution context has been destroyed. Do not read the DOM until the new page has fully loaded and a fresh snapshot has been obtained. Show a summary of test execution with the step count and pass or fail status - Run only the steps that are provided; do not hallucinate - Any deviation from these rules is not acceptable - Do not generate any additional steps - Always prioritize stability over speed. Providing instructions to an AI agent helps define its behavior, responsibilities, and the boundaries within which it should operate. These instructions act as persistent guidance, ensuring the agent follows consistent practices every time it performs a task instead of relying solely on the user’s prompt. For detailed setup instructions and troubleshooting guidance, refer to the GitHub repository. With these steps, the local AI agent is now ready to take commands. Running the AI Agent for Browser Automation To start using the AI Agent, click on New Chat.Click on the model name dropdown and select My Agents > The name of the agent that you created. Let’s use the following simple prompt and see how it works. Plain Text open http://playwright.dev verify the page title Once the prompt is submitted, we can observe the browser as the AI agent begins executing the task. The agent invokes the Playwright MCP server, which automatically launches a browser and performs the requested actions to navigate to the website and interact with the page. After the task is completed, Qwen3:8b analyzes the outcome and returns the results directly in the LibreChat conversation, demonstrating browser automation powered by Playwright MCP and Qwen3:8b. Let’s run another prompt for a login test scenario: Plain Text Navigate to https://parabank.parasoft.com/parabank/index.htm Locate "Username" field using "name=username" Enter "john" into the "Username" field. Locate "Password" field using "name=password" Enter "demo" into the "Password" field. Locator "Log In" button using "input[type="submit"] Click on the "Log In" button Verify that the "Accounts Overview" page is displayed This prompt also takes some time to understand the request before execution begins. It is important to note that the clearer and more specific the prompt, the more efficiently the AI agent can interpret and execute it. Well-structured prompts reduce ambiguity, minimize the chances of hallucinations, and typically result in faster execution and more accurate outcomes. As a best practice, break complex tasks into clear, sequential instructions whenever possible to improve the agent’s reliability and overall performance. As shown in the screenshot above, the AI agent invoked five tools from the Playwright MCP server to interact with the application and complete the requested workflow. It navigated to the website, located the username and password fields, entered the provided credentials, and submitted the login form. Finally, it verified that the login was successful by confirming that the “Accounts Overview” page was displayed. Since this setup runs entirely on a local machine, the AI agent takes approximately one minute to begin execution and around 4–5 minutes to complete a simple scenario. For more complex scenarios involving multiple steps, validations, or integrations, the AI agent is expected to take longer to analyze the request and complete the execution. But Execution time can be significantly reduced by running the setup on a machine with more powerful hardware, such as additional RAM, a faster CPU, or a dedicated GPU. Watch the step-by-step YouTube tutorial for Building your Local AI QA Engineer. Final Words Building a local AI QA engineer with Docker, Ollama, LibreChat, and Playwright MCP is an excellent way to explore the future of AI-powered software testing while keeping complete control over the data and infrastructure. By running everything locally, we eliminate recurring API costs, improve data privacy, and create a flexible environment for experimenting with AI-assisted browser automation using natural language. This setup is only the beginning of what’s possible. As we become more familiar with MCP and AI agents, the local QA assistant can be extended by integrating tools such as GitHub, Jira, databases, or custom MCP servers to automate even more of the testing workflow. Happy AI-powered testing!!

By Faisal Khatri

CORE

Building Reliable Data Pipelines for Enterprise Analytics Using PySpark

Most enterprise data problems are not caused by machine learning models or dashboard tools. They usually start much earlier in the pipeline. A reporting table misses records after a schema change. A nightly ingestion job finishes successfully but loads duplicate transactions. A downstream dashboard suddenly shows a 30% increase in revenue because one transformation joined datasets incorrectly. These issues are common in large-scale analytics environments where pipelines evolve faster than governance processes. PySpark is often adopted because it can process large distributed workloads efficiently, but scalability alone does not guarantee reliability. In practice, many pipelines become difficult to debug, validate, and maintain as the amount of data and its transformation complexity increase. This article focuses on practical techniques for building reliable analytics pipelines with PySpark, which include validation strategies, transformation design, partition management, schema handling, and operational monitoring. Reliability Problems in Enterprise Pipelines Pipeline failures rarely happen because Spark cannot process the data. They happen because the surrounding engineering practices are weak. One common issue is silent schema drift. A source system adds a new column or changes a data type, and downstream transformations continue running without immediately failing. The pipeline technically succeeds, but the analytics layer begins producing incorrect results. Another common problem appears during joins. Large transactional datasets often contain duplicate business keys, incomplete reference mappings, or delayed records. A transformation that works correctly during testing may suddenly inflate row counts in production. Operational reliability also becomes harder when transformations are tightly coupled. In many environments, a single failed stage forces the entire workflow to rerun, increasing compute costs and delaying reporting cycles. These problems are usually not visible during initial development. They appear after pipelines begin handling larger workloads, inconsistent source systems, and changing business logic. Structuring Pipelines for Maintainability One mistake teams make early is treating PySpark jobs as large monolithic scripts. That approach works temporarily, but maintenance becomes difficult once pipelines grow beyond a few transformations. Small schema changes become risky because logic is spread across multiple dependent stages. A more maintainable design separates the pipeline into distinct layers: IngestionValidationTransformationEnrichmentAggregationOutput The separation matters because each layer serves a different operational purpose. For example, ingestion should preserve source fidelity as much as possible. Validation layers should isolate problematic records before transformations begin. Aggregation logic should not contain ingestion-specific assumptions. This layered approach makes debugging significantly easier during production incidents. Validation Should Happen Early Many pipeline implementations validate data too late. Teams often begin transformations immediately after loading raw datasets, assuming upstream systems already enforce quality controls. In reality, enterprise data sources frequently contain null business keys, inconsistent timestamps, malformed identifiers, and duplicate records. Validation becomes much more manageable when it happens near ingestion. At minimum, validation checks should include: Null business keysDuplicate identifiersDatatype consistencyTimestamp integrityUnexpected categorical valuesRow count anomalies A practical pattern is separating invalid records into quarantine datasets instead of failing the entire pipeline immediately. This prevents a small number of bad records from interrupting large scheduled workflows while still preserving visibility into data quality issues. Another useful technique is maintaining row-count checkpoints between stages. Unexpected increases or decreases often identify join problems much faster than reviewing transformation logic manually. Managing Expensive Transformations Performance issues in PySpark pipelines usually come from unnecessary shuffling and poor partition strategies rather than raw data volume. Joins are one of the biggest causes of instability in large pipelines. A transformation that performs adequately in development can become extremely expensive once dataset sizes increase. Partitioning strategy matters here. Over-partitioning creates scheduling overhead and small files. Under-partitioning causes skewed workloads where a few executors process most of the data while others remain idle. The challenge is that there is no universally correct partition count. Pipelines behave differently depending on: Dataset cardinalityJoin distributionFile sizesCluster configurationTransformation complexity This is why reliable Spark pipelines require continuous observation and tuning rather than static optimization rules. Caching also needs careful handling. Many teams cache aggressively without monitoring executor memory pressure, eventually degrading overall cluster performance instead of improving it. In practice, caching should only be used for reused intermediate datasets with measurable recomputation costs. Handling Schema Evolution Safely Schema evolution becomes unavoidable in long-running analytics environments. New attributes are introduced. Legacy fields are deprecated. Source applications modify export structures without warning. Pipelines that rely on rigid assumptions eventually fail under these conditions. One practical approach is maintaining explicit schema contracts between ingestion and transformation layers. Instead of relying entirely on inferred schemas, pipelines should validate expected columns and datatypes before downstream processing begins. Backward compatibility also matters. Adding nullable fields is usually manageable. Renaming or changing datatypes is far riskier because downstream dependencies may silently break. This becomes especially important when multiple teams consume the same datasets. Reliable pipelines are not only technically correct; they are predictable for downstream consumers. Observability Is More Important Than Most Teams Realize Many Spark jobs are difficult to troubleshoot because they produce very little operational metadata. A successful pipeline should generate more than output files. Useful operational metrics include: Processed row countsRejected row countsStage execution durationPartition statisticsFreshness indicatorsSchema validation failures Without observability, debugging production incidents becomes reactive and slow. Logging also needs structure. Generic console output is rarely sufficient once pipelines become distributed across multiple workflows and orchestration systems. This is one reason mature analytics environments invest heavily in monitoring frameworks and lineage tracking systems. The goal is not simply running pipelines successfully. The goal is understanding pipeline behavior consistently over time. Reliability Requires Engineering Discipline PySpark provides distributed processing capabilities, but reliable analytics systems depend far more on engineering discipline than framework selection. Many pipeline failures are preventable: Transformations without validationUnmanaged schema changesMissing operational metricsTightly coupled workflowsInconsistent partitioning strategies As analytics environments scale, reliability becomes increasingly important because downstream systems depend on pipeline consistency for reporting, forecasting, machine learning, and operational decision-making. Teams that prioritize reliability early usually spend less time firefighting production issues later. Conclusion Reliable data pipelines are foundational to enterprise analytics, yet reliability is often treated as a secondary concern until failures begin affecting downstream reporting and operational workflows. PySpark provides the scalability needed for modern analytics workloads, but scalable pipelines are not automatically reliable. Reliability comes from disciplined validation practices, careful transformation design, observability, and operational maintainability. As organizations continue expanding their analytics capabilities, pipeline engineering quality increasingly determines whether downstream insights can actually be trusted.

By Harsh Patel

From DevOps to AIOps: How Agentic AI Tamed Our Multi-Substrate Chaos

The Mess We Started With When I took over the team, we had two substrates and a long list of problems that cut across both. On-prem ran on VMware. The core application was vendor-licensed, but over the years the team had layered so much custom code on top of it that the vendor could barely support it anymore. Upgrades were risky. Every change required manual configuration. Dependencies were undocumented. Libraries were old enough that half the team was nervous about touching them. There was no real monitoring — we found out about problems when users complained. AWS was in better shape, but not by much. The team was reactive by habit. Tickets came in and sat in a queue. Developers waited days for environment provisioning. On-call rotations were brutal because there was no good way to correlate what was happening across both substrates during an incident. The on-prem side had its own telemetry, AWS had its own, and nobody had a clean view of both at once. The team was capable. They were just spending the majority of their time on toil — manual interventions, ticket triage, configuration changes that should have been automated years ago. That is the environment we decided to change. What DevOps Actually Looked Like We called it DevOps. In practice, it was good engineers doing repetitive work because the systems weren't capable of doing it themselves. Provisioning a new environment meant opening a ticket, waiting for someone with the right access to pick it up, manually running through a checklist, and hoping nothing had changed since the last time the runbook was updated. Runbooks were the real source of truth — and they were wrong half the time. Incident response meant someone getting paged, logging into two different observability tools, trying to mentally correlate what they were seeing across VMware and AWS, and making judgment calls with incomplete information at 2 am. IT dependency was constant. Developers couldn't self-serve anything meaningful. Every access request, every config change, every environment spin-up went through a human. That queue never got shorter. This wasn't a people problem. The team was doing exactly what the system required them to do. The system was the problem. Why We Moved to Agentic AI We had tried the standard fixes. Better runbooks. Improved monitoring. More automation scripts. These helped at the margins but didn't change the underlying dynamic: humans were still the execution layer for work that didn't require human judgment. The shift to agentic AI was a deliberate decision to change that. An AI agent doesn't just surface a recommendation — it acts. It perceives its environment, reasons over a goal, executes a sequence of actions, observes the results, and adapts. Within a defined authority boundary, it owns the work. The key distinction from traditional AIOps tooling is autonomy. We had anomaly detection before. We had dashboards. What we didn't have was something that could take a detected anomaly and resolve it without waking someone up. What We Built and What Changed Self-Service Request Handling The ticket queue was the most visible symptom of how reactive our operation was. Developers submitted requests and waited. The requests weren't complex — environment provisioning, access grants, config changes, dependency updates — but each one required a human to pick it up, context-switch, and execute. We built an agentic request layer in front of our service catalog covering both substrates. A developer requests a dev environment on AWS — the agent parses the intent, validates against policy, provisions via Terraform, runs validation checks, and delivers a verified environment with an audit trail. A request that touches the on-prem VMware substrate follows the same flow, adapted to that environment's provisioning model. Requests that used to sit in a queue for a day or more now resolve in minutes. Developer self-service went from a goal to a reality. The ticket queue dropped significantly. The engineers who used to service that queue are doing more interesting work. Cross-Substrate Incident Response Incidents were hard because the blast radius of a problem rarely stayed on one substrate. Something degrading on the VMware side would show up as anomalous behavior on the AWS side, and the on-call engineer had to stitch that together manually with two separate observability tools. Our incident response agent ingests signals from both substrates continuously. When an anomaly appears, it correlates across both environments, maps the probable fault tree, and executes initial remediation — service restarts, traffic rerouting, config rollbacks — before a human is involved. If it hits a decision it isn't authorized to make, it escalates with a structured brief that tells the on-call engineer exactly what it found and what it did. Mean time to detect dropped materially. Mean time to resolve dropped further. The number of incidents that require a human for resolution is a fraction of what it was. The on-call rotation is no longer brutal. Proactive Monitoring and Drift Detection The on-prem environment had no real monitoring. Problems surfaced through user complaints. We set that as a baseline, but we went further — we deployed a drift-detection agent that continuously compares the actual state of both substrates with the desired state defined in our IaC. When it finds drift, it remediates automatically for low-risk changes and flags for human review for anything higher risk. The shift from reactive to proactive operations didn't happen because we wrote better runbooks. It happened because we deployed something that watches continuously and acts when it sees a problem, rather than waiting for a human to notice. Governance: What the Agents Can and Cannot Do Every agent operates within an Authority Envelope — a defined set of actions it can take autonomously, the conditions under which it must escalate, and an immutable audit log of every action it takes and why. Irreversible actions — production data changes, security group modifications, anything touching the vendor-licensed on-prem application's core configuration — require a human approval gate. The agent prepares the action and presents its reasoning. A human authorizes it. This boundary matters. The goal isn't to remove humans from operations. It's to make sure humans are applied to decisions that actually require judgment, and agents handle everything else. We also red-team the agents regularly — testing edge cases, contradictory signals, and misclassification scenarios. These exercises built more confidence in the system than any amount of documentation could. What Changed for the Team? The toil that used to consume the majority of engineering hours is largely gone. The ticket queue no longer defines the team's day. The 2 am context switching between two observability tools is mostly gone. Developers self-serve. Incidents resolve faster and with less human intervention. What's left is the work that actually requires engineers: improving the agents, designing for resilience, building new capabilities, making architectural decisions that compound over time. The team is more engaged. The work is more interesting. That's not incidental — it's the point. Where This Goes Next The on-prem VMware environment is a constraint we're still working within. Agents help manage it, but the longer-term path is modernization — reducing the custom code surface area, retiring the dependencies that make the vendor relationship difficult, and moving more workload to AWS where the operational model is cleaner. The agentic platform accelerates that work too. With agents handling day-to-day operations, the team has capacity to work on the modernization that was always the right answer but never had priority over keeping the lights on. The move from DevOps to AIOps is not about replacing engineers. It's about changing what engineers spend their time on. Agentic AI handles the execution. Engineers handle the judgment. When you get that division right, the whole operation gets faster, more reliable, and more sustainable — without adding headcount. That's the case for agentic AI. Not as a technology trend. As a practical answer to a real operational problem most enterprise teams are already living with.

By Mayank Jain

Avoid 10 Pitfalls of Overautomation in Software Development

Automation is an important part of efficient software development. Like all potential benefits, it needs to be applied carefully and with a clear strategy. Without a transparent, sustainable balance between human and automated innovations, what should make life easier can become a risk. Overautomation occurs when teams apply automation beyond the point where it improves efficiency, reliability, or maintainability. Failing to be cautious with how it is applied in these instances can lead to significant issues. Here are 10 important automation pitfalls to avoid. The Hidden Costs of Overautomation The collective drive to embrace automation can lead to long-term overreliance on it. Finding an appropriate level of reliance ensures that software isn’t over- or underutilized, both of which have negative impacts. Automation doesn’t fix critical issues. Its goal is to make processes more efficient. Without a clear understanding of the main pitfalls of overautomation, inefficiencies are likely to be magnified. 1. Automating a Flawed Process Automation cannot fully define the problems within a flawed process on its own. It will simply make a poorly understood approach run quicker. A flawed process still needs human input, defined business goals, and technical requirements in place before automation can be considered. 2. Creating Brittle Systems With High Maintenance Overhead Automated scripts may seem rudimentary at first. However, they can easily evolve into interconnected complexities. These scripts can become difficult to debug and costly to maintain. Every piece of automated code should be considered a liability that requires a dedicated responsible party to take ownership of. 3. Assuming Automated Security Triggers Are Enough The use of automated security scanners and log analyzers is an important contributor to security. Overreliance on automation can create a false sense of security. Sophisticated threats can often bypass the triggers of automated systems. This is because poorly configured automation only looks where it is told to specifically check. The human element in threat hunting remains necessary to identify and prevent security risks that detection systems miss. 4. Automating With Unclean or Unvalidated Data Poor data going into automation will lead to poor data coming out of automated pipelines. Raw, unvalidated analytics, testing, or machine learning models can only produce flawed outcomes due to a lack of data hygiene. Data cleaning techniques should be implemented before considering automation. Removing irrelevant information, filling in missing values, and reformatting prepares the data for effective application. 5. Ignoring the Human Element Avoid pursuing automation immediately without considering the people involved. Skill development is essential for all organizations, and removing developers from this process can limit problem-solving capabilities over time. Software engineering is a broad spectrum that extends beyond code generation. Relying solely on automation creates skill gaps among developers and in essential skill sets. 6. Putting Automation Above Developer Experience There should be a clear reason and a goal for automation. Otherwise, businesses are just automating for the sake of doing so. Helping improve the lives of human developers should be a priority. Poorly implemented, hard-to-use automation adds friction and frustration to the human element of the process, even if it looks good on a dashboard. 7. Measuring the Wrong Metrics Vanity-focused metrics aren’t always strong measures of success. Developers should avoid measuring the percentage of tasks automated and the number of pipelines run. Instead, they should look for positive outcomes in areas such as bug resolution and fewer deployment failures. 8. Ignoring Critical Trade-Offs for Simplicity Automation can be a useful instrument. However, its blunt nature often doesn’t account for nuanced trade-offs. This can mean prioritizing short-term development velocity over long-term maintainability, or sacrificing system security for a more convenient user experience. Experienced developers can handle these nuances with greater consideration. 9. Devaluing the Process of Human-to-Human Review The CI/CD anti-pattern of relying on automated checks treats the code review process as a simple pass/fail test. In reality, its purpose is far broader. Removing the human aspect of the code review reduces knowledge sharing and adherence to high standards for code architecture and style. 10. Developing Systems That Lack Clear Documentation Automation scripts require the same attention to detail as production code. This means clearly defined ownership and detailed documentation. If a complicated automation pipeline is created by someone who eventually leaves a developer team, documentation keeps that expertise in place. Without it, the script’s logic can be lost and turn automation into an operational risk. The Right Balance for Sustainable Automation Companies that use automation as a useful tool that streamlines processes rather than the end goal strike the right balance. Effective automation augments human intelligence and does not seek to replace it. Overuse of these innovations can make many processes more complex than before. Embracing a more strategic and critical approach can lead to sustainable automation practices. For developers, this can be the difference between common pitfalls and best practices.

By Zac Amos

How to Build a Solid Test Pipeline in the Era of Agentic AI Development

AI agents!! AI agents!! AI agents is the new buzzword.. they can generate code far faster than any human can review it. Because of this shift in software development, engineering teams are transitioning away from writing code to serving as pure reviewers. The sheer volume of daily commits has skyrocketed, leaving developers drowning in pull requests. This review fatigue sets in quickly, creating a massive blind spot for software quality. To adapt, testing teams must construct a multi-layered defensive pipeline. This pipeline shouldn't just check if the code works; it needs to validate performance, functionality, and security at scale. Layer 1: Static Analysis and Secret Scanning: Pre-Commit First Gate The easiest defects to catch are those found before code leaves the developer's machine, and agents known to produce many of them, like hallucinated APIs, deprecated patterns from training data, and credentials handled incorrectly. Static analysis and secret scanning run in seconds as pre-commit hooks. Their job is not to prove correctness; rather, they identify and flag any security concerns. It is to guarantee the expensive layers downstream never waste a run on code with problems a regex could have caught. Secret scanning is non-negotiable; for example, an agent pasting a connection string into source is not hypothetical, and once a secret reaches a remote branch, you are doing incident response, not code review. Layer 2: Unit Tests: Useful, But Only With an Independent Author This is the large gap I've personally encountered. An agent writes the implementation, then writes unit tests for it, and coverage hits 90%. But the problem is those tests were derived from the same assumptions as the code, often in the same session. Unit tests still belong in the pipeline, but with a changed role. AI-generated unit tests don't always catch all the issues, as the tests are also written by an agent with the same context, so it can be biased. These tests should be written by another agent with no implementation context who comes up with a new plan and validates it, or by a human who has no bias. Layer 3: Integration and Functional Tests: The Center of Gravity The most consistent finding about AI-generated code is that it works well in isolation and on the happy path. It often breaks at boundary conditions and edge cases. Let's take a real-life example at work: An agent working on building an order history endpoint. The code gets written as per specification, the unit tests pass, and it runs fine with the test data. In a production system, if a fraction of accounts have null shipping addresses, this could break the API response, and nothing earlier in the pipeline had any chance of catching it, because every earlier check validated the code against the same clean assumptions the agent generated it from. Integration and functional testing are slow to write and brittle to maintain. It is the first point in the pipeline where agent-written code meets something resembling the actual system rather than a mock of it. It must run against real or realistically simulated dependencies, because mocks written from the same specification the agent used prove nothing. It must use realistic test data that can be created from production patterns. And its failures must be treated as hard blockers. If you can invest heavily in only one layer of this pipeline, invest here. Layer 4: Performance Tests: Catching Code That Is Correct and Careless AI-generated code has a serious scalability problem because the prompt's focus is always more on implementation than on efficiency and performance. Things I have personally faced are: a query inside a loop that becomes an N+1 problem at volume, a full table loaded into memory, and missing pagination that returns 50 rows in staging and 5 million in production. Performance testing therefore moves earlier than it traditionally sat. Full load tests per commit are unrealistic, but lightweight suites are not. Creating a standard load/performance test which measures the performance of API’s before the change and with this new change and if it degrades a certain percentage then immediately flag it to avoid scalability issues in production. Layer 5: Security Tests: The Layer Where AI Is Actively Dangerous Every other layer compensates for what agents do not know. Security compensates for what they learned wrong. Models have been trained on past code and patterns that had various exploitable vulnerabilities in today's world. So, ensuring all the code written by AI is security compliant is a must to keep your application secure. SAST (Static Application Security Testing): Identifies known vulnerable patterns in the code SCA (Software Composition Analysis): Automatically checks every newly added packages/library for known CVEs in CVE databases. DAST (Dynamic Application Security Testing): Identifies vulnerabilities that only happen at runtime. Layer 6: Refined Regression Tests from Production Telemetry: Closing the Loop Every layer above shares one weakness, i.e., agents write code from documents, and documents drift from reality. The final piece attacks that drift directly by feeding observed production behavior back into the suite. Real traffic patterns become regression cases, real dependency responses refresh the contracts and fixtures, and real incidents become permanent tests. Without this loop, your environments and mocks age quietly until they describe a system that no longer exists. The Pipeline The Bottom Line Having this multi-layered pipeline, which focuses on various aspects of code quality in terms of functional and non-functional, is super critical. Having robust pipelines in place builds confidence in code being pushed to production and helps catch issues at very early stages, saving enterprises a lot of time and money in the era of agentic development.

By Sai Rakshit Yerram

Security Is a Platform Property, Not a Pipeline Step

A few weeks ago, I disabled key authentication on an Azure storage account we used for Terraform state management. It was one of the key security recommendations in Microsoft Defender for Cloud. It made sense to use RBAC-only permissions, enforce PIM approvals for the Infrastructure team, and avoid storing static credentials in config files, where leaks are possible. This is exactly the kind of control you want for state files, which contain the keys to your entire cloud environment. But I missed an important line in the azurerm backend config. If use_azuread_auth = true is not explicitly set, the provider uses key-based authentication by default. Since key authentication had been disabled, terraform init failed and the pipeline broke. The actual fix was easy, but finding what was wrong, not so much. JSON terraform { backend "azurerm" { resource_group_name = "rg-tfstate-prod" storage_account_name = "sttfstateprod001" container_name = "tfstate" key = "platform/prod.tfstate" use_azuread_auth = true } } This is not the kind of detail every engineer should have to remember in every repository. It belongs in the module. That is the gap I am talking about: the security decision was correct, but the delivery path still allowed the wrong configuration. The same pattern shows up elsewhere: storage accounts left open, IAM roles with excessive permissions, credentials committed to repositories, diagnostic settings missed, or Terraform modules that still allow insecure defaults to slip through. Security Enters Too Late, and Everyone Pays For It There is a common pattern: a developer builds a feature, security reviews it and flags something, the developer reworks it, the release gets delayed, and someone gets the blame. The cycle repeats until everyone is frustrated. The cost side of this doesn't get enough attention. Catching a vulnerability while you're still writing the code is a relatively quick fix. Finding the same issue in production is a different situation entirely: incident response kicks in, there may be regulatory questions to answer, and the reputational impact is difficult to measure. The further right security sits in the delivery process, the heavier each failure gets. Most teams are inadvertently set up to find problems at the point where they cost the most. What Shifting Left Actually Looks Like People toss around 'shift left' so much that it’s lost its punch. Here’s what it actually looks like in practice: Plan: Include threat modeling in sprint planning and spend 30 minutes on it rather than managing it in a separate process or document.Code: Use IDE plugins to flag insecure patterns in real time while you code and pre-commit hooks to run secrets detection before committing the code. The developer finds out immediately, not weeks later in a review.Build: Run SAST on every commit to catch injection risks, insecure cryptography, and hardcoded secrets/credentials before code is deployed to a shared environment.Test: Let DAST probe the application in staging as an attacker would. SAST reads code, and DAST attacks the running system. One finds what the other misses.Deploy: Scan your IaC before applying changes, check container images for CVEs, and use OPA policy gates to verify signing, permissions, and network policies before anything reaches production. Running security through each of these stages means issues come up when they are still manageable, rather than after they have already caused damage. Installing Tools Is Not a Program How DevSecOps failures look in practice: Tools like Checkov and Semgrep are configured in the pipeline, and by next month, the developers have written suppression rules for the findings so the feature can be shipped. The tools keep running, but no one is checking their outputs. Three things matter more than which tools you choose: Tuning: SAST generates false positives because it doesn’t know what’s happening at runtime. Run a co-triage session with a developer and a security engineer; work through the first 50 findings; fix the problematic rules; or write a justified suppression. After a couple of sessions, developers start trusting the output because it becomes more accurate and actionable.Signal engineering: Let critical and high CVEs block the pipeline immediately, while medium and low go to a dashboard with remediation SLAs. Developers will find ways to bypass the findings instead of fixing them if you block the commit for every medium, which will end up in a bigger mess than you started with. Ownership: Send findings straight to the person who can fix them, and give them enough info to act. A centralized security queue is where urgency goes to die. The Terraform backend scenario I opened is the exact example. The security decision to use RBAC only and disable key authentication was absolutely the right one. But here’s the catch: use_azuread_auth = true was not enforced during provisioning. If a hardened module had that flag set by default, that misconfiguration simply couldn’t have happened. That’s the real difference between having a security policy and actually building a security platform. The Platform Team Is the Structural Answer Adding more process to a structural problem doesn’t fix it. What’s required is a different model entirely. A real platform team treats the internal platform as a product, with engineers as its customers. Their job is to make secure, compliant delivery the path of least resistance: golden path templates, a shared CI/CD toolchain, secrets management, and self-service provisioning, all built with guardrails from the start. When teams repeatedly provision similar workloads — containerized APIs, data pipelines, Kafka consumers – the same security configuration decisions recur. Golden path templates address this by embedding those decisions up front. Encryption at rest is already configured, IAM permissions are scoped to what the workload actually needs, logging and network policies are in place, and the backend authentication flags in the Terraform modules are set correctly from the start. A developer selects the right template, fills in the required fields, and provisions. The repository they get back already has security gates running in the pipeline. There is no separate step to secure it afterward. Figure 1: A secure golden path platform embeds security controls into the default delivery path. This is what removes the need for individuals to get every detail right under pressure. In my experience, even when you know the correct configuration, you can still miss something in the moment. The platform handles that by making the secure option the default. In many organizations, platform teams work best when they sit within Engineering rather than reporting directly into the CISO function. If they are seen mainly as a compliance function, product teams may treat them as another gate to work around. Security should define the policies and risk boundaries; Engineering should build and operate the platform that makes those policies usable. Where to Start: Sequence the Platform, Don’t Boil the Ocean The most common mistake is trying to implement everything at once. Every scanner, every policy gate, every access control change lands in one big push. It creates noise before it creates trust, and teams lose confidence in the tooling before it has a chance to prove its value. Sequence it instead. Months 0 to 3: secrets scanning as a pre-commit hook, SAST in CI, IaC scanning before Terraform apply, and a security champions program with one dedicated developer per squad. Low friction, immediate signal, nothing that unnecessarily blocks delivery. Months 3 to 6: DAST in staging, container image scanning, OPA policy gates, and SCA on every build. At this point, the platform needs to make a clear distinction: critical and high findings stop the pipeline; everything else goes into a remediation backlog with defined ownership and SLAs. Months 6 to 12 mark the point at which platform security matures into deeper controls: workload identity, privileged access management, zero-trust network policies, and a real-time compliance dashboard. Never trust, always verify, and assume breach stop being principles on a slide and become defaults in the environment. Don't wait for a fully staffed platform team or executive sponsorship. The Terraform backend fix I mentioned earlier eventually became a hardened provisioning module used by the wider infrastructure team, turning a one-off incident into a reusable secure pattern. No one needs to remember the flag because the platform handles it automatically. That's what security as a platform property actually looks like. Not a gate at the end. A system that makes the right thing the easy thing, by default, every time.

By Naveen Kalapala

DevOps and CI/CD

DZone's Featured DevOps and CI/CD Resources

Monthly Top DevOps and CI/CD Experts

The Latest DevOps and CI/CD Topics