DZone Spotlight

TL; DR: The Product Backlog Refinement Agent and the Risk When a Suggestion Becomes a Decision Preparing a refinement session is tedious, and AI is good at the tedious part: reading every Product Backlog item, every repeated comment column, and the team’s own records on the afternoon before. The risk sits elsewhere. A refinement agent asked to “help with refinement” can take the conversation away from the team without writing a single acceptance criterion: “suggesting” to the Product Owner or Product Manager which items to close, merge, split, or create is enough, and the questions the team was supposed to work out arrive in the brief already answered. The remedy is a boundary decided before the first run: the agent marks evidence and asks questions, the Product Owner checks the brief and sets the agenda, and the team refines. Writing that boundary into an A3 Handoff Canvas, together with what the agent reads and who checks its output, turns a good intention into a rule the agent follows and the Product Owner can verify. Thesis: This article explains how Product Owners and Product Managers can use AI to prepare Product Backlog refinement without pre-empting the team’s conversation: four evidence checks, a four-part test separating suggestions from decisions, written boundaries, and human verification of the agent’s brief. Disclaimer: I read Charniak/McDermott’s book on “Artificial Intelligence” decades ago; of course, I use AI for research, translations, proofreading, challenging story arcs and article structures, creating artifacts such as graphics, and summarization. It is a production tool, not a substitute for thinking. The Danger of a Preparation Turning Into a Decision The familiar worry about AI in Product Backlog refinement is that a model writes the Product Backlog items, the acceptance criteria, and the estimates, and a tired team nods them through on a Tuesday afternoon. The worry is justified; however, it describes the easy case, because a drafted acceptance criterion is visible, and a team would usually notice that someone else did its homework. The more challenging case looks like help. Consider a hypothetical brief by a refinement agent that contains no descriptions, no acceptance criteria, and no new order, yet recommends closing one item as a duplicate, splitting another, creating a third, and calling four items “ready candidates” for the next Sprint. Every line reads as friendly advice to the Product Owner, and every line settles a question before the team has discussed it. A recommendation decides nothing until someone accepts it and acts on it. Once it sits on the agenda, however, the team is likely to discuss the recommendation instead of the question behind it, and that is how preparation quietly turns into a decision. It sneaks in through the back door, feeding on the human desire to avoid unnecessary work when organizational pressure is already high. What Refinement Is For, and What Preparation Should Leave Alone The Scrum Guide describes refinement as “an ongoing activity to add details, such as a description, order, and size,” which captures the mechanics and leaves out the purpose. I see refinement as the final part of product discovery, once hypothesis validation has been established: the team asks whether an item is really the best use of its time or whether something else would create more value for customers. A Product Backlog item is a token for that discussion, an invitation to align, and a good session ends with shared understanding sufficient to make the next investment decision, including the decision not to build something. Many teams hold a regular session for this, and often those sessions turn into a hunt for problems someone could have spotted the day before: an item a stakeholder insists on moving to the top, a backlog item without acceptance criteria that everyone expected someone else to write, an item nobody has touched for 6 months. Preparation exists to spare the team that kind of administrative work, and reading a Jira export and the team’s notes on the afternoon before is exactly the kind of thorough, tedious work an AI agent does well. The question is where preparation ends. In September, I described ten anti-patterns that get worse when teams add AI to a dysfunctional Product Backlog process, “Refinement Without the Team” among them, and an assistant that hands the Product Owner ready-made resolutions is a polite version of that anti-pattern: the team still meets, yet the decisions it was supposed to reach already arrive pre-made. In my view, the most critical consequence of dysfunctional refinement is a team that loses its ability to have productive discourse. Typically, building the wrong thing follows; think of lack of return on investment and opportunity costs. So my rule for such a refinement agent is short: mark, suggest, never solve. The agent marks what the team needs to talk about and may suggest what to discuss, and solving stays the team’s job. Four Things a Product Backlog Refinement Agent Can Check, and What It Cannot Know Of the 27 Product Backlog and refinement anti-patterns I have described, many are behavioral and leave no trace in a Jira export. Four do leave traces, provided the agent reads the team’s own records next to the export: the Product Goal, the product roadmap with its change log, the anti-product backlog, the notes from previous Sprint Reviews, and the actions from the last few Retrospectives. Readiness The agent looks at items early in the current order and reports what the records contain: whether an item states a problem or an intended outcome, whether it lists open questions, and whether a Product Backlog item waits for an open spike for technical clarification. Whether that is enough for a useful conversation remains the Product Owner’s judgment. A missing acceptance criterion at the start of refinement is no defect; it is an agenda question, because the Product Owner will have ideas about how an item should meet the desired outcome, and the team negotiates the criteria together by the end of refinement. Being early in the order does not preselect an item for the next Sprint, either: the order reflects the team’s best current judgment of how to use its time, and in Sprint Planning, the team first agrees on the Sprint Goal, and then the Developers select Product Backlog items they deem necessary to accomplish the goal. Goal Fit and Prioritization by Proxy The agent compares the Product Backlog with the Product Goal, the roadmap, and the anti-product backlog, which lists the ideas and requests the team decided not to build, either not now or not in this product, with the reason for each decision. It marks items that serve no part of the Product Goal and items that revive an idea the team already rejected. It also marks prioritization by proxy, which an export reveals only indirectly, because no outsider has write access to the Product Backlog: a stakeholder can neither create an item nor change its position. What the export does show is a member of the Scrum Team, usually the Product Owner, creating or moving an item on a stakeholder’s behalf: a comment such as “per the sales director’s request,” a label such as “mgmt-request,” a creation date right after an escalation, and no discovery decision and no link to the Product Goal behind the item. An item that skipped discovery because someone with political weight asked for it was never put to the validation test. However, the agent cannot know what the conversation behind the item looked like, or whether a rejected idea deserves a second look; that takes the people involved. Outdated Items Items nobody has touched for months, or that still name a release that shipped long ago, deserve a look, because a Product Backlog that turns into a storage for ideas hides the work the Product Goal needs. Age is a signal for inspection, nothing more: an untouched item can be the very work the Product Goal depends on, and a refinement agent that simply sorts by the “Updated” column will happily declare it dead without inspection. Missing Work The Scrum Guide calls the Product Backlog “the single source of work undertaken by the Scrum Team.” The agent can hold that claim against the team’s records: open items from the Sprint Review, Retrospective actions, and even work done outside the Sprint Backlog. A discrepancy is a finding. Whether it leads to a new item is for the Product Owner and the team to decide, because the work may sit in another team’s Product Backlog, stay outside it on purpose, or already be resolved. This check is also why the agent needs more than the export, since an export cannot show what is missing from it. For any of the four checks, “no supported finding” and “insufficient evidence” are valid results. However, the teams that would gain most from this preparation often keep the fewest records: no anti-product backlog, no product roadmap change log, Sprint Review notes that say little more than “stakeholders loved what we built,” and the reasons behind past decisions stored in the heads of the Product Owner and a few senior engineers, or buried in Slack threads. Such a team will get a brief full of “insufficient evidence.” That brief is still a finding because it shows where the team has no memory of its own decisions. Also, it helps only when people read it as a statement about their records rather than a failed run. Otherwise, the team has added AI on top of a dysfunctional system. A small start is enough to make the checks useful: the Product Goal, the last Sprint Review notes, and the last Retrospective actions all cover readiness and missing work, and a short list of recent “no” decisions, one line of reason each, gives the goal-fit check something to work with. In the agent’s instruction, name only the records the team actually keeps, so the brief does not report the same missing roadmap change log on every run. Where a Suggestion Turns Into a Decision A hypothetical example shows where this red line runs; the item keys, people, and dates are invented. A Jira export taken on Monday shows EXP-187, “Export for enterprise accounts,” created last Tuesday by the Product Owner, in the second position of the order, with the Priority field set to “Highest,” a one-line description, and the Product Owner’s own comment “Per the sales director’s request, moved to the top; a big customer is waiting.” The roadmap’s change log keeps the export feature in “Next,” no discovery decision mentions the request, and EXP-112, a CSV export item from June, may cover part of the same need. A brief that does its job marks this as prioritization by proxy and cites its sources: the comment, the creation date, the position in the order, the roadmap entry, and the missing discovery decision. It adds what it cannot know (which customer, and what problem that customer has), and it may suggest that the Product Owner revisit the request with the sales director before refinement. A brief that crosses the line writes “Close EXP-187 as a duplicate of EXP-112,” or “Move EXP-187 back down the order,” or “Ready candidates for the next Sprint: EXP-187.” Each of these settles a decision that belongs to the Product Owner, however helpful it sounds. What follows is the Product Owner’s work, and most of it happens outside refinement. The sales director’s request is exactly what my discovery article describes as a passive source: “Product requirements from internal stakeholders, which arrive unrequested and usually with political weight attached.” It goes through the funnel I described in AI-Assisted Product Discovery, where only what serves the general product vision and the current product goal passes the neck and earns a place in the work: it may become a hypothesis to validate with an experiment, an entry in the anti-product backlog with its reason, or an item the Product Owner places where it belongs. Until then, EXP-187 has no claim on the second position, until the Product Owner reconsiders the order of the Product Backlog. A sales director who can push an item to the top through the Product Owner points to an impediment for the Product Owner and the Scrum Master. Also, it is not an agenda item for the Developers, as refinement time stays reserved for work that has legitimately entered the Product Backlog. Questions that belong in refinement need the same discipline. Take a story that waits for a spike that is still open: “What do we need to learn from the spike before we can negotiate the story’s acceptance criteria?” leaves the conversation to the team. “Should we split the story by permission level?” ends with a question mark, yet it carries a solution, and it leaves the team to say yes or no to an answer the agent chose. A valid refinement suggestion by an AI agent has to pass four tests: It rests on the supplied records,It makes the uncertainty visible,It leaves materially different answers open, andIt does not assume the disputed resolution. Why be this strict about a polite suggestion? A ready-made answer shapes the judgment of the person who reads it. In two preregistered experiments with 2,582 participants, Sterling Williams-Ceci, Mor Naaman, and colleagues found that people writing about societal issues with a biased AI writing assistant shifted their attitudes toward its position, most of them unaware of the bias and its influence, and warnings before or after the task did not mitigate the shift (Biased AI Writing Assistants Shift Users’ Attitudes on Societal Issues, Science Advances, 2026). The study tested autocomplete suggestions during writing, which influenced people more than similar arguments shown as a static list. An AI-assisted refinement brief is a static list, so the study is a reason for caution and does not prove that a brief anchors a Product Owner. Writing the Line Down Before the First Run Writing the red line down before a refinement agent runs for the first time turns it from an intention into a rule. The A3 Handoff Canvas is the place for it: it records what the agent reads, what its brief looks like, who checks it, what the agent must never do, and the last part goes into the agent’s instruction word for word. For an AI-assisted refinement agent, the stop rules read: Never write acceptance criteria,Never estimate,Never split, reorder, or remove an item, and never move one to the anti-product backlog,Never advise closing, merging, splitting, creating, or editing an item; ask the team a question insteadNever propose what goes into the next Sprint or what the Sprint Goal should beNever invent a source, a reason, or a figure. (If a record needed for a check was not supplied, cannot be located in the supplied material, or cannot be read, state that limitation under “What I could not check.” Do not infer that the record does not exist. Do not treat an experiment, proposal, or unverified note as a completed decision or shipped release; state the exact status recorded in the source.) Three of these rules deserve a word. Acceptance criteria stay out because the team negotiates them by the end of refinement, and an agent’s draft would preempt that negotiation; a different workflow can still help with criteria after the team has talked. Estimates stay out because their value lies in the Developers’ discussion that reveals diverging ideas of the effort before work starts, as I argued in Estimates Are Useful, Just Ditch the Numbers; the number was never the point. And the last rule addresses a quieter error: a model that reads a planned experiment as a commitment, or a hypothesis as a result, turns the team’s open question into a fact nobody decided. In terms of the A3 Framework, this preparation is Assist: “AI expands options. You own the outcome.” The refinement agent works from a read-only export, so it cannot change the Product Backlog even if a rule fails. And if you already run an agent for product discovery, it needs its own Handoff Canvas, because it reads different inputs, delivers different output, and stops at a different line; I consider that AI transparency at the operational level. The Product Owner Routes Every Finding A brief that sounds thorough invites a dangerous shortcut: forwarding it to the team unread. Polished AI artifacts quickly fool people into believing that the AI is doing the real job. So the Product Owner opens the item or record behind every finding that could change the agenda and then looks at a few things the brief did not flag, for example, the items nearest the top that it never mentions, because a brief cannot report its own omissions. Then the Product Owner decides where each finding goes, and only some of them belong in refinement: Settled by the Product Owner: Housekeeping, such as an obvious duplicate created by mistake or a release field nobody updated, and restoring the order of the Product Backlog.Sent into discovery: Requests that skipped the funnel, such as the sales director’s export, and rejected ideas whose revisit condition may have occurred.On the refinement agenda: Questions that need the team’s knowledge, such as what a Product Backlog item needs before the team can negotiate its acceptance criteria.Raised as an impediment: Systemic causes, such as stakeholder pressure that bypasses discovery or records the team does not keep, for the Product Owner and the Scrum Master, or for the next Retrospective. Sharing the agenda with the Developers and the Scrum Master before the session gives them the chance to add the questions the brief missed. If a brief proposes resolutions anyway, correct the instruction and run it again; if it keeps failing, prepare the agenda by hand, the way the Product Owner hopefully did before the agent existed. The remaining failure mode relates to the Product Owner: Treating the brief as sufficient grounds to settle a question that needs the team’s knowledge brings back the very problem the stop rules were meant to prevent, only with a human signature. The brief must not become a substitute for a discussion that has not happened. Conclusion: Try It Before Three Refinements, Then Decide Treat the first runs as a local experiment. Note how long your usual preparation takes, then run the refinement agent on a sanitized export, in an AI account your company has approved, before three or four refinements. Record the total time, including checking the findings that proved useful, the findings the records did not support, and the omissions you noticed, knowing you won’t catch all of them. If checking and correcting the brief keeps taking longer than preparing the agenda by hand, stop, because the agent then costs more than it saves. Otherwise, ask the team whether refinement gained time for the critical discussion, and compare the results with your usual preparation; a few sessions give you the basis to decide whether to continue, adjust, or stop. If you would rather practice first, the AI4Agile Online Course v4, which launches on November 27, 2026, includes an exercise in which you build this refinement preparation agent: you fill its A3 Handoff Canvas, run it on a 40-item Product Backlog export and the team records of MegaBrain.io, the course’s fictional startup, check its brief, and set the Product Owner’s agenda before comparing your decisions with my worked solution. When the next brief arrives on your desk the afternoon before refinement, which of its suggestions already contains an answer your team has not given yet? Key Questions This Article on the Product Backlog Refinement Agent Answers How Can AI Help a Product Owner Prepare Product Backlog Refinement? A refinement agent can read the Product Backlog export and the team’s records on the afternoon before refinement and mark Product Backlog anti-patterns: items not ready for a conversation, items that serve no part of the Product Goal or entered at a stakeholder’s request without a discovery decision, outdated items, and work the Product Backlog does not show. The Product Owner checks the brief and routes each finding; the team refines. What Can an AI Agent Check in a Product Backlog Before Refinement? Four checks leave traces in an export and a team’s records: readiness, goal fit, outdated items, and missing work. Prioritization by proxy shows only indirectly, because no outsider can write to the Product Backlog: an item the Product Owner created or moved up at a stakeholder’s request, visible in comments, labels, and a missing discovery decision. None of the checks can establish what the team understands or still wants; the people involved need to do that. How Do You Tell an Open Question From a Disguised Recommendation? A question passes four tests when it rests on the supplied records, makes uncertainty visible, leaves materially different answers open, and does not assume the disputed resolution. “Should we split the story by permission level?” fails despite its question mark, because it carries a solution. “What do we need to learn from the spike before we can negotiate the acceptance criteria?” leaves the answer to the team. Should a Refinement Agent Write Acceptance Criteria or Estimates Before the Team Refinement? Not in a preparation brief. The Product Owner brings ideas about how an item should meet the desired outcome, and the team negotiates the acceptance criteria together by the end of refinement; a refinement agent’s draft would preempt that negotiation. The value of an estimate lies in the Developers’ discussion of diverging ideas of the effort; the Scrum Guide assigns sizing to “the Developers who will be doing the work.”

By Stefan Wolpers CORE

Finding an eval() call in a Python repository is easy. Understanding where its argument comes from takes more work. The same problem appears with shell commands. A scanner can point to subprocess.run(), but a reviewer still needs to trace the command through assignments, string formatting, and branches. Then comes the harder question: can someone outside the intended trust boundary control that input? I built rce-path to make that investigation more structured. It scans a local Python checkout, identifies modeled execution sinks, and records the input flow it can establish from the source. Each finding includes locations, ordered evidence, and the assumptions that still need review. The package also supports optional OpenAI and Ollama annotations. Static scanning works independently, so you can inspect the evidence before deciding whether to involve a model. The Problem I Wanted to Address A suspicious function call is a starting point for security review. It leaves several questions unanswered: Where did the argument originate?Which assignments or expressions carried it to the execution call?Does the path depend on a branch or an unknown value?Is the source request data, a function parameter, or local input?Which parts of the flow remain unresolved? These details matter because a function parameter is not necessarily remotely controlled. A request-derived value may pass through authentication or validation outside the scanner's view. A shell invocation using fixed literals needs a different investigation from one assembled from request data. My goal with rce-path is to give the reviewer a traceable path and make the remaining uncertainty visible. The Core Idea The current release, version 0.1.0, uses Python's abstract syntax tree and per-scope control-flow analysis to track modeled inputs toward execution sinks. A source is an input origin the analyzer recognizes. A sink is an operation that can execute Python code or a shell command. Between them, the analyzer follows supported assignments and expressions, including concatenation and f-strings. It also handles branches, early returns, and bounded loop analysis. The report preserves the steps connecting these operations. If a call's return value cannot be resolved, the scanner records that uncertainty rather than filling the gap with an assumed data flow. Targets are read as source data. A normal scan does not import the target package, install its dependencies, invoke its build hooks, or execute its code. What the First Release Covers The initial sink rules cover: eval() and exec(), including supported built-in aliases.os.system() and os.popen().Shell execution through subprocess.run(), call(), check_call(), check_output(), and Popen().subprocess.getoutput() and getstatusoutput(), which imply shell execution. Modeled sources include function parameters and Flask request inputs such as request.args, request.form, request.data, and request.get_data(). Environment variables, command-line input, stdin, and file reads are categorized as local inputs. The subprocess rules account for shell behavior. An unknown shell value introduces an explicit condition. Cases using shell=False have coverage limitations; they should not be interpreted as a general safety guarantee. This is a focused first release. It lets researchers and Python developers investigate supported execution paths without requiring an AI backend. A Small Example Consider this deliberately unsafe example. Save it as demo.py, but do not run it as an application: Python from flask import request import subprocess def lookup(): host = request.args.get("host", "") command = f"nslookup {host}" subprocess.run(command, shell=True) The security question is whether request data reaches a shell command. Here, the analyzer can model the request source, its assignment to host, its inclusion in the f-string, and the assignment to command before the shell invocation. There is no route registration in this snippet. It illustrates a data-flow pattern; it does not establish that the function is deployed or reachable over a network. To try rce-path, use Python 3.11 or newer and install the package from PyPI: PowerShell pip install rce-path Save the example as demo.py, then generate a Markdown report: PowerShell rce-path scan demo.py --format markdown --output report.md For this example, the report identifies the subprocess.shell rule at line 7, with static_path evidence and a network_request source. Its reachability field is network_source_modeled. The evidence follows the request read on line 5, assignment to host, f-string construction on line 6, assignment to command, and the execution sink on line 7. Those fields need careful interpretation. network_source_modeled means the analyzer recognized a network-input pattern. Authentication, deployment exposure, and actual exploit feasibility remain questions for human review. Using the Python API The command is named rce-path; the Python import is rce_path. The same scan is available through the public API: Python from rce_path import ScanConfig, scan if __name__ == "__main__": report = scan( "demo.py", config=ScanConfig(ai_enabled=False), ) print(report.summary) for finding in report.findings: print( finding.rule_id, finding.location.file, finding.location.line, ) for step in finding.steps: print( step.kind, step.location.line, step.description, ) The report and findings are typed Pydantic records. You can export them with report.to_json() or report.to_markdown() and use the structured fields in your own review workflow. Each finding carries a fingerprint, source category, evidence tier, source locations, and review state. Conditions, assumptions, and unresolved calls are available alongside the evidence. The report also records analyzed and skipped files, source content hashes, and coverage diagnostics. Fingerprints exclude the absolute checkout root and AI annotations, which helps compare the same finding across equivalent checkouts. They are location-based, so moving code can change a fingerprint. Where AI Review Fits An LLM can explain a finding, identify follow-up questions, or assess whether the available evidence supports an interpretation. In rce-path, that review is an optional annotation layer. Both provider implementations use a common evidence-and-review contract. Responses are validated, including references to evidence IDs. AI annotations do not rewrite the static finding or change its fingerprint. For a local Ollama review, install the extra: PowerShell pip install "rce-path[ollama]" Start your Ollama server and choose a model you already have installed: PowerShell rce-path scan demo.py --ai ollama --model YOUR_MODEL The default endpoint is http://127.0.0.1:11434. The package does not automatically download a model or fall back to a cloud provider. For OpenAI, install the corresponding extra and set OPENAI_API_KEY in your environment: PowerShell pip install "rce-path[openai]" rce-path scan demo.py --ai openai --model YOUR_MODEL --allow-remote-code YOUR_MODEL is a placeholder for an operator-selected model that supports the provider's required structured output. OpenAI and non-loopback Ollama endpoints require explicit permission to transmit code. The package sends bounded evidence slices and applies best-effort redaction of known secret patterns. Redaction can miss sensitive content, so review what you permit it to transmit. OpenAI requests use store=False, which is not a promise of zero retention. Review failures stay visible. A refused, unavailable, invalid, or budget-exceeded review is reported as unsuccessful rather than being presented as a completed assessment. Running It in a Review Pipeline For machine-readable output, scan a local checkout and save JSON: PowerShell rce-path scan ./checkout --format json --output findings.json You can also choose a findings policy: PowerShell rce-path scan ./checkout --fail-on static-path This fails the selected policy when the report contains a modeled static path. It does not mean the package has confirmed an exploitable vulnerability. The CLI uses three exit codes: the scan completed and the selected policy passed.the findings policy failed.an operational error, incomplete scan, or failed requested AI review occurred. The default policy is --fail-on none. Completion status takes precedence over the findings policy, so an incomplete scan cannot be treated as a successful clean scan. A repository-level rce-path.toml can configure rules, exclusions, and analysis budgets. Repository configuration cannot enable AI or grant permission to transmit code; those decisions require runtime opt-in. Current Limits rce-path uses bounded static analysis. Cross-function and cross-file flow, dynamic dispatch, closures, complex heap mutation, comprehensions, and precise exception handling are outside its current coverage. Branches are joined conservatively. A reported path can include conditions that are not jointly feasible at runtime. An unknown function is not automatically treated as a sanitizer, and a guard does not by itself prove an argument safe. For that reason, static_path means the analyzer established a path within its models. A sink_only finding identifies a sink whose input flow remains unresolved. Neither establishes remote code execution, and the package assigns no automatic CVSS score or exploitability probability. Read the coverage diagnostics alongside the findings. An empty completed report means "No modeled findings," not that the repository has been proved secure. What Comes Next The roadmap includes bounded local call summaries, additional framework and deserialization models, SARIF output, and separate human-approved validation workflows. These capabilities are planned for later phases and are not implemented in the current release. For now, I want rce-path to make a specific part of vulnerability review easier to inspect: how an input reaches an execution operation, what evidence supports that path, and what remains unknown. Start with a small file whose behavior you understand. Read the evidence steps, check the assumptions, and then apply the same workflow to a local repository. That is the most useful way to judge whether the package fits your research or code-review process. Project Links PyPIGitHubUsage and limitations

Every generation of technology arrives with the same expectation: this time, organizations will transform successfully, absorb the new technology, and reap the benefits. And sometimes they do. Yet after decades of technological progress, one pattern remains consistent. Organizations continue to spend valuable resources and dollars on technology initiatives that fail to achieve their intended outcomes. They fail because organizations attempt to automate what they don’t understand. They fail because the strategic purpose is not defined, change-readiness is not assessed, cultural barriers are not explored, and the outcomes expected are not clarified. Projects overrun. Adoption stalls. Strategic priorities drift. Employees become cynical. Competitive advantage slips away. A Pattern That Repeats With Every New Technology The history of business technology is a narrative of change. Mainframes gave way to personal computers. Enterprise applications consolidated isolated systems. The Internet created a global network. Cloud computing redefined infrastructure. Agile promised a better way to build software. Today, artificial intelligence promises another transformation. Technology is tangible. It can be purchased, implemented, configured, and deployed. Organizational clarity cannot. It requires conversation, alignment, judgment, and difficult choices. Those choices are often postponed because technology appears to offer a shortcut. Technology does not resolve organizational ambiguity. It exposes it. What Technology Can’t Decide For You An ERP system cannot decide how purchasing should work. A CRM system cannot define what constitutes a customer. Analytics cannot settle disagreements about which metrics matter. Artificial intelligence cannot compensate for unclear objectives or conflicting priorities. Every technology faithfully reflects the decisions that came before it. When those decisions are thoughtful, technology amplifies them. When they are confused, technology scales the confusion. This lesson has resurfaced in every technological era. The industry has repeatedly searched for technical solutions to organizational problems. We debated Waterfall versus Agile. On-premises versus cloud. Build versus buy. Low-code versus custom development. Today we debate copilots, agents, and autonomous systems. These are important decisions. None of them substitutes for clarity. Methodologies improve execution. Platforms expand capability. Artificial intelligence accelerates delivery. But no methodology, platform, or model can determine why an organization is changing, what success should look like, who owns the decisions, or how people will ultimately work. Clarity Before Technology This is why our engagement philosophy begins with clarity before technology. Clarity comes from ensuring that before the conversation turns to technology, software, infrastructure, or AI agents, leaders should have clear answers in the following dimensions: Purpose: Why are we making this change?Outcomes: What benefits will we gain when this effort succeeds?Ownership: Who has the authority to make decisions?Process: How does the work actually happen now? What will it look like after the initiative succeeds?Priorities: What matters most when tradeoffs arise?Tradeoffs: What are we willing to sacrifice to achieve something valuable?Governance: What constraints, risks, or obligations must shape our decisions?Culture: How much change can our teams absorb? How will we prepare them to succeed in the new environment? The Cost of Skipping This Step The rush to technology doesn’t answer these questions. It amplifies the lack of answers. The consequences of ignoring them extend far beyond an unsuccessful implementation. Organizations lose momentum. Investments fail to produce strategic advantage. Executive attention is consumed by avoidable challenges. Employees stop believing the next initiative will be different. Competitors, often with fewer resources, move ahead because their direction is clearer. The cost of ambiguity isn’t measured by a project plan. We’ve learned the lesson repeatedly that projects can be on time, on budget, and in scope, yet still deliver no business benefit. Lack of clarity in these dimensions is measured in lost opportunity and squandered advantage. Why AI Raises the Stakes And now, artificial intelligence has made the clarity discipline even more important. For the first time, organizations can prototype capabilities in days that once required months of engineering effort. Building has become dramatically easier. Experimentation has become far cheaper. This is an extraordinary opportunity, but it can also pose immense risk. The easier technology becomes to build, the easier it becomes to build the wrong thing. The faster an organization can act, the more certain it must be that it is acting on the right objectives, with the right governance, and in service of the right strategy. How Much Clarity Is Enough? Beginning by establishing clarity before turning to technology manages these potential pitfalls. This doesn’t have to mean that every initiative must begin with lengthy planning exercises. Because in truth, not every initiative requires months of analysis. Some decisions require only a few hours of focused alignment. But others deserve deeper examination. The discipline should always match the complexity and risk of the work. Navor’s engagement philosophy is to help organizations reach the level of understanding necessary to make the right investment decisions, execute them with confidence, and adopt change that lasts. Technology is one of the most powerful forces available to modern organizations. The potential of AI is monumental. The decisions firms make now require clear objectives, clear metrics, clear expectations. Successful transformation begins with clarity.

Have you considered what the defining infrastructure challenges of the next few years will be? If not, my interview with Srinivas Chippagiri may have you thinking more about how AI could change the way you manage infrastructure and what that means for building reliable, secure systems. But before we get into all of that, I always like to start by learning a little more about the person behind the expertise. So, let's kick it off with where Srinivas's journey began! Tell me what initially drew you into the field of technology. My journey began in software engineering, drawn to what I still think is the hardest question in the field: how to build software systems reliably at scale? I didn’t want to just build systems. I wanted to understand and document why they behave the way they do, which is what led me into research. Over the years, I’ve published peer-reviewed papers and book chapters on cloud security, Kubernetes performance, distributed scheduling, and applied AI. For me, engineering and research feed each other. Building at scale gives me the hard problems, and writing about them forces the clarity that makes the next system better. You mentioned that engineering and research feed into each other, which, for some developers, can make it a bit harder to navigate. When you're faced with a difficult technical problem, how do you typically approach finding a solution? I start by quantifying the problem before I touch a design. Who depends on this, what breaks if it fails, and what “good” actually looks like in numbers. Then I design for reuse rather than for the immediate fix. A lot of my research work comes directly out of that discipline. My work on Kubernetes network performance and on distributed task scheduling and load balancing using nature-inspired optimization began as real engineering problems that I then studied rigorously and published. That loop is my process: observe a hard problem in production, model it, measure it, and generalize it into something reusable. I trust measurement over intuition, and I try to leave behind a pattern others can apply, not just a one-off solution. I like that you focus on creating solutions others can reuse rather than just addressing the immediate problem. With that in mind, what's something you wish more people understood about cloud-native development? That “moving to the cloud” and “being cloud-native” are not the same thing. Many teams lift-and-shift a monolith onto managed infrastructure, watch the bill climb, and conclude cloud-native was oversold. The real win isn’t the platform. It’s designing systems as reusable, observable, governed building blocks, with security and compliance treated as engineering problems from day one rather than checklists bolted on after an incident. I’ve argued this in my research too. Several of my papers and book chapters look at cloud security and the organizational side of cloud transformation, because the hardest part of cloud-native isn’t the technology. It’s the discipline and architecture around it. You brought up the importance of building systems with security and compliance in mind from the start. Looking ahead, what emerging technology or trend has your attention right now, and why? Applied AI in infrastructure, meaning using AI not just as the workload but to operate and self-manage the systems underneath it. It sits right at the intersection of my two research threads: distributed systems and applied AI. My published work has gone deep on both sides. In a recent journal article, my co-authors and I designed an IoT-based AI framework for real-time health monitoring that pairs edge and fog processing with federated learning and differential privacy, so intelligence runs close to the data while privacy is preserved. On the systems side, I’ve studied scheduling and optimization. What fascinates me now is the reliability question. As we let intelligent systems make operational decisions, how do we keep them observable, safe, and accountable? That’s the same reliability-first instinct that has driven my entire career, applied to a far more autonomous stack, and I think it’s the defining infrastructure challenge of the next few years. It's interesting that reliability has remained a priority throughout your career, even as the technology evolves. What's one lesson you've learned along the way that's had a lasting impact on how you approach your work? That your highest-impact work is the work other people can build on. That reframed how I work. I now deliberately design and document for reuse, whether that’s a system architecture or a paper. I optimize for the multiplier, not just the deliverable, and the recognition I value most isn’t finishing something. It’s seeing my approach become a pattern others adopt. And now for my final question, which is usually my favorite: When you’re not working or thinking about technology, how do you like to spend your free time? A lot of my off time still points back toward the community. I’m active in the professional cloud computing community, and I mentor younger engineers and researchers, which I genuinely enjoy. Outside of tech, I like unhurried time with family and getting away from screens entirely. Travel and being outdoors help me return to hard problems with a clearer head. I’ve found the best architectural ideas rarely arrive at a keyboard. Check out more of his content here.

Modern data platforms are increasingly responsible for powering critical decisions, applications, analytics systems, and AI workloads. As organizations move more business processes onto data-driven architectures, the reliability of data pipelines becomes as important as the reliability of application services. A pipeline that fails loudly is usually easy to diagnose. A failed job, broken dependency, or unavailable service typically generates an alert and directs engineers toward a specific issue. The more difficult problems are the ones that do not interrupt execution at all. A pipeline can complete successfully while producing incorrect, incomplete, or misleading data. This is where data observability becomes essential. Traditional monitoring focuses primarily on whether systems are running. Data observability focuses on whether the data flowing through those systems remains trustworthy. It examines changes in structure, volume, freshness, distribution, and behavior to identify failures that traditional operational monitoring cannot detect. The Problem With Silent Data Failures Modern data ecosystems are rarely static. Enterprises often operate hundreds or thousands of pipelines connecting transactional systems, applications, event streams, warehouses, and analytical platforms. These pipelines evolve continuously as teams add new features, modify applications, and introduce new data sources. A database team may add a new column to a production table. An application team may change the format of an event payload. A third-party provider may alter the frequency or structure of delivered data. Each change may be intentional and technically valid, but it can create unexpected consequences downstream. Consider a customer analytics pipeline that receives transaction events from multiple systems. The pipeline may continue processing records after an upstream application changes a field format from a numeric value to a string. The ingestion process may not fail. The transformation jobs may still complete. Dashboards may continue refreshing. However, the business logic depending on that field may now produce incorrect results. The challenge is not detecting that a system is unavailable. The challenge is detecting that a system is operating while producing unreliable output. Detecting Schema Drift Before It Creates Downstream Issues Schema drift occurs when the structure of data changes over time. This can include adding or removing fields, changing data types, renaming attributes, or modifying relationships between datasets. Many organizations still rely on static schemas and manual documentation to manage these changes. While these approaches provide value, they often cannot keep pace with highly distributed environments. Data observability introduces automated schema monitoring that continuously compares incoming data against expected patterns. Instead of only checking whether a pipeline completed, observability systems examine whether the data structure remains consistent. For example, an observability layer can identify: A previously required field becoming consistently emptyA new field appearing unexpectedly in event streamsA data type changing from integer to textA significant reduction in the number of columns arriving from a source system The goal is not to prevent every schema change. Modern systems need to evolve. The goal is to identify changes quickly, understand their impact, and determine whether they represent expected evolution or an unintended failure. Understanding Data Drift Through Behavioral Changes Schema validation only examines structure. Many data quality issues happen without any structural change. Data drift occurs when data characteristics change over time. A dataset can maintain the same columns while its underlying patterns shift significantly. Examples include: A sudden increase in transactions from a specific regionA sharp decrease in average order valuesUnexpected changes in customer behavior patternsA growing percentage of missing valuesAbnormal distributions in machine learning features These changes are particularly important for AI and analytics systems. Machine learning models often assume that incoming data resembles the data used during development. When production data changes significantly, model performance can degrade without any visible infrastructure failure. Data observability helps teams establish baselines for normal behavior. Instead of defining every possible failure condition manually, systems can analyze historical patterns and identify statistically unusual changes. Combining Deterministic Rules With Intelligent Detection A common misconception is that data observability requires replacing traditional validation with machine learning. In practice, reliable data platforms need both deterministic checks and intelligent anomaly detection. Deterministic validation remains valuable because some conditions are clear. A customer identifier should not be null. A timestamp should follow an expected format. A transaction amount should not be negative. These rules provide strong guarantees for known requirements. However, not every failure can be captured through predefined rules. Unknown issues require systems that can recognize unusual behavior. Statistical and machine learning approaches can analyze: Record volume changesDistribution shiftsUnusual null patternsProcessing delaysUnexpected relationships between datasetsChanges in key business metrics The combination creates a layered defense. Rules catch known problems. Intelligent detection identifies emerging problems that engineers did not anticipate. Data Lineage: Understanding Where Problems Begin Detecting an anomaly is only the first step. The harder question is determining where the issue originated. Large organizations often have complex dependency chains. A single dashboard may depend on dozens of upstream tables, transformations, and source systems. Without visibility into these relationships, engineers may spend hours investigating symptoms rather than causes. Data lineage provides a map of how information moves through the organization. When an anomaly appears, lineage allows teams to answer important questions: Which pipelines depend on this dataset?Which upstream source introduced the change?Which downstream applications may be affected?When did the behavior begin changing? This turns incident response from manual investigation into a structured troubleshooting process. Building Effective Alerting and Root Cause Analysis Poor alerting can create another operational challenge. If every unusual data point generates an alert, engineering teams quickly experience alert fatigue. If alerts are too limited, important failures remain unnoticed. Effective data observability requires context-aware alerting. A sudden increase in records may not indicate a failure if it aligns with a known business event. A small change in a critical dataset may be more significant than a large change in a less important pipeline. Alerts should incorporate factors such as: Business importance of the affected dataSeverity of the anomalyHistorical behaviorDownstream dependenciesConfidence level of detection The objective is not simply generating more notifications. It helps engineers prioritize issues that require action. Designing Reliable Data Platforms for the Future As organizations adopt real-time analytics, AI applications, and increasingly complex data architectures, data reliability becomes an engineering discipline of its own. Reliable pipelines require more than successful job execution. They require confidence that the information moving through those pipelines remains accurate, complete, and useful. A mature data observability strategy combines multiple capabilities: Automated schema monitoringData quality validationStatistical anomaly detectionFreshness trackingPipeline performance monitoringData lineageIntelligent alertingRoot cause analysis The future of data engineering will not be defined only by how much data organizations can process. It will be defined by how confidently they can trust that data. Data observability provides the foundation for building that trust. By detecting silent failures before they become business problems, engineering teams can create data platforms that are not only scalable but also dependable.

Everyone is rushing to build AI agents right now, but GenAI integration really just puts a stronger spotlight on solid system design and backend reliability. An AI agent relies entirely on the APIs and data pipelines underneath it. If your backend drops events or double-processes payments during a network blip, even the best agent will just execute a broken workflow faster. To build resilient systems, you need to see how they break. Reading Kafka documentation on fault tolerance helps, but intentionally introducing network chaos shows how the underlying mechanics actually behave. In this post, we will set up a local testing environment using Spring Boot, Kafka, and Toxiproxy. We'll configure producer properties for topic-level idempotence and walk through common network simulation pitfalls that lead to duplicate data. 1. Setting Up the Chaos Test Architecture To test network issues accurately, stopping the Kafka broker isn't enough. We need to simulate realistic network conditions like latency spikes, packet drops, and bandwidth throttling. Toxiproxy acts as a proxy between our producer and broker to simulate these conditions. The Network Architecture Creating a Local/Dev Kafka Cluster We start by spinning up the Kafka cluster and Kafka UI using Docker. YAML # docker-compose-kafka.yml version: "3.8" #avoid this services: kafka: image: apache/kafka:latest container_name: kafka ports: - "9092:9092" environment: CLUSTER_ID: "4L62xdwCQKWvZs96A39vGA" KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: "broker,controller" KAFKA_CONTROLLER_QUORUM_VOTERS: "1@kafka:9093" KAFKA_INTER_BROKER_LISTENER_NAME: "INTERNAL" # Configure distinct internal/external routing ports KAFKA_LISTENERS: "INTERNAL://0.0.0.0:29092,EXTERNAL://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093" # We will bookmark this line, dear reader! KAFKA_ADVERTISED_LISTENERS: "INTERNAL://kafka:29092,EXTERNAL://localhost:19092" KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER" KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT" KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_LOG_DIRS: "/var/lib/kafka/data" volumes: - kafka_data:/var/lib/kafka/data kafka-ui: image: provectuslabs/kafka-ui:latest container_name: kafka-ui ports: - "9080:8080" depends_on: - kafka environment: DYNAMIC_CONFIG_ENABLED: "true" KAFKA_CLUSTERS_0_NAME: "local-cluster" KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: "kafka:29092" volumes: kafka_data: To spin it up, we use Docker Compose: docker compose -f "docker-compose -kafka.yml" up Both your local Kafka cluster and the kafka-ui should spin up. Note that KAFKA_ADVERTISED_LISTENERS includes the external port localhost:19092; we'll see why. A Local Proxy to Sit Between Kafka Broker and Producer We use Docker for this, too! YAML # docker-compose-toxiproxy.yml version: "3.8" # Don’t use this version - it’s deprecated services: toxiproxy: image: shopify/toxiproxy:latest container_name: toxiproxy ports: - "8474:8474" # Toxiproxy Management API - "8000:8000" # Example port exposed for a proxied application service - "19092:19092" networks: - toxiproxy-net toxiproxy-ui: image: buckle/toxiproxy-frontend:latest container_name: toxiproxy-ui ports: - "8080:8080" # We don’t use UI in this article environment: - TOXIPROXY_URL=http://toxiproxy:8474 depends_on: - toxiproxy networks: - toxiproxy-net networks: toxiproxy-net: driver: bridge docker compose -f docker-compose-toxiproxy.yml up Once running, create a proxy tunnel via the Toxiproxy CLI to listen on port 19092 and forward traffic to Kafka on port 9092: ./toxiproxy-cli create -l 0.0.0.0:19092 -u host.docker.internal:9092 kafka-proxy Note: We use the above CLI inside the toxiproxy container. So, you must use docker exec. In some versions, the CLI tool might not be included in the $PATH, and you might need to switch the current directory: Shell /go/bin # ls toxiproxy toxiproxy-cli If you're using this toxiproxy tool for the first time, you can refer to the official Toxiproxy documentation or this starter guide we created. 2. The Kafka Client Producer Code We create a basic application using Spring Boot 4.1 (though we do not have any version restriction until now) and use the following dependency: XML <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-kafka</artifactId> </dependency> We'll keep the Spring Boot service simple. We define a basic payload configuration and direct messages to topic T3 via KafkaConfig.java. Config: Java // KafkaConfig.java package com.soham.selfteaching.kafka.kafkadeepdive.config; import lombok.Data; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Configuration; @Configuration @Data public class KafkaConfig { @Value("${app.topic.name:T3}") private String topic; } Our application.yml is minimal, accepting defaults provided by the Kafka library. Properties files # application.yml server: port: 8082 spring: application: name: kafka-deep-dive kafka: bootstrap-servers: "localhost:19092" producer: acks: "1" # Required to bypass idempotence shield key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: org.springframework.kafka.support.serializer.JsonSerializer properties: request.timeout.ms: 3000 delivery.timeout.ms: 120000 enable.idempotence: false # We disable it first before we explore producer level idempotence The domain event is defined as a Java record. Java // EventDto.java package com.soham.selfteaching.kafka.kafkadeepdive.dto; public record EventDto(String id,String payload) { } Producer Code The core logic sits in ProducerService.java. We use CompletableFuture to handle delivery timeouts asynchronously without blocking thread execution. If the broker is unreachable initially, max.block.ms handles the immediate block on the main thread while background retries handle transient issues. Java // ProducerService.java package com.soham.selfteaching.kafka.kafkadeepdive.service; import lombok.RequiredArgsConstructor; import lombok.extern.slf4j.Slf4j; import org.springframework.kafka.core.KafkaTemplate; import org.springframework.kafka.support.SendResult; import org.springframework.stereotype.Service; import java.util.concurrent.CompletableFuture; @Service @Slf4j @RequiredArgsConstructor public class ProducerService { private final KafkaTemplate<String, Object> kafkaTemplate; public void sendMessage(String topic, String key, Object payload) { log.info("Attempting to send message to topic: {} with key: {}", topic, key); CompletableFuture<SendResult<String, Object>> future = kafkaTemplate.send(topic, key, payload); future.whenComplete((result, exception) -> { if (exception == null) { log.info("Message successfully delivered. Topic: {}, Partition: {}, Offset: {}", result.getRecordMetadata().topic(), result.getRecordMetadata().partition(), result.getRecordMetadata().offset()); } else { log.error("Fatal delivery failure for key {}. Exhausted delivery.timeout.ms.", key, exception); } }); } } 3. Configuring Topic-Level Idempotence During a network timeout, the producer cannot verify whether the message failed to reach the broker or if the acknowledgment (ACK) was lost on the return path. By default, the producer retries the request, which can produce duplicate records without proper safeguards. Exactly-Once Semantics (EOS) and topic-level idempotence address this by assigning a producer ID (PID) and sequence number to each message batch. If the broker receives a retried batch with a sequence number it has already processed, it ignores the duplicate write. Disabling Idempotence to Force Duplicates To demonstrate retry behavior, we can explicitly disable these safety settings in application.yml. Since Kafka 3.0, enable.idempotence defaults to true. To allow duplicates during testing, set acks: "1" and explicitly set enable.idempotence: false. Setting request.timeout.ms to 3000 (3 seconds) creates a tight window to trigger network retries during latency testing. 4. Running the Kafka Producer and Verify We run the Spring Boot application: Java @SpringBootApplication public class KafkaDeepDiveApplication { public static void main(String[] args) { SpringApplication.run(KafkaDeepDiveApplication.class, args); } } And hit a REST API (Hold on! It's a simple HTTP API which may NOT quite follow REST paradigms). curl -X GET http://localhost:8082/api/send Java @RestController @RequestMapping("/api") @Slf4j @RequiredArgsConstructor public class ProducerController { private final ProducerService service; private final KafkaConfig kafkaConfig; @GetMapping("/send") public String send() { long ts = System.currentTimeMillis(); Object payload = new EventDto("p1.t1." + UUID.randomUUID().toString(), ("Hello message " + i)); service.sendMessage(kafkaConfig.getTopic(), "k1", payload); return "Process took :%s ms".formatted(System.currentTimeMillis() - ts); } } In the case when idempotence is disabled, it's fair to expect that duplicate messages will arrive on a topic if a producer does not receive timely ACK from the broker leaders in alignment with the ack flag. But we do not yet see any duplicate messages at all. This is fairly explainable, since you're all set up locally, and there's usually no lost acknowledgment. Introduce Chaos to the Local Network Add 5000ms latency to the proxy. ./toxiproxy-cli toxic add -n false_positive -t latency -a latency=5000 -d kafka-proxy Send a message to the endpoint. After an API hit, wait for 2–3 seconds and remove the toxic Remove the Toxic While the Retry Is in Flight Instantly run this command to remove the toxic while the retry is in flight: ./toxiproxy-cli toxic remove -n false_positive kafka-proxy While this should have produced duplicates due to retries by the producer following a timed-out broker ACK, we find NO DUPLICATE MESSAGES even after repeating the process above many times. 5. Let Us Debug Our Disconnect Common Pitfalls During Network Injection Simply starting the proxy and adding latency doesn't always trigger duplicates as expected. Below are two common issues related to how Kafka clients handle metadata and connections. Issue 1: Advertised Listeners Bypass If Toxiproxy is running on port 19092 and a 5000ms latency is emulated, messages sent through the proxy might still complete in milliseconds. ./toxiproxy-cli toxic add -n false_positive -t latency -a latency=5000 -d kafka-proxy This occurs because of how Kafka advertises broker addresses during metadata discovery. If you look closely at the internal Kafka NetworkClient logs during startup, you will find out why: Plain Text brokers=[MetadataResponseBroker(nodeId=1, host='localhost', port=9092)] When the producer connects to port 19092 for initial cluster metadata, the broker returns its configured advertised listener (e.g., localhost:9092). The producer client caches this response and routes subsequent produce requests directly to port 9092, bypassing the proxy entirely. We must ensure that our KAFKA_ADVERTISED_LISTENERS explicitly broadcasts EXTERNAL://localhost:19092 so the producer continues routing traffic through Toxiproxy. If updated, clear existing Docker volumes (docker-compose down -v) to avoid using cached listener configurations. Issue 2: Initial Metadata Fetch Blocking If high latency (e.g., 15,000ms) is applied before sending any messages, the application may time out after 60 seconds with a TimeoutException: Topic T3 not present in metadata, rather than failing on delivery.timeout.ms. The delivery.timeout.ms timer only applies once a record enters the producer's accumulator buffer. If the client hasn't fetched topic metadata yet, the call blocks on max.block.ms (defaulting to 60 seconds) while attempting to retrieve metadata through the degraded network path. We apply the fix: Send an initial message without active toxics to populate the local metadata cache before injecting network latency. Finally With latency removed, the retried request reaches the broker successfully. Because acks=1 and enable.idempotence=false, the broker writes the payload again without verifying duplicate sequence numbers. Checking Kafka UI confirms two identical records are present in the topic partition. Simulating network edge cases directly demonstrates why configurations like advertised listeners, metadata timeouts, and idempotence defaults exist in modern Kafka clients. Now we get duplicate messages across a topic partition. Duplicate Messages We can now enable back idempotence flag to see that no duplicate messages are created repeating the above steps with enable.idempotence:true, ensuring that idempotence is preserved enable.idempotence= true only ensures idempotence across a partition only and not the entire topic. We shall explore it in a next article.

A full VPN is not always necessary. In addition, VPN connections may be unavailable because of restrictions imposed by a local or national internet provider. For the same reasons, a direct connection to an external SOCKS proxy may not work. Sometimes an alternative network route is needed not for the entire system, or even the entire browser, but only for specific addresses: an external API, an admin panel, documentation, a registry, a test environment, or another resource that is unavailable through the usual route. In this article, we’ll look at a narrower approach to solving this problem: To route traffic at the Chrome level, we use an extension suited to the task. In this case, we use FoxyProxy;Chrome sends only traffic for domains specified in FoxyProxy rules through the proxy;A local SOCKS5 endpoint runs on 127.0.0.1:1080;The endpoint is created using SSH dynamic forwarding;A PowerShell controller starts ssh.exe, monitors its state, and reconnects after a disconnection;An icon in the Windows notification area shows whether a local listener exists at the configured address and port;Launching the launcher again does not create a second controller or a second tray icon. I could not find an available tool that met my needs: one that provided this functionality, started quickly, was simple to use, and made its status easy to see. Existing solutions either have unnecessary settings or have complicated, unfriendly interfaces that require lengthy configuration before they can be used for this task. So, to create a local SSH tunnel on Windows, I developed my own PowerShell controller, launched through a VBS script. It displays a system tray icon, lets the user restart the tunnel, and automatically restores the connection when it drops. The complete source code for the PowerShell controller and VBS launcher, along with a sample configuration and installation instructions, is available in the repository: https://github.com/dtiumentsev/socks-via-ssh-tunnel-tray The controller is free to use under the MIT license. When Selective Routing Beats a Full VPN Suppose a developer needs access to a single API: api.example.dev Other websites, corporate services, local resources, and applications should continue to use the usual route. We also assume that outbound SOCKS traffic or VPN protocols may be unavailable. In this scenario, FoxyProxy matches a URL or domain to a proxy profile defined in its settings, while the proxy itself runs only locally: 127.0.0.1:1080 The goal is to manage the local tunnel as a tray application and be able to check its status at any time without extra steps. A local SOCKS5 proxy created through SSH dynamic forwarding is available at this address. Chrome connects to the local SOCKS5 endpoint, SSH encrypts the connection and carries it to the SSH host, and the remote side establishes a connection to the requested web resource. The setup looks like this: Chrome │ ├── Regular domains ──────────────────────────────► Direct connection │ └── api.example.dev │ ▼ FoxyProxy rule │ ▼ SOCKS5: 127.0.0.1:1080 │ ▼ ssh.exe with dynamic forwarding (-D) │ ▼ VPS / SSH host │ ▼ api.example.dev The advantage of this approach is its limited scope. The tunnel does not change Windows’ system-wide routing and does not affect applications that are not configured to use this SOCKS5 proxy. Prerequisites This article assumes that a VPS or other SSH host already exists and that the reader has legitimate key-based access to it. Preparing a VPS, installing sshd, and configuring server-side authentication are outside the scope of this article. The minimum requirements are: Windows 10 or Windows 11;The OpenSSH Client component installed;PowerShell 5.1 or later;A VPS or another SSH host accessible with a key;A private key corresponding to a public key authorized on the server;Google Chrome, Chromium, or another compatible browser;The FoxyProxy extension;Permission to use the SSH host and access the target resources. To check whether the built-in OpenSSH Client is present, run: Test-Path "$env:WINDIR\System32\OpenSSH\ssh.exe" A result of True means the path to ssh.exe exists. To check whether the private key is available, run: Test-Path "$env:USERPROFILE\.ssh\id_rsa" All environment-specific parameters are kept in a separate config.ps1 file. Here is an example configuration: $SshTunnelConfig = @{ SshPath = "$env:WINDIR\System32\OpenSSH\ssh.exe" KeyPath = "$env:USERPROFILE\.ssh\id_rsa" SshHost = "[email protected]" RemotePort = 22 BindAddress = "127.0.0.1" LocalPort = 1080 } The main script loads config.ps1 from its own directory using $PSScriptRoot. The repository contains config.example.ps1; the working config.ps1 contains the actual local settings. SSH Dynamic Forwarding as a Local SOCKS5 Proxy The solution is based on SSH dynamic forwarding, an OpenSSH client mode enabled with the -D option. A simplified command looks like this: ssh.exe ` -i "$env:USERPROFILE\.ssh\id_rsa" ` -p 22 ` -D 127.0.0.1:1080 ` -N ` [email protected] After a successful connection, the SSH client begins listening on a local TCP port 1080 at 127.0.0.1. Any application that supports SOCKS5 can use this endpoint. Importantly, the listener is bound to the loopback address: 127.0.0.1:1080 rather than: 0.0.0.0:1080 The first option is accessible only from the local machine. The second would make the SOCKS proxy accessible to other devices on the network, which is unnecessary for this scenario and increases the risk of unauthorized use. SSH Options The controller uses the following arguments: $arguments = "-4 -i `"$keyPath`" -p $remotePort -D ${bindAddress}:$port -N " + "-o ExitOnForwardFailure=yes " + "-o ServerAliveInterval=15 " + "-o ServerAliveCountMax=3 " + "-o ConnectTimeout=10 " + "-o StrictHostKeyChecking=accept-new " + "-o IdentitiesOnly=yes " + $sshHost Here is what the main options do: OptionPurpose-4Forces IPv4 for the connection to the SSH host-i <key>Specifies a particular private key-p <port>Sets the server’s SSH port-D 127.0.0.1:1080Creates a local SOCKS5 listener on the loopback interface-NDoes not open an interactive shell session or execute a remote commandExitOnForwardFailure=yesExits SSH if dynamic forwarding cannot be establishedServerAliveInterval=15Sends SSH-level keepalive packets every 15 seconds of inactivityServerAliveCountMax=3Closes SSH after three unanswered keepalive messagesConnectTimeout=10Limits the time allowed for a TCP connection attempt to the SSH hostStrictHostKeyChecking=accept-newAdds a new server’s key to known_hosts but does not silently accept a change to a previously known keyIdentitiesOnly=yesUses the specified key rather than trying every identity available through ssh-agent ExitOnForwardFailure=yes is especially important for automation. If the local listener cannot be created, SSH should not keep running as though the tunnel were available. This option makes the client exit when it cannot establish dynamic, local, remote, or tunnel forwarding. StrictHostKeyChecking=accept-new is a practical compromise for the first connection to a new SSH host. It does not disable checking a previously known host key: if the server’s key changes unexpectedly, the client should terminate the connection rather than silently trust the new key. Host key verification remains part of SSH’s security model. Why Starting SSH Once Is Not Enough Running ssh.exe -D solves only the initial problem: it creates a SOCKS listener. In practice, long-lived connections regularly encounter unstable conditions: A laptop goes to sleep and wakes up;The device switches between Wi-Fi, Ethernet, and mobile internet;The external IP address changes;A home router restarts;The SSH host becomes temporarily unavailable;The TCP session’s network state becomes stale;The ssh.exe process exits because of an authentication error, timeout, or network interruption. If SSH was started manually in a separate window, the user has to notice the interruption, determine that the listener is gone, and run the command again. That is inconvenient for a background tool. A small controller can handle four tasks: Start ssh.exe in the background.Check whether the local SOCKS port is listening.Wait for a specified interval after SSH exits, then try to reconnect.Show the current state through a system tray icon. The PowerShell Controller The complete script is in the repository, but its logic is fairly straightforward. The controller keeps a reference to the SSH process: $script:sshProcess = $null It also sets a reconnect delay: $reconnectDelaySeconds = 5 After starting ssh.exe, the controller does not immediately declare the tunnel operational. It checks for a local listener using Get-NetTCPConnection: Get-NetTCPConnection ` -LocalAddress $bindAddress ` -LocalPort $port ` -State Listen ` -ErrorAction SilentlyContinue This is better than relying only on the PID returned by Start-Process. A process starting successfully does not mean SSH has authenticated, established a session, and opened the local SOCKS listener. A simplified version of the check looks like this: function Is-Port-Listening { $listener = Get-NetTCPConnection ` -LocalAddress $bindAddress ` -LocalPort $port ` -State Listen ` -ErrorAction SilentlyContinue return ($null -ne $listener) } When the controller detects a TCP listener at the configured address and port, it shows a green icon. If the listener disappears, the icon turns red. Why Use a WinForms Timer? To keep the tray icon working, the PowerShell script runs a WinForms message loop: [System.Windows.Forms.Application]::Run() If the script exits, the icon disappears. The tray interface also stops responding to clicks and context-menu actions. The tunnel is checked periodically using System.Windows.Forms.Timer: $timer = New-Object System.Windows.Forms.Timer $timer.Interval = $checkIntervalMs In the current version, the check runs every five seconds. This reduces how often the port is polled; the exact time of the next connection attempt depends on when the timer fires after the reconnect delay expires. NotifyIcon is the standard Windows Forms component for displaying an icon in the Windows notification area. It works well for a small background controller with a minimal interface. How Reconnection Works The recovery logic is intentionally simple: ssh.exe exits ↓ The controller clears the saved PID ↓ It waits five seconds ↓ It starts another attempt if the port is free Here is the part that detects when SSH has exited: if ($script:sshProcess -and $script:sshProcess.HasExited) { $script:sshProcess = $null $script:nextStartTime = (Get-Date).AddSeconds($reconnectDelaySeconds) } Then, once the wait is over: if ( -not $script:sshProcess -and -not (Is-Port-Occupied) -and (Get-Date) -ge $script:nextStartTime ) { Start-Ssh } Before starting a new process, the controller checks whether the configured local port is free. It does not terminate a process that may have occupied the port: in the event of a conflict, the user must free the port manually or choose another LocalPort in config.ps1. Reconnection cannot begin until the SSH client realizes that the connection is no longer working. The following SSH keepalive settings help with that: ServerAliveInterval=15 ServerAliveCountMax=3 With ServerAliveInterval=15 and ServerAliveCountMax=3, SSH may detect an unresponsive server after approximately 45 seconds (15 × 3). The controller then waits another five seconds before attempting to reconnect. This is not a strict recovery-time guarantee. The actual result depends on the TCP connection’s state, the type of interruption, network latency, and the behavior of the SSH host. Still, this mechanism is much more reliable than passively waiting for the system’s TCP timeout. The System Tray Icon Keeping a PowerShell window open for a background tunnel is inconvenient. An icon in the notification area provides a simple interface: A green icon means a local TCP listener was detected at the configured ${BindAddress}:${LocalPort}; in the example configuration, that is 127.0.0.1:1080; A red icon means the listener is absent or the tunnel has been stopped; A context menu provides manual control over the connection lifecycle. Version 1 provides the following commands: Menu itemActionReconnect nowTerminates the current managed SSH process and initiates another connection attemptStop tunnelStops SSH and disables automatic reconnectionStart tunnelResumes the controller and starts another connection attemptExitStops SSH, removes the tray icon, and exits the PowerShell controller A manual stop is fundamentally different from a network failure. After Stop tunnel, the controller sets an internal stopped state and does not start SSH again. Otherwise, the script would restore the tunnel five seconds after the user stopped it. Why the Icon Is Not Updated on Every Timer Tick I do not consider reassigning NotifyIcon.Icon on every timer tick a good approach. Even when the state has not changed, Windows has to redraw the icon, which may look like flickering. In version 1, the port state is stored: $script:lastPortState = $null The icon changes only when the state changes: if ($alive -eq $script:lastPortState) { return } This is a small detail, but it matters for the normal behavior of a background tool. Preventing Duplicate Launches It is easy to launch a tray application again—for example, by double-clicking the launcher a second time. Without protection, this could produce two tray icons and multiple PowerShell controllers trying to manage the same local port. A named mutex prevents that: $mutexName = "Local\Denis_SocksTunnelTray_v1" $createdNew = $false $mutex = New-Object System.Threading.Mutex( $true, $mutexName, [ref]$createdNew ) if (-not $createdNew) { $mutex.Dispose() exit 0 } The first instance creates the mutex and owns it for as long as it runs. When launched again, the second instance detects the existing mutex and exits immediately. It does not create a second SSH process or add another tray icon. The Local\ prefix limits the mutex’s scope to the current Windows user session. That is sufficient for a desktop tool running in a normal interactive session. Launching Without a Black Console Window Launching a .ps1 file through a .cmd file opens a cmd.exe window. Even if the PowerShell process is minimized or hidden, the console launcher can still be visible briefly. A minimal VBS launcher is more convenient for users. It starts PowerShell in hidden mode and then exits. The Start-SocksTunnelTray.vbs file sits next to the PowerShell script: Option Explicit Dim fso, shell, folder, ps1, command Set fso = CreateObject("Scripting.FileSystemObject") Set shell = CreateObject("WScript.Shell") folder = fso.GetParentFolderName(WScript.ScriptFullName) ps1 = folder & "\Start-SocksTunnelTray.ps1" command = "powershell.exe -NoLogo -NoProfile -STA -WindowStyle Hidden -ExecutionPolicy Bypass -File """ & ps1 & """" shell.Run command, 0, False Here: -NoLogo removes the PowerShell startup banner;-NoProfile prevents the user’s PowerShell profile from affecting the process;-STA starts PowerShell in a single-threaded apartment model suitable for Windows Forms;-WindowStyle Hidden hides the PowerShell window;-ExecutionPolicy Bypass applies only to this process and does not change the system’s execution policy;shell.Run command, 0, False starts the process without displaying a window and does not wait for it to exit. The user double-clicks the VBS file and sees only the tray icon. Configuring FoxyProxy Once the tunnel is running, the browser needs to know which requests to send through the local SOCKS5 proxy. In FoxyProxy, create a proxy profile with these settings: Proxy type: SOCKS5 Host/IP address: 127.0.0.1 Port: 1080 Next, add a rule that applies the profile only to the required domain. For example: https://api.example.dev/* Or, if subdomains are needed: https://*.example.dev/* The key is not to enable proxying for all URLs when the goal is selective routing. The narrower the rule, the easier it is to control its effects and troubleshoot problems. DNS Through SOCKS If the extension offers a “DNS resolution through SOCKS5” setting, it makes sense to enable it for domains that should use the proxy route. The practical reason is simple: a standard SOCKS5 profile may forward the TCP connection, but the browser or operating system may resolve the hostname locally before opening that connection. Depending on the configuration, this means some network information does not follow the selected route. To test from the command line, you can use: curl.exe --proxy socks5h://127.0.0.1:1080 https://api.example.dev/ The h in socks5h indicates that the proxy, rather than the local system, should resolve the hostname. Verifying the Tunnel First, make sure Windows sees the listener: Get-NetTCPConnection -State Listen -LocalPort 1080 | Select-Object LocalAddress, LocalPort, OwningProcess Expected output looks like this: LocalAddress LocalPort OwningProcess ------------ --------- ------------- 127.0.0.1 1080 12345 Next, you can test proxying through an external service that reports the IP address of the outgoing connection: curl.exe --proxy socks5h://127.0.0.1:1080 https://api.ipify.org If the command returns the address of your external proxy host, the traffic is passing through the local SOCKS5 endpoint and the remote route. Diagnosing an Occupied Local Port If the tray icon remains in the reconnecting state, check which process is listening on the local port. The repository includes a separate diagnostic script for this: .\scripts\Check-SocksPort.ps1 The script checks the fixed TCP port 1080 and displays the LocalAddress, LocalPort, state, owning PID, and process name. If config.ps1 specifies a different LocalPort, change $port = 1080 in scripts/Check-SocksPort.ps1 to the required port. The script does not terminate processes or change the configuration. If an old ssh.exe process occupies the port, exit the old controller using Exit, or stop the process in a Windows session with the same privileges under which it was started. If another program occupies the port, do not terminate it automatically: change LocalPort in config.ps1 and in the FoxyProxy profile, for example, from 1080 to 1081. Testing Recovery After a Failure A minimal test: Start the controller using Start-SocksTunnelTray.vbs.Wait for the green icon.Open Task Manager.Find the ssh.exe process started by the controller.End the process.Confirm that the tray icon changes to the unavailable state.Wait for several check cycles.Check that a new ssh.exe process appears.Confirm that the listener at 127.0.0.1:1080 is available again. You can also test the following scenarios: ScenarioExpected behaviorTemporarily disconnect from the networkSSH detects the lost connection through keepalives, exits, and the controller begins reconnect attemptsClick Stop tunnelSSH exits, the listener disappears, and no reconnection occursClick Start tunnelThe controller starts SSH againClick Reconnect nowThe current SSH process exits, followed by another connection attemptLaunch the VBS launcher a second timeThe second instance exits silently because of the mutexHave another program occupy port 1080SSH cannot establish forwarding; the conflict must be resolved or another port selectedCheck a regular site without a FoxyProxy ruleTraffic should go directly rather than through the SOCKS profileCheck api.example.dev with a ruleThe request should use the local SOCKS5 profile The Indicator Checks the Local Port A green icon means there is a TCP listener at the configured address and port. The controller does not check whether the listener belongs to the ssh.exe process it started, nor does it confirm that the target site is reachable. If an old SSH process or another program occupies the port, check its owner using scripts/Check-SocksPort.ps1. The controller does not automatically terminate that process. SOCKS Dynamic Forwarding Is Primarily for TCP SSH dynamic forwarding works as a SOCKS proxy and is primarily intended for TCP connections. It is not a universal way to route every protocol on the system. If you need to proxy UDP, the entire network stack, or multiple applications without built-in proxy support, you will need a different type of solution. Security Recommendations A few useful practices: Bind the SOCKS listener to 127.0.0.1, not 0.0.0.0.Use a passphrase-protected SSH key if your workflow supports ssh-agent.Verify the SSH host’s fingerprint before connecting for the first time.Do not disable host key checking globally with StrictHostKeyChecking=no.Keep the list of FoxyProxy domains short and explicit. Repository Structure The socks-via-ssh-tunnel-tray repository is organized as follows: socks-via-ssh-tunnel-tray/ ├── CHANGELOG.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── SECURITY.md ├── Start-SocksTunnelTray.ps1 ├── Start-SocksTunnelTray.vbs ├── config.example.ps1 ├── icons/ │ ├── down.ico │ └── up.ico └── scripts/ └── Check-SocksPort.ps1 config.example.ps1 contains only a template: $SshTunnelConfig = @{ SshPath = "$env:WINDIR\System32\OpenSSH\ssh.exe" KeyPath = "$env:USERPROFILE\.ssh\id_rsa" SshHost = "[email protected]" RemotePort = 22 BindAddress = "127.0.0.1" LocalPort = 1080 } The user copies it to config.ps1 and fills in their own settings. The repository’s code is distributed under the MIT license. Conclusion An application using SSH dynamic forwarding provides a local SOCKS5 endpoint without requiring a separate proxy service on the client machine. Combined with FoxyProxy, it becomes a selective routing tool: the browser uses an alternative route only for explicitly selected domains, while the rest of the system continues to work as usual. This PowerShell controller handles the operational side of the task: It starts ssh.exe in the background;It shows the listener’s status through a tray icon;It restores the connection after the SSH tunnel drops;It distinguishes a manual stop from a network failure;It prevents duplicate instances when launched again;It does not leave a console window open. For a developer or infrastructure engineer, this is a simple, functional alternative to a system-wide tunnel when a managed route is needed for only a few resources rather than changing the routing of the entire computer, including in situations involving restrictions imposed by internet providers.

Every system that ingests data from someone else's API eventually writes a poller. It starts as fifteen lines in a cron job: hit the endpoint, compare against the last timestamp, insert what's new. It works immediately, which is the problem — it keeps appearing to work long after it has started losing records. I maintain an ingestion layer that polls public discussion APIs continuously, and the gap between "the poller is running" and "the poller is correct" took considerably longer to close than I expected. What follows are the four failure modes that caused real data loss, and the patterns that fixed them. None of this is exotic. It is the set of things I wish had been in one place before I wrote the first version. Failure Mode 1: The Fixed Interval The naive poller runs on a fixed schedule — every 60 seconds, every source, forever. This is wrong in both directions simultaneously. High-volume sources produce more records between polls than a single page returns, so you lose everything past the page boundary. Low-volume sources return nothing on the vast majority of calls, burning quota that the high-volume sources needed. With a shared quota across sources, the fixed interval guarantees you are under-sampling exactly where the data is. The fix is to let observed yield drive the interval. If the last poll came back full, poll sooner. If it came back empty, back off. TypeScript interface PollState { sourceId: string; intervalMs: number; consecutiveEmpty: number; } const MIN_INTERVAL = 30_000; const MAX_INTERVAL = 1_800_000; // 30 min function nextInterval(state: PollState, recordsFound: number, pageSize: number): number { // Page came back full — we are probably truncating. Poll aggressively. if (recordsFound >= pageSize) { return MIN_INTERVAL; } if (recordsFound > 0) { // Productive but not saturated. Ease toward the current rate. return clamp(state.intervalMs * 0.8, MIN_INTERVAL, MAX_INTERVAL); } // Empty. Back off geometrically, but bounded. const factor = 1.5 ** Math.min(state.consecutiveEmpty, 6); return clamp(MIN_INTERVAL * factor, MIN_INTERVAL, MAX_INTERVAL); } function clamp(n: number, lo: number, hi: number): number { return Math.min(hi, Math.max(lo, n)); } The condition worth highlighting is recordsFound >= pageSize. A full page is not a success signal — it is evidence that records may exist beyond what you fetched. Treat it as a truncation warning and either paginate immediately or shorten the interval. Systems that treat a full page as "great, got everything" lose data proportional to source velocity, which means they lose the most from the sources that matter most. After deploying adaptive intervals, our total request volume dropped by roughly 40% while records captured went up, because quota moved from silent sources to active ones. Failure Mode 2: The Watermark Boundary The standard incremental pattern stores the timestamp of the newest record seen and requests everything after it: SQL SELECT * FROM source WHERE created_at > :watermark ORDER BY created_at This loses records, for three independent reasons. Clock skew. The upstream timestamp is assigned by their server, not yours. Distributed writers mean created_at is not guaranteed to be monotonic in the order records become visible to the API.Visibility lag. A record can be written at T but not appear in query results until T+2s due to replication or index lag. Poll at T+1s, advance the watermark past T, and that record is never returned.Ties. Multiple records sharing the same timestamp, split across a page boundary, silently drop the remainder when you use a strict >. The fix is to rewind the watermark by a safety margin and deduplicate downstream: TypeScript const OVERLAP_MS = 120_000; // tune to observed upstream lag function queryWindow(watermark: number): { since: number; until: number } { return { since: watermark - OVERLAP_MS, until: Date.now(), }; } This deliberately re-fetches records you already have. That is the point — you trade a small amount of redundant bandwidth for the guarantee that a record has to be missed by two consecutive polls to be lost permanently. Size OVERLAP_MS From measurement, not intuition. Log the delta between upstream created_at and your own ingest time, take the 99th percentile, and double it. Ours sat around 40 seconds at p99, so 120 seconds gives real headroom without meaningful cost. Failure Mode 3: Deduplication That Isn't Overlap windows only work if dedup is correct, and ID-based dedup handles less than people assume. Upstream records get edited. A post whose body changes after ingestion keeps its ID, so an INSERT ... ON CONFLICT DO NOTHING on the ID silently discards the update. Meanwhile, some sources reuse or re-issue identifiers across deletion, so the ID alone is not always a stable key either. Hash the fields that constitute a meaningful change, and store both: TypeScript import { createHash } from "node:crypto"; function contentHash(record: SourceRecord): string { // Only fields whose change means "this record is materially different." // Volatile counters (score, comment count) are excluded on purpose. const canonical = JSON.stringify({ id: record.id, title: record.title ?? "", body: record.body ?? "", author: record.author ?? "", }); return createHash("sha256").update(canonical).digest("hex"); } SQL INSERT INTO records (source_id, external_id, content_hash, payload, first_seen_at, last_seen_at) VALUES ($1, $2, $3, $4, now(), now()) ON CONFLICT (source_id, external_id) DO UPDATE SET content_hash = EXCLUDED.content_hash, payload = EXCLUDED.payload, last_seen_at = now(), revision = records.revision + 1 WHERE records.content_hash IS DISTINCT FROM EXCLUDED.content_hash; Two details carry most of the weight. The WHERE clause means the overlap window costs nothing when nothing changed — no write amplification, no spurious updated_at churn, and downstream consumers watching for changes don't get woken up by re-fetches. And excluding volatile counters from the hash is what makes this viable at all. Include a score that ticks every few seconds and every single poll registers as a change, which defeats the purpose and turns your overlap window into a write storm. Failure Mode 4: Retry Storms and Undetected Gaps When a rate-limited API returns 429, the instinct is to retry. Retry immediately, and you extend the penalty. Retry on a fixed delay across N workers, and they resynchronize into a thundering herd that hits the limit together, forever. Most rate-limited APIs tell you what you need in response headers. Read them instead of guessing: TypeScript interface RateLimitState { remaining: number; resetAt: number; } function parseRateLimit(res: Response): RateLimitState | null { const remaining = res.headers.get("x-ratelimit-remaining"); const reset = res.headers.get("x-ratelimit-reset"); if (remaining === null || reset === null) return null; return { remaining: Number(remaining), // Some APIs send seconds-until-reset, others an absolute epoch. Normalize. resetAt: Number(reset) > 1e9 ? Number(reset) * 1000 : Date.now() + Number(reset) * 1000, }; } async function backoff(attempt: number, res?: Response): Promise<void> { const retryAfter = res?.headers.get("retry-after"); if (retryAfter) { const ms = Number(retryAfter) * 1000; if (Number.isFinite(ms)) return sleep(ms); } // Full jitter — see AWS Builders' Library on backoff. const ceiling = Math.min(60_000, 1000 * 2 ** attempt); return sleep(Math.random() * ceiling); } const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms)); Full jitter matters more than the backoff curve. Exponential backoff with a deterministic delay keeps clients synchronized; randomizing across the whole interval is what actually spreads load. Retry-After, When present, it is authoritative — honor it over your own calculation. The subtler problem is that these failures are invisible. A source that throws 429 for six hours produces no errors in your application logs if the poller catches and backs off correctly. It just quietly returns nothing, and your data has a hole in it that nobody notices until someone asks why a report looks wrong. Detect gaps explicitly rather than inferring them from error rates: TypeScript async function detectGaps(sourceId: string): Promise<Gap[]> { const { rows } = await db.query(` SELECT window_end AS gap_start, LEAD(window_start) OVER (ORDER BY window_start) AS gap_end FROM poll_log WHERE source_id = $1 AND status = 'ok' AND window_start > now() - interval '48 hours' ORDER BY window_start `, [sourceId]); return rows .filter(r => r.gap_end && r.gap_end - r.gap_start > GAP_THRESHOLD_MS) .map(r => ({ sourceId, from: r.gap_start, to: r.gap_end })); } This requires logging every poll attempt with its requested window and outcome — a table most people skip because it feels like overhead until the first time it tells you about four hours of missing data you would otherwise never have found. Feed detected gaps into a low-priority backfill queue that runs when quota is available. What Actually Matters If you take three things from this: Log the window, not just the result. Without a record of what range each poll requested, gaps are undetectable. This is the single highest-value thing to add to an existing poller.Overlap deliberately, dedup by content. A safety margin plus content-hash dedup converts "we might have missed something" into "a record must be missed twice to be lost."Treat a full page as a warning. It is the most common silent truncation point in production pollers, and it is one comparison to check for. None of these are difficult. They are just easy to omit, and their absence produces a system that looks healthy while quietly dropping records. I work on the ingestion layer at SignalMelo, which polls public discussion sources continuously — the patterns above came out of production incidents there. Disclosure: it's a commercial product I help build.

Intermittent test failures in a distributed CI/CD environment can slow down development and make failures harder to investigate. When an automated test suite fails on a local dev or staging machine, an engineer can view local logs, attach a debugger, or manually follow state changes. But when tests execute in a temporary container, AWS Lambda, or an agentless runner, post-mortem debugging is significantly more involved. In a traditional setup, a failed build only shows a generic assertion error (504, ElementNotFoundException, etc.). An engineer may need to manually sift through CloudWatch log groups for an approximate timestamp, query Splunk, or reason about state changes across downstream microservices to understand why the test failed. Once the log retention period expires or if the execution did not have a shared, identifiable correlation ID, the engineer has no further information about the cause of the failure. An enterprise test framework should save the test results and logs before the test runner shuts down. This article explores how to capture rich failure artifacts, stitch distributed logs (via correlation ID), and preserve retry states per-attempt. 1. Capturing Evidence Before the Ephemeral Runner Exits Capturing telemetry on failure seems trivial, but ephemeral environments introduce special exit modes which may skip cleanup routines: Assertion failures: The framework captures the exception; standard teardown hooks executeExecution timeouts: The process or handler is externally aborted; finalizers may be skippedHard kills (OOM/SIGKILL): The underlying container or VM is killed; no in-process code executed Your test harness needs to capture evidence in gracefully terminated failure modes and before hard timeouts Managing Runner Lifecycles AWS Lambda: A test runner deployed as a Lambda must check context.get_remaining_time_in_millis(). If it falls below a certain threshold (2,000 ms), it must terminate early, flush diagnostic buffers, and write metadata to S3. Your ephemeral /tmp storage defaults to 512 MB; holding large trace files or video recordings in memory would lead to out-of-memory (OOM) errors, so artifacts should be streamed directly to object store.AWS Fargate/ECS: When a task is stopped, it will receive a SIGTERM followed by a SIGKILL after a configured stopTimeout (30 seconds). Signal handlers in your test suite must capture SIGTERM and begin uploading artifacts.CI Build Agents (Bamboo/Jenkins): Agent-side logs are ephemeral and will be deleted when the agent's workspace is cleaned. Remote execution tasks should push telemetry to central storage (like AWS S3) and pass back the stable HTTP reference, not the local agent's file path. Pytest Hook Architecture In a Python setup, pairing a pytest_runtest_makereport hookwrapper with an autouse fixture guarantees that failure status is captured and attached to the test item before final teardown. Python # conftest.py import pytest @pytest.hookimpl(tryfirst=True, hookwrapper=True) def pytest_runtest_makereport(item, call): # Execute all other hooks to obtain the report object outcome = yield rep = outcome.get_result() # Store the report on the item for fixture access setattr(item, f"rep_{rep.when}", rep) @pytest.fixture(autouse=True) def failure_evidence_collector(request): yield # Code below executes during fixture teardown rep_call = getattr(request.node, "rep_call", None) if rep_call and rep_call.failed: execution_id = request.config.getoption("--execution-id", default="local") attempt = getattr(request.node, "execution_count", 1) # Extract environment details captured at run start env_metadata = { "test_name": request.node.name, "stage": getattr(request.node, "current_stage", "unknown"), "commit_hash": os.getenv("GIT_COMMIT", "HEAD"), "aws_request_id": getattr(request.node, "aws_request_id", None) } upload_failure_artifacts(execution_id, attempt, env_metadata) 2. Run IDs and Cross-Service Correlation Although capturing client-side logs is easy, if an integration test makes an HTTP call to a microservice architecture, the underlying defect may be found in downstream service logs. Without a unified identifier, correlating a client-side test failure with backend anomalies requires time-consuming manual timestamp matching. Propagating the Correlation Context To stitch frontend execution to backend telemetry, every outgoing request from the test runner must inject a unique tracking ID into transport headers. HTTP headers: Standardize on custom headers such as X-Correlation-ID or X-Run-ID. Initialize this ID once on the test client's session object so that every HTTP call automatically inherits itOpenTelemetry baggage: The standard W3C Trace Context (traceparent) requires a rigid 32-hex trace ID and 16-hex span ID format. CI build keys (e.g., PROJECT-JOB1-102) will not fit in standard trace fields. Instead, carry build context in the W3C baggage header (baggage: ci_build_key=PROJECT-JOB1-102,test_id=test_user_checkout)Async boundaries (SQS/SNS): HTTP headers do not automatically transition across message queues. When a test triggers an async workflow via AWS SQS, inject correlation metadata in the Message attributes. If there is an SNS in front of SQS, enable raw message Delivery in order to prevent SNS from wrapping the message payload in a JSON envelope (SQS attributes would need to be explicitly parsed by the consumer).Unowned platform services: Managed components like AWS API Gateway or Application Load Balancers (ALBs) will not process your application headers, but they will include their own tracking keys. API Gateway returns x-amzn-requestid in every response header, while ALBs preserve incoming X-Amzn-Trace-Id headers and record them in access logs. When an assertion fails, the test framework outputs a pre-formatted log query targeting the exact correlation key, enabling one-click diagnostic retrieval in CloudWatch Logs Insights or Splunk. MS SQL -- CloudWatch Logs Insights Query fields @timestamp, @message, service_name, log_level, error.type | filter correlation_id = "BUILD-8492-test_checkout_flow-att1" | sort @timestamp asc | limit 500 Fallback strategy: For legacy third-party microservices that do not accept or log correlation headers, configure the evidence collector to generate a time window query that is constrained to the specific UTC start and end timestamps of that specific test case execution. 3. Retries Kept Per Attempt Retries as implemented, say, by pytest-rerunfailures are often used to mitigate the risk of infrastructure-related failures, but such practice has its costs: retrying all tests that failed on the first attempt may mask intermittent bugs in the application under test. If a test fails on the first attempt but passes on the retry, the build will be marked as successful, even though the intermittent bug may still be present. If a test failed on attempt 1 due to a race condition but passed on attempt 2 because a resource freed up, overwriting attempt 1's evidence erases the bug's trace. Standard JUnit XML output often only records the final execution state, hiding the initial failure. Never write diagnostic files to a static path (e.g., s3://bucket/logs/test_checkout.log). Telemetry storage paths must include both the run identifier and the attempt counter. PowerShell s3://ci-test-evidence-bucket/ └── build-key=PROJECT-8492/ └── test_checkout_user/ ├── attempt-1/ │ ├── metadata.json │ ├── client_session.log │ └── failure_screenshot.png └── attempt-2/ ├── metadata.json └── client_session.log Once per-attempt isolation is implemented, diff analysis can be performed: JSON // S3 Path: build-key=PROJECT-8492/test_checkout_user/attempt-1/metadata.json { "test_name": "test_checkout_user", "execution_id": "PROJECT-8492", "attempt": 1, "status": "FAILED", "duration_seconds": 12.4, "http_status_code": 503, "target_host": "ip-10-0-4-12.ec2.internal", "failure_reason": "ServiceUnavailable: Database connection pool exhausted" } If attempt 1 fails with a 503 Service Unavailable from a particular host, and attempt 2 succeeds against a different host, the delta immediately indicates a node-specific configuration drift or pool exhaustion, not a code-level assertion flaw. 4. Keeping Artifacts Safe, Small, and Reachable Collecting unlimited telemetry rapidly leads to two major issues: credential leakage and excessive storage costs. Automated Redaction at Write Time Diagnostic artifacts (typically HTTP request/response dumps) often contain authorization tokens, cookies, or personal data. They must be redacted from memory before writing to disk: Strip headers that match Authorization, Cookie, X-Api-Key, or Set-Cookie.Remove signature tokens from URL parameters. Note that AWS S3 Presigned URLs act as credentials, and logging them directly would accidentally expose an ability to read/write to any S3 bucket the application has access to. Storage Optimization and Retention To keep storage costs at enterprise scale under control, enforce explicit lifecycle and collection rules: Selective retention: only store full-payload logs and screenshots if a test failed. Store only a short tail of a log output for passing tests.S3 Lifecycle Rules: configure S3 bucket lifecycle rules to delete failed test artifacts 30 days after creation, and passing test run artifacts 3 days after creation. It should be configured with either prefix- or tag-based policies.Do not put raw S3 presigned links inside JUnit XML or CI failure logs. Create links using a stable internal proxy/route that is capable of signing requests dynamically upon access. Presigned links created with temporary IAM role credentials will become invalid as soon as the web server process that issued them reloads. It would make sense to have it turned into a dead link if an engineer looks at a CI failure several hours (or days) after its occurrence. Failure Summary Surface Inject the direct artifact link and correlation ID straight into the framework's assertion exception message. Because virtually all CI engines parse and display the top-level exception string in build summaries, engineers can jump directly to the evidence: Python # Custom assertion failure formatter def raise_with_evidence(message: str, correlation_id: str, evidence_url: str): formatted_msg = ( f"{message}\n" f"--------------------------------------------------\n" f"Correlation ID : {correlation_id}\n" f"Evidence Bundle: {evidence_url}\n" f"--------------------------------------------------" ) raise AssertionError(formatted_msg) Architectural Limitations to Consider While evidence collection is made dramatically easier by the structured nature of the solution, there are a few boundary conditions that should be considered: Hard OS/platform kills: An AWS Lambda function reaching its 15-minute hard limit or worker node power loss will prevent in-process code execution. Secondary monitoring (container stdout monitoring via CloudWatch/Datadog) becomes a necessary alternative.Unowned external third parties: SaaS downstream APIs or gateway services will not instrument custom correlation IDs. In these scenarios, evidence would be strictly client-side HTTP request timestamp logging.Redaction leakage: Regex sanitization could miss non-standard secret key formats in unstructured response bodies. References W3C Recommendation: Trace Context - W3C Recommendation (2020).Luo, Q., Hariri, F., Eloussi, L., & Marinov, D.: "An Empirical Analysis of Flaky Tests" (ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2014).Google Testing Blog: "Flaky Tests at Google and How We Mitigate Them" (2016).AWS Documentation: AWS Lambda Context Object in Python & ECS Task Lifecycle.Pytest Documentation: Writing and using plugins & pytest-rerunfailures specification.

Most explanations of HTTP verbs in ASP.NET Core Web API stop at the dictionary definition: GET reads, POST creates, PUT updates, DELETE removes. That's true, but it doesn't explain why real enterprise APIs often bend these rules — and understanding why matters more in practice than reciting the definitions. This article walks through HTTP verb selection using a real HR/Payroll workflow scenario: submitting, updating, and removing leave requests. The Textbook Mapping In a typical Web API controller, the four core verbs map to CRUD operations like this: C# [ApiController] [Route("api/[controller]")] public class LeaveRequestController : ControllerBase { [HttpGet("get-emp-details")] public IActionResult GetEmployeeDetails(int finSlno, string empId, int createdBy) { // fetch and return data return Ok(employeeDetails); } [HttpPost("save-leave-request")] public IActionResult SaveLeaveRequest([FromBody] LeaveRequestDto request) { // create a new record return Ok(result); } [HttpPut("update-leave-request")] public IActionResult UpdateLeaveRequest([FromBody] LeaveRequestDto request) { // update an existing record return Ok(result); } [HttpDelete("delete-leave-request/{id}")] public IActionResult DeleteLeaveRequest(int id) { // remove a record return Ok(result); } } This is correct, and it's what most tutorials teach. But real enterprise systems — particularly HRMS and Payroll applications — frequently deviate from this pattern for reasons that only make sense once you understand what's actually happening to the data underneath. Why "Delete" Often Isn't Really DELETE Consider a leave request that an employee wants to withdraw. The textbook answer says: call [HttpDelete], remove the row from the database, done. In practice, most HR and Payroll systems don't want a "delete" action to genuinely erase the record. There are real, practical reasons for this: Audit trails – organizations often need to prove what happened and when, especially for anything touching attendance, payroll, or approvals. Permanently deleting a leave request destroys that history.Compliance – payroll systems in particular are subject to regulations that may require retaining records for a defined period, regardless of whether the user considers them "deleted."Traceability during disputes – if an employee claims a leave request was withdrawn incorrectly, having the original record (even if marked inactive) is the only way to investigate what actually happened. Because of this, many systems implement what's called a soft delete: instead of removing the row, the operation updates a status field: C# public class LeaveRequest { public int LeaveRequestId; public string EmpID; public int Status; // 0 = Pending, 1 = Approved, 2 = Withdrawn } A "delete" action becomes, in reality, an update to Status: C# [HttpPost("save-delete-role-mapping")] public IActionResult SoftDeleteLeaveRequest([FromBody] List<LeaveRequestDto> requests) { foreach (var req in requests) { req.Status = 2; // mark as withdrawn, don't remove the row } // save changes return Ok(result); } Because this is technically an update operation, not a true deletion, using [HttpPost] (or [HttpPut]) instead of [HttpDelete] is a defensible, common real-world choice — not a mistake or a shortcut. The verb should reflect what actually happens to the data, not just what the button on the screen says. Why POST Sometimes Wins Even for Genuine Deletes There's a second, more practical reason POST shows up in place of DELETE in real systems: payload complexity. [HttpDelete] is traditionally designed around simple identifiers, often passed in the URL: C# [HttpDelete("delete-leave-request/{id}")] public IActionResult Delete(int id) { // remove a single record by id return Ok(); } This works cleanly for deleting one record by one ID. But real operations are frequently batch operations — deleting or withdrawing multiple leave requests at once, each potentially needing additional context (who's performing the action, why, which workflow step it affects): C# [HttpPost("save-delete-role-mapping")] public IActionResult BatchDelete([FromBody] List<LeaveRequestDto> requests) { // process a batch of records, each carrying extra context return Ok(); } Passing a list of complex objects in a DELETE request body is technically possible but goes against how DELETE has traditionally been used and supported across tooling. POST, which has always supported rich request bodies, handles this case more naturally — which is exactly why it's common to see "delete" operations implemented as POST endpoints in real enterprise codebases. How This Looks From the Angular Side The verb choice on the backend directly shapes how the frontend calls it. In an Angular service, each HTTP verb maps to a specific HttpClient method: C# // GET - fetching data, no body needed getEmployeeDetails(empId: string): Observable<EmployeeDto> { return this.http.get<EmployeeDto>(`${this.apiUrl}/get-emp-details`, { params: new HttpParams().set('empId', empId) }); } // POST - creating a new record, or a batch/complex operation saveLeaveRequest(request: LeaveRequestDto): Observable<BaseResultModel<boolean>> { return this.http.post<any>(`${this.apiUrl}/save-leave-request`, request); } // POST used for a soft delete - batch of complex objects saveDeleteWorkFlows(models: LeaveRequestDto[]): Observable<BaseResultModel<boolean>> { return this.http.post<any>(`${this.apiUrl}/save-delete-role-mapping`, models); } Notice that both the create and the soft-delete operations use .post() from the Angular side, mirroring the [HttpPost] decorator on the backend. This consistency matters: if the backend soft-deletes via POST but the Angular service tries to call .delete(), the request will fail or hit the wrong endpoint entirely, since Angular's HttpClient.delete() doesn't send a request body the same way .post() does by default. Keeping verb usage consistent across frontend and backend isn't just a style preference — mismatched verbs are a common, easy-to-miss source of failed API calls that can take a while to diagnose, since the error often looks like a routing problem rather than a verb mismatch. A Note on Idempotency A related concept worth understanding alongside verb selection is idempotency — whether calling the same operation multiple times produces the same result as calling it once. GET, PUT, and DELETE are conventionally idempotent: fetching the same data twice returns the same result; updating a record to the same values twice leaves it in the same final state; deleting an already-deleted record (in systems that handle this gracefully) doesn't cause additional harm. POST is conventionally not idempotent — calling it twice can create two records instead of one. This matters directly for the soft-delete pattern discussed above. If SoftDeleteLeaveRequest is implemented as a POST that simply sets Status = 2, calling it twice by accident (say, from a double-clicked button or a retried failed request) produces the same end state — the record is still just marked as withdrawn. This is a case where a POST endpoint behaves idempotently in practice, even though POST isn't required to be idempotent by convention. It's worth being deliberate about this: an endpoint that changes a status flag is safer to retry than one that increments a counter or appends to a list, and understanding which category an operation falls into helps predict how it will behave under real-world conditions like network retries or double submissions. A Practical Rule of Thumb None of this means HTTP verb conventions don't matter — they're still a useful default. The practical guidance is: Use GET for anything that only reads data and has no side effectsUse POST for creating new records, and also for operations — including some deletes — that involve complex payloads or don't map cleanly to a single identifierUse PUT for straightforward updates to an existing, fully-identified recordUse DELETE for simple, single-identifier removals where a true, permanent delete is actually intended When you encounter an API that doesn't follow the textbook mapping exactly — like a "delete" endpoint implemented as POST — it's worth asking why, rather than assuming it's wrong. Often, as in the soft-delete case above, there's a real business reason: the data isn't actually being removed; it's being marked, and the verb reflects that reality rather than the UI label. Takeaway HTTP verb selection in a real Web API isn't just about matching a CRUD acronym to a decorator. It's about accurately representing what actually happens to the underlying data. A "delete" that's really a status update should probably be a POST or PUT, not a DELETE — and recognizing that distinction is a sign of understanding the system, not a shortcut around convention.