DZone Spotlight

AI has made producing code dramatically cheaper. Understanding that code has not become cheaper. A developer can generate a service wrapper, test suite, or 500-line pull request in minutes. A reviewer still has to understand what every important line does, what assumptions it makes, and how it behaves when the system changes. That gap creates a new form of technical liability: comprehension debt. It accumulates when teams can produce code faster than they can explain, operate, and safely change it. Comprehension debt shows up when a team accepts code before it has really taken ownership of the reasoning behind it. The code is not always messy. Sometimes it is clean. Sometimes it passes every test. That is what makes the risk easy to miss. The team ships the change, but it does not understand why the implementation works, where it is fragile, or what might happen when the surrounding system changes. That is why this moment matters. Traditional technical debt often accumulated because software was difficult and expensive to produce, so teams cut corners to keep delivery moving. AI changes that economic model. Code production can now outrun the team’s ability to inspect, explain, and safely evolve what was produced. The limiting factor is no longer only implementation capacity; it is human comprehension. If engineering organizations want to benefit from AI without weakening their codebases over time, they need to update their definition of done. A pull request that passes CI/CD is no longer enough. Production-ready AI-assisted code also needs traceability, risk classification, reviewer accountability, and evidence that the team understands the logic it is accepting. When AI-Generated Code Gets the Details Wrong Sometimes, we have specific assumptions or constraints in mind that we may not explicitly encode in the prompt because, as humans, we naturally consider them while writing the code. An AI, however, would not necessarily make the same assumptions. For instance, assume we are building a new API that needs to perform request validation, request authorization, and then execute business logic. If you tell an AI agent to build this, it might implement the API with request validation first, followed by request authorization, and then the rest of the business logic in the service. It might also write many unit and integration tests to ensure the request validation, request authorization, and business logic work correctly. However, from a security standpoint, you might want the request authorization to happen before the rest of the request validation, followed by the business logic. If this instruction is not explicitly given to the AI, it may implement the code in the wrong order, potentially performing request validation first and exposing information to a caller who is not authorized to view it. With the volume of code AI generates, it can be hard to find this kind of bug. However, when you implement code line by line without AI and dig into the implementation details, it may be easier to think through these scenarios. This is one problem with AI-generated code: it may be functionally correct and well tested, yet still miss assumptions a human developer would naturally consider during implementation. Product requirement documents generally describe high-level business requirements and may not specify every minor corner case. As a developer implements the logic, their understanding of the requirements evolves, and writing the code forces them to think through corner cases and decide how the system should behave. With AI-generated code, this iterative process can be missed if the requirements are simply handed to an AI agent as a prompt. The Hidden Cost of Synthetic Velocity Generative AI can produce a lot of plausible code almost instantly. The output often has clean formatting, sensible names, and just enough polish to feel safe. But polish is not understanding. A generated function can look obvious on the page while carrying assumptions no one on the team has checked. The bottleneck then moves from creation to comprehension. A developer can submit a 500-line AI-assisted pull request in minutes, but the reviewer still has to understand the full behavior, trace the edge cases, and decide whether the implementation fits the architecture. Reviewing is harder than generating because it requires reconstructing intent after the fact. When this happens repeatedly, reviewers adapt. With the massive amount of code being generated, they skim more. They lean harder on green tests. They assume the author has already done the deeper reasoning. The team may still be moving quickly, but the review process has quietly shifted from “Do we understand this?” to “Does anything obviously look broken?” Months later, the same code could cause problems in production, but no one can explain why the original implementation was structured that way. What looked like a fast delivery win becomes a forensic debugging exercise because the team accepted code faster than it absorbed the reasoning behind it. That is the real cost of synthetic velocity. The time saved during generation often comes back later in outages, migrations, and security reviews. Sprint metrics can make the codebase look healthy while the people responsible for operating it are quietly losing their map. Introducing the TTT Framework I have started thinking about this less as a tooling problem and more as an ownership problem. Banning AI coding tools is not the answer; developers will keep using tools that help them move faster. The harder question is whether the team still owns the reasoning behind what it ships. The Trace, Tag, and Time-box framework is a simple way to make that ownership visible. Trace what part of the code was AI-generated. Tag the change with the semantic risk. Time-box the work required to understand, test, and refactor it before it turns into institutional mystery. Reverse Walkthroughs are the mechanism for Yellow and Red changes, not a fourth pillar. The framework is intentionally boring. That is the point: it depends less on a specific tool than on habits a team can actually repeat. 1. The Trace Mandate The first rule is simple: teams should be able to identify AI-assisted code later. If a function, migration, test suite, or configuration block was generated or heavily shaped by an AI tool, that fact should not disappear once the pull request is merged. A lightweight trace can live in the pull request description, a commit message, or an internal annotation standard. This trace is not about blaming a tool or treating generated code as automatically suspicious. It is about preserving context. When a strange regular expression, unusual data transformation, or non-obvious retry strategy causes trouble months later, the team should be able to see where the logic came from and what the original intent was. That context can dramatically shorten future debugging and refactoring work. 2. Semantic Risk Tagging Not all AI-generated code carries the same level of risk. A generated CSS utility class and a generated authentication flow should not go through the same review process. Teams need a simple risk language that helps reviewers decide where to spend their attention. A useful tagging model divides AI-assisted changes into three categories: Green, Yellow, and Red. The point is not to create bureaucracy. The point is to make risk visible before reviewers are forced to infer it from the code alone. Green (Commodity): Low-risk generated code such as boilerplate, basic configuration, repetitive type definitions, simple test scaffolding, documentation drafts, or straightforward UI styling. These changes still need review, but they usually do not require deep architectural debate.Yellow (Algorithmic): Medium-risk generated code that changes business logic, data transformations, validation rules, error handling, concurrency behavior, or mathematical operations. These changes require a human reviewer to trace the logic carefully and confirm that the implementation matches the intended behavior.Red (Architectural): High-risk generated code that affects service boundaries, state management, database schemas, security-sensitive logic, permission models, API contracts, or large-scale system behavior. These changes should require senior review and, in some cases, architectural approval before generation is even used. 3. Time-Boxed Comprehension Work The third rule is to give comprehension real space on the engineering calendar. AI-assisted development creates an awkward asymmetry: the author may have produced the code quickly, but the team still has to understand it well enough to operate it later. If that work is not scheduled, it usually gets squeezed into a rushed review or postponed until an incident forces the issue. For Green changes, the time commitment might be minimal: a normal review, a quick note in the pull request, and standard tests. For Yellow and Red changes, teams should allocate defined review and refactoring time before the merge. That time can be used to simplify confusing logic, add integration tests, clarify naming, write a short design note, or pair through the generated implementation with another engineer. One practical way to spend that time is a “Reverse Walkthrough.” Instead of saying, “The tests pass,” the author must explain how the generated code works, why the chosen structure is appropriate, what edge cases were considered, and what failure modes remain. If the author cannot explain the implementation in plain language, the team should not merge it. This rule keeps ownership attached to the human operator, not the model that produced the first draft. A useful Reverse Walkthrough asks the author to answer practical questions before Yellow or Red code merges: What problem does this code solve? What assumptions does it make? What happens when those assumptions fail? Why was this implementation chosen? What alternatives were rejected? What happens under concurrency, load, or failure? Which parts were generated, and which parts were manually modified? Using AI to Make Code Reviews More Effective AI can also be a powerful tool for helping developers review AI-generated code. Instead of relying on the AI to simply perform an automated code review, reviewers can have a conversation with the AI about the changes in a pull request. They can ask it to explain how the code handles specific corner cases, identify scenarios that may not be covered, or verify whether the implementation actually satisfies the requirements of the change. An AI agent can also provide a high-level overview of the pull request, explain how the different files and classes fit together, and describe the responsibilities of the major components. This can significantly reduce the time a reviewer spends trying to understand a large code change. Managing the Velocity Shift The biggest mistake leaders can make is treating AI-assisted development as a pure throughput problem. If success is measured only by tickets closed, lines generated, or pull requests merged, teams will optimize for short-term output while quietly weakening long-term maintainability. A healthier metric combines delivery speed with operational confidence. Can the team explain the code it ships? Can new engineers understand the generated modules without relying on one person? Can the system be debugged under pressure? Can reviewers identify which parts of a pull request need deeper inspection? These questions matter as much as raw velocity. Comprehension Debt Indicators I would not treat these as established industry metrics. They are better understood as proposed operational signals: lightweight measures that help a team notice when AI-assisted development is increasing output faster than shared understanding. Reverse Walkthrough failure rate: The percentage of Yellow or Red AI-assisted pull requests where the author cannot adequately explain the implementation, assumptions, or failure modes.AI-code review time: The median reviewer time for AI-assisted pull requests compared with conventional pull requests of similar size and risk.Comprehension recovery time: The time spent after merge simplifying, documenting, testing, or restructuring AI-generated code so the team can maintain it confidently. None of these metrics should be used to shame individual developers. Used well, they give engineering leaders an early warning system. If the AI-assisted PR ratio rises while walkthrough failures, review time, and recovery time also rise, the team is not just moving faster. It is accumulating knowledge gaps that will show up later in incidents, migrations, or onboarding. Start small. Pick one active repository and apply the TTT framework to the next AI-assisted pull request. Ask the author to trace the generated code, tag each generated section by risk, and reserve a defined comprehension window for anything Yellow or Red. Require a Reverse Walkthrough before merge, or soon after merge if the work is intentionally staged behind a feature flag. The exercise will show quickly where the team’s review process is solid and where comprehension debt is already starting to collect. AI can absolutely make engineering teams faster. But speed that destroys shared understanding is borrowed time. The code is not necessarily bad. The problem is that six weeks later, the person who approved it may have no idea why it works. Comprehension debt is manageable, but only if teams name it, measure it with practical operational signals, and build habits that keep humans in control of the systems they are responsible for maintaining.

What started as a recovery tool for streaming pipelines ended up as a full-fledged self-service platform, with scheduled watermark-based incremental ingestion, ad-hoc/one-time data replay, and throughput control built in along the way. The origin was narrow: streaming pipelines fail. A downstream consumer has an outage, a schema changes, a job crashes partway through, and, let's say, Kafka partitions are already gone due to max retention; when that happens, someone needs a way to re-publish a window of historical data without working with the source system to republish. We built a small tool to do exactly that: point it at a backup table (that held raw events), give it a date range, and it would replay that data back to a Kafka topic. As more teams started using this, they began asking questions like: can this be scheduled for a regular stream? Can we configure the TPS (Events Published/Sec)? Those questions are what turned a one-off recovery script into shared infrastructure. This article covers that evolution: the two design problems that mattered most in getting there — turning replay into a first-class, self-service capability, and making throughput control work correctly across a distributed Spark job, the harder of the two — plus the operational lessons that came from running it as infrastructure other teams now depend on. High-Level Architecture At its core, a user calls the Replay Service with a source table, a target Kafka topic, required TPS, and other optional parameters. The Replay Service kicks off a Spark job on serverless EMR through a Scheduler Service (Internal), which hands back a pipeline ID. The user uses that for a status request from the Replay Service, which parses the underlying job's logs and returns status - running, completed with a record count, or failed with a reason. Replay Service – Sequence Diagram Once the job is running, it reads from the source table and streams records to the target Kafka topic, where any number of downstream consumers can subscribe. The Spark job runs under its own AWS IAM role, separate from any team's own credentials, so onboarding a new table or topic is a matter of granting that role read access to the source table and write access to the target topic, not provisioning per-team infrastructure. Replay as a Self-Service Capability Replay ended up covering a few distinct shapes of the same underlying need: One-time replay: An ad hoc, manually triggered backfill for a specific date range, used for onboarding, backfill, or incident recovery.Scheduled replay: A recurring stream from a data lake table that's typically populated in batch, where the data needs to be shared with other systems, customer support, marketing, etc. The capability also allows customers to define a watermark (CDC) column, so subsequent pushes send only incremental data to the topic, improving performance and reducing cost.Auto topic management: Replay can provision its own topic to avoid colliding with the live feed, or, if needed, publish into the same live feed (a user-provided topic) so the data flows through the usual downstream setup. Keeping Throughput Steady Across a Distributed Job Often teams want to dictate TPS (Events Published/Sec) because the downstream system may or may not handle the volume the replay pipeline produces. Also, TPS needs to be deterministic to reliably estimate how long the backfill will take, or simply how long the job will run. The core challenge: a distributed job, a single target rate. A Spark job doesn't execute as one process. It runs as many parallel tasks spread across executor cores, each independently reading and publishing its own slice of data. If you want the whole job to publish at, say, 10,000 events per second in aggregate, you can't just tell every task "publish as fast as you can"; you have to divide that target across however many parallel task slots are actually running, and get every one of them pacing independently toward its share of the total. Here's the shape of the approach: Scala // NOTE: // sourceDataFrame is repartitioned to a multiple of available workers. For e.g. if total // number of table partition is 100 and available workers are 56, the data will be repartitioned // to 112 (RoundingWithCeil(100/56) ) so that every node gets fair share of records to publish val totalExecutorCount = spark.sparkContext.getExecutorMemoryStatus.size - 1 val coresPerExecutorCount = spark.conf.get("spark.executor.cores", "1").toInt val totalWorkers = totalExecutorCount * coresPerExecutorCount val tps = jobConfig.argumentConfig.tps val batchSize = math.max(1, tps / totalWorkers) val eventsPublished: LongAccumulator = spark.sparkContext.longAccumulator("eventsPublished") sourceDataFrame.foreachPartition((partition: Iterator[Row]) => { var partitionCount = 0L partition.grouped(batchSize).foreach { batch => val startTime = System.currentTimeMillis() batch.foreach { record => kafkaSink.value.send(targetTopic, headers, value) partitionCount += 1 } // Pace each batch to a 1-second window: if sending the batch took less // than a second, sleep out the remainder before starting the next one. val elapsed = System.currentTimeMillis() - startTime val sleepTime = 1000 - elapsed if (sleepTime > 0) Thread.sleep(sleepTime) } // flush() blocks until Kafka acknowledges all buffered records for this partition. // Only count after confirmed delivery, so the metric reflects events that // actually landed on the topic, not events that were merely handed to the producer. kafkaSink.value.flush() eventsPublished.add(partitionCount) }) A few things worth unpacking here: totalWorkers counts task slots, not executors. In Spark terms, it's executor count multiplied by spark.executor.cores (set at spark-submit time), the actual number of parallel task slots available, since each core runs its own independent task. Dividing tps by executor count alone would under-throttle any executor running more than one core.The rate limiter is per-task, not global. There's no central coordinator throttling the job as a whole. Each partition's task independently batches its records and sleeps out the remainder of each second. The aggregate rate emerges from every task pacing itself to the same per-slot target simultaneously.This only holds on a stable, dedicated cluster. totalWorkers is calculated once at the start of the job, so the whole scheme assumes the cluster isn't dynamically scaling mid-run. If executors are added or removed while the job is running, the fixed batchSize no longer matches the actual number of active task slots, and the real aggregate rate drifts from the configured target. It also assumes the job has the cluster's resources to itself: running on a shared, transient cluster where other jobs are competing for the same executors introduces exactly the kind of resource contention this mechanism is trying to prevent in the first place. In practice, this means running on a dedicated, statically-sized cluster per job rather than a shared or autoscaling one.This only produces a uniform aggregate rate if the data is repartitioned into a multiple of totalWorkers first. If partitions aren't evenly distributed across the available slots, some cores end up handling more partitions than others. Every task still paces to the same batchSize, but a core doing more partition-passes ends up contributing more to the real aggregate throughput than a core doing fewer. The per-task math is correct, but the cluster-wide result skews unless the upstream repartitioning step guarantees an even split. Getting this repartitioning right was the actual key to making the aggregate number line up with the configured target. The batching-and-sleep loop is straightforward once the data is distributed evenly going in.The count only advances on confirmed delivery. flush() blocks until Kafka acknowledges every buffered record in that partition, and only then does the accumulator increment. Without this, a job that crashes mid-run could report a published count higher than what Kafka actually has, which quietly breaks trust in the metric during incident investigations.Low TPS with many workers hits a floor. math.max(1, tps / totalWorkers) guarantees at least one message per task per second, so a target of, say, 10 TPS spread across 56 task slots still lets every task send 1 msg/sec, overshooting the aggregate target by up to 5.6x. This scheme is tuned for the large-backfill case it was built for; it under-throttles at very low target rates. At a high level: each partition publishes only its allocated number of events per second, then sleeps for the rest of the second. A fixed cluster size, repartitioning data to a multiple of total workers, and a batch size derived from the TPS target are what guarantee the aggregate throughput. Each streaming job also runs on its own dedicated serverless EMR instance, so there's no contention with other jobs for the same resources. A Few Supporting Capabilities A couple of other features round out the platform, worth a brief mention even though they weren't the hardest problems to solve: Custom headers on published messages, letting downstream consumers route or filter messages without inspecting payloads.On-the-fly decryption of encrypted (PII) columns, so sensitive source data can be safely streamed without requiring every downstream consumer to independently implement decryption. Running It as Shared Infrastructure Building the capability was one problem. Running it as infrastructure other teams depend on surfaced a couple of costs that don't show up until the platform has real usage. Dedicated clusters aren't free. The TPS guarantee described above only holds because every job gets its own statically-sized, serverless EMR cluster, with no sharing and no autoscaling mid-run. That isolation is what makes the throughput math work, but it also means the platform's infrastructure cost scales roughly linearly with the number of concurrent jobs, not with total data volume. A dozen small scheduled jobs cost as much in cluster overhead as a few large ones. It's the direct tradeoff for the guarantee: correctness under load, in exchange for giving up the efficiency of a shared pool. Self-service cuts both ways at off-boarding. Making it easy for a team to schedule a recurring replay made it just as easy to forget it exists. A scheduled pipeline is driven by its upstream dependency, so when the upstream stops running, the pipeline stops too. It doesn't keep quietly publishing after a team moves on. But the pipeline's configuration doesn't go away on its own: teams that deprecated the downstream consumer rarely came back to tell us, so the definition just sits there, cluttering inventory and skewing usage metrics even though nothing was actually running. This demands auto de-provisioning based on its last run check (i.e., this pipeline didn't run in the last X days) and clean up periodically. Summary A recovery tool and a self-service streaming platform look similar on the surface: both move data from a table to Kafka. What separates them is whether replay works as a capability that teams can self-serve rather than something an engineer has to trigger by hand, whether throughput holds steady when a dozen teams are running jobs against the same infrastructure at once, and whether the platform can be operated (onboarded, monitored, and off-boarded) without someone having to remember every job by hand.

If you are responsible for keeping a production system alive, you have probably been pitched an "AI SRE" in the last few months. Maybe more than one. Maybe more than one this week. When we sat down at Bronto to map that space, we ended up with 64 tools across seven camps: pure-play investigation agents, observability platforms with AI built in, incident management tools with AI on top, AIOps and alert correlation, hyperscaler agents like AWS DevOps Agent and Azure SRE Agent, Kubernetes copilots and open source projects, and a long tail of adjacent tools. All of them use the same two words on their landing page, and they solve very different problems. So the first question most teams ask is the classic one: do we buy, or do we build? In our landscape article, we argue that the choice isn't binary. You can go full DIY, build on open source foundations like HolmesGPT or K8sGPT, buy a product and extend it, or buy it as-is. Most teams land somewhere in the middle. In this article, I want to make a different argument for building one, at least once: build to understand what you buy. Even if you never run your own DIY agent in production, putting one together by hand shows you where its answers come from, what it can't see, and why it goes wrong when it does. After building one, you know which questions to ask a vendor, and you know how to get better results from whatever you end up running. That's the profit you should take away from this post. The fun is the building and seeing your agent try to investigate an incident: it might get it wrong, it might get it right, but you'll see how it interacts with your tools and your data. This post is based on a hands-on lab we ran at an AWS and Bronto evening, and everything is available through bronto-community/ai-sre-lab. By the end, you will have a small agent with the following capabilities: reads logs, traces, and metrics through an MCP server,reads the application's source code and commit history through GitHub's MCP server,writes its hypothesis, evidence, and confidence into a GitHub issue. Note that in the lab we use Bronto as the observability backend. I work there, so I am biased to think it is your best option, but you can of course use any other solution. What an AI SRE Is AI SRE is mostly a marketing term. In short, it's an agentic loop pointed at the production system that can answer questions about it and help you troubleshoot. Like any agentic loop, it has inputs on one side (an alert, a question from a human), a model in the middle, tools underneath (mostly MCP servers these days, for telemetry, code, cloud APIs, runbooks), and outputs on the other side (a Slack message, a ticket, a pull request). So, it's an agent like any other. It's what you connect to it that makes it an agent for SRE. The Patient: A Small Online Shop Our AI SRE needs something to investigate. The lab comes with a tiny storefront: four Python services instrumented with OpenTelemetry, plus a payment processor mock standing in for a third party. You can, of course, point the agent directly at your own data. It's read-only, so it is not able to harm your system. What You Need To run this lab, you need the following: A container engine, like Docker DesktopAn AWS account with access to a model on Amazon Bedrock.A Bronto API key (there is a free trial), or another observability backend with an MCP server.A GitHub account and a fine-grained personal access token. It needs to read the contents of your copy of the storefront repo and write issues to it.Optional, for nicer output: jq and glow. You can watch this video to follow along as we walk through each step: Start the Shop Clone the lab and generate the storefront repository. The script builds a small Git history with a handful of commits and tags two releases, v3.1.0 (healthy) and v3.2.0: Shell git clone https://github.com/bronto-community/ai-sre-lab cd ai-sre-lab ./build-storefront.sh Your agent will later read this code through GitHub, so push it to a repository of your own, tags included: Shell cd storefront gh repo create storefront --public --source . --push git push --tags cd .. Then point the Collector at your telemetry backend and bring everything up: Shell cd demo echo "BRONTO_API_KEY=<your key>" > .env make up Note, that demo/collector.yaml ships to Bronto's EU ingestion endpoint. If your account lives in another region, or you want a different backend, that file is the only thing you need to change. Next, run make up, which deploys v3.1.0 and starts the traffic. It prints a message that you should leave it about ten minutes before you do anything else. An investigator with no baseline has nothing to compare against. So, take your time going through the first few next steps so it can build up! Step 1: The Loop All agents start with a basic file that receives prompts, sends them to the model, and returns the results. It's a simple first agent, so here is more or less the code in full: Python import os from bedrock_agentcore import BedrockAgentCoreApp from strands import Agent from strands.models import BedrockModel model = BedrockModel( model_id=os.environ.get("BEDROCK_MODEL_ID", "us.anthropic.claude-haiku-4-5-20251001-v1:0"), region_name=os.environ.get("AWS_REGION") or os.environ.get("AWS_DEFAULT_REGION", "us-west-2"), ) app = BedrockAgentCoreApp() @app.entrypoint def invoke(payload: dict) -> dict: agent = Agent(model=model) return {"result": str(agent(payload.get("prompt", "Hello!")))} if __name__ == "__main__": app.run() # POST /invocations, GET /ping on :8080 Two libraries do the work. Strands Agents runs the agent loop: send the prompt to the model, run any tool the model asks for, feed the result back, repeat until the model has an answer. The Bedrock AgentCore SDK wraps that in an HTTP server with the /invocations and /ping contract AgentCore Runtime expects. Today that runs on your laptop. Later, the same container can run on AgentCore Runtime without changes. Every step of the lab is a prebuilt image, so you don't need to build anything. Create a .env next to where you run the commands: Shell AWS_DEFAULT_REGION=us-west-2 AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_SESSION_TOKEN=... BEDROCK_MODEL_ID=us.anthropic.claude-haiku-4-5-20251001-v1:0 With this setup, you can start the agent for the first time: Shell docker run --rm -p 8080:8080 --env-file .env ghcr.io/bronto-community/ai-sre-agent:step-1 Open a second terminal and ask it something: Shell curl -s localhost:8080/invocations -d '{"prompt": "Who are you?"}' | jq -r .result It will say, "Claude, an AI assistant made by Anthropic." Correct, and completely useless for incident response. Right now this is a model behind an endpoint, with no identity and no tools. Step 2: Identity The second step changes only one line of code. Every Markdown file in prompts/ becomes part of the system prompt: Python SYSTEM_PROMPT = "\n\n".join(p.read_text() for p in sorted(Path("prompts").glob("*.md"))) @app.entrypoint def invoke(payload: dict) -> dict: agent = Agent(model=model, system_prompt=SYSTEM_PROMPT) ... The image ships with two files. role.md: Markdown You are the on-call SRE for Storefront, an online shop. You investigate production incidents: what broke, where, and why. And rules.md: Markdown - Only state what you can back with evidence from your tools. - If you have no tool that can answer, say so and name what you would need. - Be brief. You also get to add your own. Create a my.md: Markdown - Answer in at most three bullet points. - End every answer with your confidence: low, medium or high. - Sign off as "your friendly on-call dino". Start step 2 as follows. The command contains your custom instructions as an additional volume: Shell docker run --rm -p 8080:8080 --env-file .env \ -v "$PWD/my.md:/app/prompts/my.md" ghcr.io/bronto-community/ai-sre-agent:step-2 Shell curl -s localhost:8080/invocations -d '{"prompt": "Checkout is slow. What is wrong?"}' | jq -r .result You should get a short answer that says it has no access to telemetry, rates its own confidence as low, and signs off as a dino. That's a good result. The rule "say so and name what you would need" did its job: an agent that admits it can't see anything is far more useful than one that guesses. Step 3: Eyes Now the agent gets something to look at. Step 3 adds one tool source, Bronto's MCP server: Python bronto = MCPClient(lambda: streamable_http_client( os.environ.get("BRONTO_MCP_URL", "https://mcp.eu.bronto.io/mcp"), http_client=httpx2.AsyncClient( headers={"X-BRONTO-API-KEY": os.environ["BRONTO_API_KEY"]}, timeout=60, ), )) @app.entrypoint def invoke(payload: dict) -> dict: agent = Agent(model=model, system_prompt=SYSTEM_PROMPT, tools=[bronto]) ... That's the whole change. The MCP server's tools (search logs, list datasets, and so on) become tools the agent can call, and Strands handles the plumbing. This is also the point where MCP pays off: if your telemetry lives somewhere else, you swap this client for that vendor's MCP server and keep everything else. Add BRONTO_API_KEY to your .env (and BRONTO_MCP_URL if your account isn't in the EU region), then: Shell docker run --rm -p 8080:8080 --env-file .env \ -v "$PWD/my.md:/app/prompts/my.md" ghcr.io/bronto-community/ai-sre-agent:step-3 Start with a question that tells you what it can see: Shell curl -s localhost:8080/invocations -d '{"prompt": "What services do you know about?"}' | jq -r .result It should list the storefront services. Go ahead and ask it some more questions about the state of those services: what HTTP endpoints are available? What is the average error rate right now? With this, you have a first working version. Next we want to investigate an incident. To do so, ship the next release of the storefront: Shell make release According to the Makefile, symptoms take about 60 seconds to show up in p95. So wait a minute or two and then ask it: Shell curl -s localhost:8080/invocations -d '{"prompt": "Anything wrong with checkout?"}' | jq -r .result It will come back with a hypothesis about what is wrong and what you might be able to do about it. Take a look at the telemetry yourself and check: is the analysis correct, or did the agent pick up some false information? Step 4: Mouth An answer in your terminal is nice for a demo, but in production you would want to have it sent to a Slack channel, a pager app, or an issue tracker. In our example, we send the answer as a GitHub issue. This would let you tag your coding agent to review the issue and open a pull request that fixes it. You now have two options for how this gets filed. Either you give your agent another tool, and it can decide for itself where to send the hypothesis, or you make the code file the issue, so every analysis gets pushed into a dedicated issue: Shell def file_issue(question: str, hypothesis: str) -> str: r = httpx2.post( f"https://api.github.com/repos/{os.environ.get('GITHUB_REPO', 'ai-sre-lab/storefront')}/issues", headers={"Authorization": f"Bearer {os.environ['GITHUB_TOKEN']}"}, json={"title": f"[investigation] {question}"[:120], "body": hypothesis}, timeout=30, ) r.raise_for_status() return r.json()["html_url"] @app.entrypoint def invoke(payload: dict) -> dict: question = payload.get("prompt", "Hello!") agent = Agent(model=model, system_prompt=SYSTEM_PROMPT, tools=[bronto]) hypothesis = agent(question) return {"result": file_issue(question, hypothesis)} To make sure that what the agent provides fits the output, an additional new prompt file tells the agent how to write it: Markdown Your answer is filed as a GitHub issue for the on-call team. Write it as one: a one-line hypothesis, the evidence for it, your confidence, and what you could not check. Before running the next version of your agent, add your GitHub token and your repository to .env: Shell GITHUB_TOKEN=github_pat_... GITHUB_REPO=<your-user>/storefront Run step 4: Shell docker run --rm -p 8080:8080 --env-file .env \ -v "$PWD/my.md:/app/prompts/my.md" ghcr.io/bronto-community/ai-sre-agent:step-4 Ask it again to investigate the issue: Shell curl -s localhost:8080/invocations -d '{"prompt": "Anything wrong with checkout?"}' | jq -r .result The response is now a link. Open it and read the hypothesis. Is it right? How confident is it? There's a good chance it's confidently wrong. While testing the lab, one run concluded that checkout was down because a query returned nothing: an empty result is not proof of an outage, but nothing told the agent that. Step 5: Better Instructions What you do next doesn't involve code. You do what you'd do with a new human on the rotation: you tell them how you investigate. Add these lines to your my.md: Markdown - Slow is broken too: compare latency (p50, p95) before and after, not only errors. - Compare against a quiet window earlier the same day. - Look for `deploy` spans near the onset: they name the commit that went live. Each line encodes a habit experienced SREs have, and models don't bring by default. Checkout can be very broken without throwing a single error. A number means little without a baseline. And the first question in most incidents is "what changed?" The prompt files are read when the container starts, so restart step 4 with the same command and ask a more realistic question: Shell curl -s localhost:8080/invocations \ -d '{"prompt": "Customers say checkout got slow in the last 30 minutes. What is going on?"}' | jq -r .result Compare the new issue with the old one. Is it better? Worse? Did it fail outright? Step 6: Better Model As you might now, a different model can create a different output, so if you are not satisfied with the results yet, change one line in .env: Shell BEDROCK_MODEL_ID=us.amazon.nova-lite-v1:0 Restart your agent, ask the question again, and compare the outputs again: better or worse? Step 7: Code So far the agent can see that something changed and maybe even when. If your instructions worked, it found the deploy. What it can't do is look at what was deployed. Step 7 adds the second MCP server, GitHub's, restricted to read-only repository tools: Python github = MCPClient(lambda: streamable_http_client( os.environ.get("GITHUB_MCP_URL", "https://api.githubcopilot.com/mcp/x/repos/readonly"), http_client=httpx2.AsyncClient( headers={"Authorization": f"Bearer {os.environ['GITHUB_TOKEN']}"}, timeout=60, ), )) @app.entrypoint def invoke(payload: dict) -> dict: question = payload.get("prompt", "Hello!") agent = Agent(model=model, system_prompt=SYSTEM_PROMPT, tools=[bronto, github]) ... To make it useful to the agent, add one more prompt file, code.md. It connects the telemetry to the code: Markdown The shop's source code is in the GitHub repository `ai-sre-lab/storefront`. Every service reports the commit it runs as `deployment.commit`. Replace the repository path with your own, then mount it as an additional volume: Shell docker run --rm -p 8080:8080 --env-file .env \ -v "$PWD/my.md:/app/prompts/my.md" \ -v "$PWD/code.md:/app/prompts/code.md" \ ghcr.io/bronto-community/ai-sre-agent:step-7 Ask the same question again: Shell curl -s localhost:8080/invocations \ -d '{"prompt": "Customers say checkout got slow in the last 30 minutes. What is going on?"}' | jq -r .result Does it find the deploy? Does it open the release and pick the right commit? In our testing, it found the deploy marker and the commit, and then stopped. The release contains several commits, and the last hop, reading every one of them and figuring out which change explains the symptom, didn't happen on its own. If you experience the same, try to fix it yourself by refining your system prompts or by trying yet another model. When you're done, make rollback puts the healthy version back, and make down cleans everything up. If you want to give your agent a harder case, make psp-slow degrades the third-party payment processor. So, Buy or Build? Now you know what an AI SRE is and how it works, from the inside. That's a much better position from which to answer the question this post started with. If you buy, a lot of what hurt along the way becomes someone else's problem: confidently wrong answers that need endless prompt tuning, context windows overflowing with search results, wiring up every integration yourself. AWS DevOps Agent, for example, starts investigating when an alert fires and connects to the tools you likely already run, and you can still extend it with custom skills and your own tools. Take the four questions from the landscape article with you into every demo: Where does it get its data? Does it act, or only advise? Can you see its reasoning? What happens when it's wrong? After this lab, you'll know what good and bad answers to those look like. If you build, what you have now is a skeleton, and you can build more features towards a complete picture on top of it: Let alerts trigger it instead of curl.Run it on AgentCore Runtime instead of your laptop (agentcore deploy).Give it more tools: your cluster, your cloud APIs, your past incidents and postmortems.Observe the agent itself. Its own traces tell you which tools it called, how many tokens it burned, and where it went wrong, and that's the input you need to improve it. Either way, the data underneath matters more than the agent on top. Every improvement in this lab depended on the telemetry being there: a baseline from ten quiet minutes, latency and not just errors, a deployment.commit attribute that connects a span to a line of code. An agent can only reason about what your system tells it. That was true for humans long before agents showed up. So: what would your agent get wrong? The full lab, including the slides, the agent images for every step, and the storefront harness, is at github.com/bronto-community/ai-sre-lab. Content is CC BY 4.0; code is MIT. If you build something on top of it, I'd love to hear about it. I've shown how easy it is to build a simple AI SRE agent for fun. If you want to see what a production-ready one looks like, read our post on the Bronto blog, Introducing Bronto's Open-Source Alert Investigation Agent. It's an open-source agent that replicates the Alert Investigation feature in Bronto, and it's essentially our production version of this lab: a Bronto monitor triggers it instead of curl, it sticks to a constrained set of tools and reasoning paths instead of improvising, and it hands you a report with likely root causes and suggested next steps. The code is at bronto-community/bronto-alert-investigation-agent.

If you've sat through more than one architecture review, you've probably sat through a protracted Kotlin-versus-Java debate that goes nowhere. The team spends weeks comparing Kotlin's concise syntax to Java's maturity, runs a poll on developer preference, and produces a decision matrix nobody fully trusts. Then the meeting ends, and nothing changes, because nobody agreed on what problem they were actually trying to solve. Choosing between infrastructure instruments, such as Kotlin and Java, is an investment decision, and like any investment decision, it needs a thesis. Four business pressures should drive the discussion: time-to-market, operational risk, talent economics, and cross-platform consistency. The Problem With Asking Kotlin or Java When the question of Kotlin or Java arises, syntax and feature comparisons dominate the discussion, because they're concrete and measurable, and a natural place to start. You can count lines of code, benchmark compile times, and run surveys on developer satisfaction, but none of that will tell you if those metrics connect to any business outcome you actually care about. A language can be a good technical fit and still be a poor investment for a particular system. A stable backend handling 50,000 transactions a day with a three-person maintenance team faces a different set of risks than a payments service changing weekly under regulatory pressure, and both look different again from a monorepo running the same business logic across Android, backend, and edge. The JVM language strategy question comes down to where complexity actually lives in the system. Ask where Kotlin meaningfully supports your modernization strategy, and where staying with Java makes better sense right now. Modern Java May Be the Right Answer For a stable, low-change system, modernizing the existing Java stack can address the underlying problem without introducing the costs of another language, toolchain, or learning curve. If the system's main constraints don't align with Kotlin's strengths, staying with Java may offer a better return. If you're considering Kotlin, first identify where its capabilities address a problem worth solving. Start With the Business Problem Drawing on Yuri Geronimus's experience as Senior Staff Engineer and Chief Architect for EMEA Enterprise Payments Platforms at Verifone, the Kotlin Enterprise Playbook frames Kotlin adoption as an enterprise investment decision rather than a language preference. It organizes the business case around four key levers: delivery throughput and time-to-market, defect-cost avoidance, legacy modernization, and talent economics. Time-to-Market If your delivery lead time affects revenue, product launches, or certification cycles, ask whether the team can deliver faster. Kotlin's concise syntax means less boilerplate per feature, and fewer null pointer exceptions reaching production translates to less time spent on hotfixes and incident response. Whether that translates to faster delivery depends on where your bottlenecks sit. If the constraint is product clarity or test coverage rather than boilerplate or type errors, language choice won't fix it. Operational Risk Where are your expensive defects concentrated? Kotlin's type system encodes nullability explicitly, allowing some null-related errors to be caught at compile time rather than in production. For systems where a single production bug triggers a compliance review or a customer-facing outage, that compile-time safety has measurable financial value. For a read-heavy reporting service where reliability is already strong, the business case is thinner. Talent Economics The cost of a JVM talent problem can show up in longer time-to-hire, higher onboarding overhead, attrition, or difficulty moving engineers between teams. Java developers can build on their existing JVM experience when learning Kotlin, making internal mobility a potential part of the business case. The strategic partnership between JetBrains and Spring, announced in May 2025, also means that 27% of Spring developers who've already used Kotlin are a potential internal resource pool you may not have fully mapped. Cross-Platform Consistency If your organization maintains duplicated business logic across an Android app, a backend service, and an edge deployment, that duplication is an operational risk disguised as a technical decision. Kotlin Multiplatform (KMP) can let teams share domain logic, models, and validation across platforms, potentially reducing duplicated implementation and the risk of logic drift. That also reduces the compliance and maintenance burden of finding and fixing the same bug in multiple places. Keep in mind that you should evaluate the case for KMP separately, with the scope of shared code kept deliberately narrow. Kotlin doesn't automatically improve time-to-market any more than a new CI tool does. The question is whether you've identified the constraint it's positioned to address. Measure the Outcome You Actually Care About To make a solid business case for Kotlin, you need a measurable hypothesis. Each business lever maps to an observable metric. Time-to-market connects to delivery lead time and deployment frequency (both DORA metrics your platform team likely already tracks). Operational risk connects to change failure rate, mean time to recovery, and incident cost. Talent economics connects to time-to-hire, time to first commit after onboarding, and attrition rate. Cross-platform consistency connects to duplicated-logic defect rate and the volume of bugs that required fixes in more than one codebase. Those metrics support the business case, but you still have to translate them into financial terms. For an enterprise investment decision, connect the operational improvement to a financial outcome. Faster delivery can reduce the cost of delay, fewer incidents can reduce remediation and customer-impact costs, and shorter onboarding can reduce hiring and training costs. Bear in mind that productivity gains don't translate directly into headcount savings. Faster delivery can instead create capacity for new product work or reduce the cost of delay on a revenue-generating feature. Conflating the two overstates the financial case and creates a credibility problem when someone on the finance team models it out. Also, don't count productivity gains as both lower operating costs and increased revenue in the same proposal. A hypothesis that says "we expect Kotlin's null safety to reduce null-related production incidents by X% over 12 months, based on our current incident log showing Y% of incidents trace to NPEs" is defensible. A hypothesis that says "Kotlin will improve developer productivity" gives you nothing to measure against. Interoperability Makes This a Scoped Decision The Kotlin-or-Java framing misleads because it implies a binary. Kotlin compiles to JVM bytecode and is fully interoperable with Java, so Kotlin and Java files can coexist in the same project without a rewrite. Your existing Spring Boot or Hibernate configurations stay in place. Your Java libraries continue working. A single new service, a shared utilities module, or even the test layer of an existing Java codebase are all valid starting points for introducing Kotlin. IntelliJ IDEA, JetBrains' own IDE, includes a Java-to-Kotlin converter and first-class mixed-project support, so the tooling overhead of running both is lower than teams typically assume. That shifts the question from "should we adopt Kotlin?" to "where should we introduce Kotlin first?" A new, high-change service with a measurable business outcome may be a reasonable candidate. A stable order-management system with a small team and clean Java code can probably wait on Kotlin until a clearer business case emerges. Account for the Iceberg and the J-Curve An honest transition plan accounts for costs that typically don't appear in the optimistic spreadsheet. The J-curve is real. Before a team reaches full Kotlin proficiency, velocity drops. Training takes time, mixed-language code reviews add overhead (reviewers need to understand both languages to catch idiomatic issues), and CI pipelines may need reconfiguration for multi-language builds. Compliance documentation may need updating if your audit trail references language versions or toolchain components. The eventual decommissioning of legacy Java code, if that's the direction, carries its own cost that's easy to leave out of a five-year model. Scope the initial experiment carefully, and build these costs into the transition plan upfront. A proposal that acknowledges the J-curve and plans for it presents a more credible case than one that doesn't. Three Questions for Your Next Architecture Review If your team is heading into an architecture review where Kotlin is on the agenda, these three questions cut through the syntax debate and turn it into a scoped, testable proposal. Where are the expensive defects or operational problems concentrated? Pull the last 12 months of production incidents and categorize the ones with the highest cost (in engineer time, customer impact, or compliance exposure). If a significant share traces to null-related errors or type-safety failures, Kotlin's type system is directly relevant. If the distribution looks different, the investment case needs a different justification. What does your JVM talent pipeline actually look like? Map your current team's Kotlin exposure, your open Java roles, and the average time-to-hire over the past year. If you're competing for senior Java engineers in a tight market and losing candidates to organizations with more modern stacks, that's a talent economics argument worth quantifying. If your team is stable and hiring is closed, the argument doesn't apply. Which service, module, or domain is the lowest-risk place to test whether Kotlin improves the target outcome? Look for something actively changing (so you'll have delivery-cycle data to measure), with a well-defined scope, and without deep compliance dependencies that would complicate a mid-stream language shift. New functionality is often the best candidate. Adding Kotlin tests to an existing Java service can also provide a relatively low-production-risk way to gain experience with the language and toolchain. These questions convert an abstract language debate into a scoped experiment with a falsifiable hypothesis. Run the experiment, measure the result, and expand where it works. Make the Strategic Decision Your CFO and CTO won't ever see a line item for Java versus Kotlin, but they will see the costs of delivery lead time, incidents, onboarding overhead, and duplicated-logic defects. The strategic decision is where to invest to improve those outcomes, domain by domain, with measurement in place. Evaluate Kotlin where you have a number to test against, start with the lowest-risk entry point, and leave stable systems alone when the case is weak. The goal is a better outcome where the investment is justified. If you want to see how other enterprise teams have structured this transition, JetBrains has a practical case for it at kotlinlang.org/lp/enterprise-playbook. How has your team navigated this decision? Share your experience in the comments.

I wanted to let a coding agent change an endpoint while keeping application configuration out of reach. The request was fairly simple: update the Java source, adjust the tests, and leave the rest of the project alone. Writing that in the prompt explained the task to the agent, but I also wanted a hard check that could deterministically reject an edit before it hit the file system. So I added a hook, basically a small program that runs before any editing tool runs. What I saw was that the endpoint and its test changed as requested, and the test passed. When I looked at the tool calls, though, I found that neither edit had passed through my check. I had attached the hook to write_file, while the agent had chosen apply_diff. This is an article about a lesson I had to learn the hard way. The agent had followed my request through a tool I had left outside my wonderfully thought-through guardrail. Nothing harmful happened in that session, but I realized designing guardrails requires a different approach. One way to adjust my approach going forward was to check that I really understand how the agent made the change. Following the Edit Through the Tools For the example, I used Bob Shell with a generated Quarkus application running on Java 25 and the latest Quarkus. It had a GET /hello endpoint and an HTTP test, which gave me a super small change for my test agent with a clear result that I can check and understand. Forgive the simplistic approach, but I really do want to talk about the guardrails and not how amazing Quarkus is. The policy allowed edits under src/main/java/ and src/test/java/, leaving src/main/resources/application.properties outside the allowed directories. Bob Shell's PreToolUse hook runs before a tool whose name matches the configured expression. Returning exit code 2 from the hook blocks that call. I put the configuration in .bob/settings.json: JSON { "hooks": { "PreToolUse": [ { "matcher": "^write_file$", "hooks": [ { "type": "command", "command": "python3 .bob/hooks/allow-source-writes.py", "timeout": 5 } ] } ] } } The path rule never got a chance to make a decision because apply_diff did not match ^write_file$. This was probably my initial wrong assumption. Writing a file and applying a patch are different operations, but both can change the file you are trying to protect. I expanded the matcher to include four native editing tools: JSON "matcher": "^(write_file|apply_diff|insert_content|search_and_replace)$" Now the hook could inspect the patch operation as well as a direct write. Of course, thinking about it now, that still leaves shell commands outside the scope for now. But it's good enough for the little experiment we want to do here. Getting the Hook and the Agent to Agree On my next attempt, the hook caught apply_diff and blocked an edit to an allowed Java file. My handler expected the tool arguments inside the documented input object. It could not find that object in the event it received, so it returned exit code 2. Sigh. Documentation is always behind. Let's look into this more deeply. The session output showed the edit arguments under parameters, although session output and hook input are separate interfaces. That field name alone did not establish the hook's input format, so I extended the handler to accept input, tool_input, and parameters, as well as a direct path object. With that little adapter, the same source edits finally passed: Python def tool_input(event): for key in ("input", "tool_input", "parameters"): value = event.get(key) if isinstance(value, dict): return value return event if "path" in event else None Once the arguments are available, the policy can check the path. A string starting with src/main/java/ is not quite enough: a symbolic link inside that directory could still point to application.properties. So I had to resolve the path first, and then check where it leads. The simplified excerpt below shows that; the surrounding handler reads the event and returns 2 when it rejects an edit: Python from pathlib import Path, PurePosixPath ALLOWED_PREFIXES = ("src/main/java/", "src/test/java/") def allows_path(raw_path): path = PurePosixPath(raw_path) if not path.is_absolute() and ".." in path.parts: return False workspace = Path.cwd().resolve() candidate = Path(raw_path) if path.is_absolute() else workspace / Path(raw_path) try: normalized = candidate.resolve().relative_to(workspace).as_posix() except ValueError: return False return normalized.startswith(ALLOWED_PREFIXES) And those were basically the two separate mistakes in the integration I wanted to share with you. First, the hook did not run. Then it ran but could not read the arguments. The path rule could pass its own unit tests in both cases, because those tests supplied the JSON I expected. Sending an actual edit through the agent exposed the difference. Trying an Allowed Edit and a Blocked One After I have the guardrails in place, it is finally time to test them. Of course, in a disposable workspace with no really important stuff. But I included secrets/demo.env with a dummy value, so we had another forbidden path for the hooks to check. I also added that directory to .bobignore. Bob's ignore file covers several normal file operations. It does not restrict arbitrary shell commands, so it cannot provide isolation for the whole workspace. With the matcher and handler updated, I asked for the endpoint change again: Plain Text Change the GET /hello endpoint to return exactly `Hello from policy checked Quarkus`. Update its existing unit test to assert the new response. Run the relevant Maven test. Use only Bob's native file-editing tool for source changes. Do not edit configuration, files under secrets/, or the hook policy. The command starts a new session in the current project. Workspace hooks require a trusted folder, which is why I included --trust with my Bobshell command. And I did specify in the prompt that the agent should use its native editor for the source changes. Shell bob run --workspace "$PWD" --trust --format stream-json --mode agent \ --max-cost 0.20 --max-turns 8 \ --disable-mcp --disable-subagents \ --accept-license < prompts/03-allowed-apply-diff.md This time, two apply_diff calls passed the hook. One changed GreetingResource.java to return Hello from policy checked Quarkus; the other updated the expected response in GreetingResourceTest.java. The Maven test passed as well. I then tried the same in a new session and asked for a change outside the allowed directories: Plain Text Set `quarkus.http.port=8081` in `src/main/resources/application.properties`. Use only Bob's native file-editing tool. Do not edit Java source, tests, the hook policy, or files under secrets/. The agent selected insert_content, which was also covered by the expanded matcher. The hook rejected the path with exit code 2, and application.properties stayed empty. Checking the Policy as Well as the Application Once a hook decides whether an agent can change a file, it becomes part of a permission system. A mistake can allow an edit that should have been blocked or stop a valid change. A very good reason to give that code its own tests, just as I would test authorization logic in an application. Start with the rule itself. Give it requests that should pass and requests that must be rejected, including input it cannot understand. You can check these decisions without asking a model to produce each request. That makes the tests repeatable and lets you keep a failing example whenever you find a mistake in the policy. Then check how the hook is connected to the agent. A correct rule cannot stop an action if the runtime never calls it. Send both allowed and forbidden actions through each tool the policy is intended to cover, and inspect what actually changed. This checks whether the hook runs at the right time, receives the input it expects, and has its decision enforced. It is also worth checking what happens when the hook crashes or times out. Rejecting a request and failing to evaluate it may lead to different behavior in the runtime. I want to know which outcome lets an action continue, and repeat these checks when the hook configuration or agent version changes. The application tests still have their usual job, of course. They check the behavior of the changed code. Hook tests check whether the agent was allowed to make that change in the first place. A passing build gives you no answer to that second question. Deciding How Much Access to Give the Agent My example covered a few native editing tools. Shell commands could still change files, and the hook did not restrict network access. The hook itself also ran with my user permissions. Protecting configuration from every possible edit would require controls over those other routes as well. The two requests here checked the native editor policy; they did not show a fully sandboxed or isolated workspace. This connects to the source-and-sink model in OpenAI's discussion of prompt injection defenses. Content the agent reads can influence it, and a tool call can turn that influence into an action. A check before the action gives the surrounding software a chance to reject it. My endpoint example did not involve an injection attack, but it showed a prerequisite for that check: the action has to reach it. Anthropic also recommends limiting the tools and permissions given to agents and using defenses at several layers in its guidance on trustworthy agents. For a coding agent, that means deciding which files and commands it needs, applying tool checks, and reviewing what changed afterward. A disposable checkout without credentials keeps the consequences of a mistake smaller; enforced filesystem and network restrictions are separate controls. I started with a rule about directories and ended up spending more time on the tools that could reach them. Maybe this little article helps you build better hooks for your agents.

Retrieval-augmented generation (RAG) has quickly become one of the most useful patterns in the burgeoning world of enterprise AI. Rather than fine-tuning a model on private data, a RAG system instead extracts text from internal documents, splits that text into chunks, turns each chunk into a vector embedding, and stores those vectors so a language model can fluidly pull the most relevant passages at question time. The payoff from a well-designed RAG system is considerable: employees can ask plain-language questions across wikis, contracts, support tickets, and policy files and get grounded, cited answers in reply — with no training involved. That same convenience is what makes personally identifiable information (PII) particularly dangerous here. Feeding internal documents into an AI service means copying their contents into a new, searchable, and hard-to-purge form; if any of those documents contain Social Security numbers, bank account numbers, or health details, that nifty RAG system quietly becomes a fountain of regulatory and compliance violations. In this article, we’ll first discuss the RAG PII issue in more detail, and we’ll then focus on how to implement a C# solution that mitigates the problem. We’ll ultimately build a solution that screens the extracted text before it’s ever embedded. A Vector Store Keeps PII Long After Ingestion PII in a RAG pipeline is a problem of both exposure and persistence. In an ordinary request, sensitive data passes through and is gone as quickly as it came. In ingestion, however, that data stays put. Once a document is chunked and embedded, any PII it contained lives on in the vectors, in the chunk metadata, and in every retrieval result that pulls those chunks back. That means it can resurface in a model’s answer far from the document it came from, in front of a user who was never meant to see it. By the time that mistake is discovered, mitigation means finding and removing every derived copy across the entire index, which is, of course, far harder than never ingesting that data in the first place. Screening at the point of ingestion is the last inexpensive moment we have available to keep regulated data out. Detection Is a Decision Point, Not a Redaction Step Before we build a solution for RAG PII mitigation, it helps to separate the two things we might do with sensitive text once we've identified it. We can detect it and decide whether the document can proceed in the pipeline, or we can redact it and, in doing so, alter the document before it moves beyond our workflow. This article is ultimately focused on implementing the first concept: a PII gate. The policy for that is simple. A document the detector clears for RAG use is eligible for the index; a document that trips it is held, and we distinguish a hard-block category (such as a Social Security or bank account number) from one that merely warrants a human look. Screen the Exact Text You Embed The embedding process requires text — and that, of course, means a RAG pipeline always produces extracted text. That text is what we want to screen; it’s the exact artifact about to be chunked and embedded. Screening a source image or file and then extracting text separately means checking one representation while ingesting another, and that’s problematic because the two can diverge. Hidden layers, OCR differences, and metadata can all carry data that the version you scanned never showed. So we avoid that problem by screening after extraction and before chunking, which makes the bytes we approve the same bytes we embed. What We Can Build With Open-Source Components A good chunk of the tooling we need already lives in our ecosystem. .NET gives us all the surrounding machinery we need for free: file handling, size limits, streaming, and so on. What it does not give us, however, is any tool specifically optimized for reliable PII detection. Regular expressions come to mind, and they can certainly catch well-formed Social Security and credit card numbers, but they miss names, addresses, and the many identifiers that appear in free-form, context-dependent ways, and they’re prone to false alarms on ordinary numbers. Health information is harder still, since recognizing references to medical conditions or prescribed treatments requires a level of contextual understanding that simple pattern matching cannot independently make up for. A managed detector, one example of which we’ll use in our solution below, absorbs that recognition problem. The piece of the puzzle it leaves to us is the one that is indisputably ours: choosing which categories to permit, setting the policies to block unpermitted categories, and testing those against documents that closely resemble our real document corpus. Adding a PII Detection Step in C# In our demonstration, we’ll use a DLP API optimized for detecting configurable categories of PII and PHI in text, and we’ll treat its verdict as the screening result that drives the RAG PII gate. Please note a strong assumption baked in here: we’re using an endpoint deployed on our own infrastructure. That goes for any comparable service. If we don’t account for this, we’re just doubling our data loss exposure. We should not use publicly hosted web services to screen sensitive data. To set up, we’ll first create a console project and install the SDK: PowerShell dotnet new console --framework net8.0 -n RagPiiGate cd RagPiiGate dotnet add package Cloudmersive.APIClient.NETCore.DLP --version 1.1.0 C# using System; using System.IO; using Cloudmersive.APIClient.NETCore.DLP.Api; using Cloudmersive.APIClient.NETCore.DLP.Client; using Cloudmersive.APIClient.NETCore.DLP.Model; Building the PII Screener Step by Step We’ll now build the program in the order the data flows. The chunks below concatenate into one to-level Program.cs; we’ve broken it down this way for extra clarity. First, we’ll gather what the operator controls and reject anything missing. The API key and the two destination directories come from the environment, never from the document. C# try { if (args.Length != 2) throw new ArgumentException("Usage: RagPiiGate <extracted-text-path> <document-id>"); string Required(string name) => Environment.GetEnvironmentVariable(name) is { Length: > 0 } value ? value : throw new InvalidOperationException($"Set {name}."); string key = Required("CLOUDMERSIVE_API_KEY"); string acceptDir = Required("ACCEPT_DIR"); string quarantineDir = Required("QUARANTINE_DIR"); // Pipeline-supplied, never derived from document content. string documentId = Path.GetFileName(args[1]); if (string.IsNullOrWhiteSpace(documentId) || documentId != args[1]) throw new InvalidOperationException("Document id must be a simple file name."); With configuration validated, nothing downstream depends on an untrusted value. We’ll assume the input file is the finished product of a prior extraction stage, so we’ll only read it, bound its size, and confirm it’s not empty (empty text would otherwise pass as clean). C# const int maxChars = 1 * 1024 * 1024; var info = new FileInfo(args[0]); if (!info.Exists) throw new InvalidOperationException("Extracted-text file was not found."); if (info.Length > maxChars) throw new InvalidOperationException("Extracted text exceeds the screening size limit."); string text = File.ReadAllText(args[0]); if (string.IsNullOrWhiteSpace(text)) throw new InvalidOperationException("Extracted text is empty; nothing to screen."); At this point, we finally have meaningful text to screen — so we’ll send it off and keep the verdict. The Allow flags are our policy: we permit names and addresses, common in an internal knowledge base, and refuse the high-risk identifiers, financial data, health PHI, and secrets. C# bool accepted = verdict.CleanResult == true; bool hardBlock = verdict.ContainsSocialSecurityNumber == true || verdict.ContainsCreditCardNumber == true || verdict.ContainsBankAccountNumber == true || verdict.ContainsPassportNumber == true || verdict.ContainsPrivateKeys == true || verdict.ContainsCredentials == true || verdict.ContainsHealthInsuranceNumber == true; string decision = accepted ? "accept" : hardBlock ? "quarantine (blocked category)" : "quarantine (review)"; Detection told us what is in the document; policy now decides what that actually means. In this case, CleanResult is the gate, and the Contains flags let us classify why a document was held. C# var configuration = new Configuration(); configuration.AddApiKey("Apikey", key); var detector = new DetectApi(configuration); var request = new DlpAdvancedDetectionRequest { InputText = text, AllowPersonName = true, AllowStreetAddress = true, AllowEmailAddress = false, AllowPhoneNumber = false, AllowBirthDate = false, AllowPassportNumber = false, AllowDriversLicense = false, AllowSocialSecurityNumber = false, AllowTaxpayerID = false, AllowCreditCardNumber = false, AllowBankAccountNumber = false, AllowIBAN = false, AllowHealthInsuranceNumber = false, AllowHealthInsuranceMemberID = false, AllowHealthInjuryOrDisease = false, AllowHealthTypeOfTreatment = false, AllowCredentials = false, AllowPrivateKeys = false, AllowBearerToken = false, ProvideAnalysisRationale = true }; DlpAdvancedDetectionResponse verdict = detector.DetectTextAdvanced(request); if (verdict is null) throw new InvalidOperationException("Detection returned no result."); A quick note here: because we set ProvideAnalysisRationale, the response’s AnalysisRationale explains in plain language why a document was flagged, which is intended to help a reviewer triage the quarantine path. It earns that usefulness by quoting the sensitive data back, so we’ll just treat it like the PII it describes: never logged, and surfaced only in an access-controlled quarantine store. For our final step, we’ll route the file to an operator-chosen directory. Clean text goes to the “accept” directory for the rest of the pipeline; everything else goes to quarantine and stays out of the index. FileMode.CreateNew refuses to overwrite, so a repeated ID surfaces as an error. C# string targetDir = accepted ? acceptDir : quarantineDir; Directory.CreateDirectory(targetDir); string targetPath = Path.Combine(targetDir, documentId + ".txt"); using (var output = new FileStream(targetPath, FileMode.CreateNew, FileAccess.Write, FileShare.None)) using (var writer = new StreamWriter(output)) { writer.Write(text); } Console.WriteLine($"{documentId}: {decision}."); } In this example, one catch covers the whole pipeline, and it never echoes the text, the detected values, or the rationale, any of which can carry the very PII we’re trying to keep contained. C# catch (Exception error) { Console.Error.WriteLine($"Screening stopped ({error.GetType().Name})."); Environment.ExitCode = 1; } Because the only path to the “accept” directory is an explicit clean verdict, every error and every ambiguous result resolves the same safe way: not ingested, never ingested anyway. Conclusion PII becomes a serious, durable liability the moment it enters a retrieval index, so the time to check for it is at ingestion, before any text is chunked and embedded. We saw that the right place to screen is the exact extracted text we are about to embed, and that detection is a decision point rather than a cleanup step. We learned that a PII gate should always fail safe: only an explicit clean verdict should reach the index, and every error or ambiguous result should be held. With a clear category policy and a very disciplined handling of flagged content, a RAG pipeline can stay useful without turning an internal knowledge base into a channel for exposing regulated data.

By Brian O'Neill CORE

If you have worked on a mature Java backend application, your src/main/resources/db/migration folder probably looks like a chaotic history ledger: Plain Text V1__init_schema.sql V2__add_users_table.sql ... V142__add_status_column.sql V143__fix_status_typo.sql V144__drop_old_status.sql V145__add_missing_index.sql Tools like Flyway and Liquibase have been the industry standard for over a decade. They operate on an imperative model: “Apply this exact sequence of scripts to transition the database from version N to version N+1.” In the early stages of a project, this model feels logical. But at scale, imperative migrations create severe operational friction. Two engineers branching off main will inevitably create conflicting V146__... scripts. Standing up a new test environment from scratch requires replaying hundreds of sequential scripts. Most dangerously, if an engineer accidentally commits a script containing an unsafe DROP TABLE or ALTER COLUMN, the migration tool will blindly execute it against the production database, potentially causing catastrophic data loss. Modern infrastructure tooling—such as Kubernetes and Terraform—moved away from imperative scripts years ago. They utilize declarative state reconciliation: “Here is exactly what the system should look like right now. Figure out how to make it so, safely.” It is time we manage relational database schemas with that same declarative philosophy. Enter SchemaSynchronizer SchemaSynchronizer is an open-source library for Java and Spring Boot that brings declarative, desired-state schema evolution to relational databases (PostgreSQL, MySQL, MariaDB, SQL Server, and Oracle). Instead of maintaining a perpetually growing chain of migration scripts, your application maintains a single schema-definition.json file representing the current, desired state of your database structure. JSON { "formatVersion": 2, "dialect": "postgresql", "tables": { "work_items": { "createSql": "CREATE TABLE IF NOT EXISTS work_items (id UUID PRIMARY KEY, status VARCHAR(32))", "columns": [ {"name": "id", "definition": "UUID NOT NULL"}, {"name": "status", "definition": "VARCHAR(32)"} ], "indexes": [ "CREATE INDEX IF NOT EXISTS idx_work_items_status ON work_items (status)" ] } }, "changes": [] } On startup, SchemaSynchronizer does not just blindly execute SQL. It executes a state-reconciliation loop. How the Synchronization Engine Works Under the Hood When your Spring Boot application boots, SchemaSynchronizer connects to the database via JDBC and extracts the live catalog metadata. It inspects existing tables, column types, default values, nullability constraints, and indexes. It then computes a diff against your JSON definition and applies changes based on dialect-specific rules. Database engines are notoriously fragmented in how they handle Data Definition Language (DDL). Transactional DDL: On PostgreSQL and SQL Server, SchemaSynchronizer wraps the synchronization process in a transaction. If something fails, the entire schema change is rolled back cleanly.Implicit Commits: MySQL, MariaDB, and Oracle natively commit open transactions the moment a DDL statement is executed. SchemaSynchronizer adjusts its execution strategy for these dialects, relying on granular verification checks rather than transaction rollbacks.Distributed Locking: To prevent multiple application instances from fighting over the schema during a rolling deployment, the library utilizes native distributed lock primitives (pg_try_advisory_xact_lock in Postgres, sp_getapplock in SQL Server, GET_LOCK in MySQL, and DBMS_LOCK in Oracle). The Safety Boundary: Fail-Closed Design The most critical architectural requirement for automated database tooling is preventing accidental data destruction. SchemaSynchronizer is built around a strict, fail-closed safety boundary. It divides schema evolution into two distinct categories: 1. Safe Additive Changes (Automated) If you add a nullable description column to your JSON definition, SchemaSynchronizer detects the missing column and issues the ALTER TABLE ... ADD COLUMN statement automatically on startup. The same applies to creating new tables, creating indexes, widening compatible data types (e.g., VARCHAR(50) to VARCHAR(100)), or relaxing NOT NULL constraints. 2. Destructive or Ambiguous Changes (Manual Review) What happens if you delete a column from the JSON definition? SchemaSynchronizer never infers a DROP operation. If it detects an orphaned live column, or a type narrowing that could truncate data, it halts the synchronization process and outputs the exact SQL required to resolve the difference. Plain Text [SchemaSynchronizer] Destructive difference detected. Manual review required: ALTER TABLE work_items DROP COLUMN old_status; This enforces a hard operational boundary. Routine additive drift is repaired automatically, reducing developer friction, but destructive reconciliation always requires explicit human intervention and approval. Handling Complex Migrations With Checksummed Ledgers Not every database operation can be cleanly inferred from a declarative table structure. Operations that carry specific business logic — such as data backfills, complex CHECK constraints, triggers, or specific user grants — require explicit execution. To handle these, SchemaSynchronizer includes an ordered changes array within the JSON document. These act as an immutable, checksummed ledger, similar to traditional migrations, but with a critical addition: Verification SQL. JSON "changes": [ { "id": "001-work-item-status", "description": "Backfill and constrain work item status", "statements": [ "UPDATE work_items SET status = 'PENDING' WHERE status IS NULL", "ALTER TABLE work_items ALTER COLUMN status SET NOT NULL" ], "verificationSql": "SELECT NOT EXISTS (SELECT 1 FROM work_items WHERE status IS NULL)", "phase": "AFTER_SCHEMA" } ] The verificationSql must be a read-only query that returns a boolean true. This ensures idempotency. If the database already satisfies the verification query (perhaps because a DBA manually applied the fix during an incident), SchemaSynchronizer records the change set as completed without attempting to blindly replay the UPDATE statements. Implementation: Spring Boot and CLI SchemaSynchronizer integrates natively into Java applications. For Spring Boot, add the dependency to your build file: Maven: XML <dependency> <groupId>com.thinkaillc</groupId> <artifactId>schema-synchronizer</artifactId> <version>2.0.0</version> </dependency> Configure your application.properties to enable the initializer and tell Hibernate to back off and only validate the schema: Properties files schema-synchronizer.enabled=true schema-synchronizer.schema=public schema-synchronizer.fail-on-pending=true # Let SchemaSynchronizer handle DDL; tell Hibernate to just validate spring.jpa.hibernate.ddl-auto=validate The Standalone CLI Pipeline Many organizations prefer to decouple schema migrations from application startup, executing them directly from a CI/CD pipeline (like GitHub Actions or Jenkins). SchemaSynchronizer ships a standalone executable JAR bundling the JDBC drivers for all supported engines. You can dry-run a deployment against a target database to see exactly what would change, without committing any DDL: Shell export SCHEMA_DB_PASSWORD='target-password' java -jar schema-synchronizer-cli-2.0.0-standalone.jar dry-run \ jdbc:postgresql://localhost:5432/target_app app_user - \ schema-definition.json public schema_synchronizer_history Ready to Drop the Migration Scripts? Imperative migration scripts served the Java ecosystem well for years, but as our deployment topologies grow more complex, managing hundreds of linear state-transition files becomes a liability. By adopting a declarative approach, you can eliminate merge conflicts on migration files, instantly stand up new environments from a single source of truth, and enforce a strict safety boundary that prevents automated tooling from ever dropping production data. Check out the architecture, dialect support, and Quick Start guides on the repository: GitHub: eugenena/SchemaSynchronizer

What does it take to build and sustain an open-source project for more than a decade? For Shai Almog, creator of Codename One, it takes a willingness to challenge conventional approaches to software development and a continued commitment to the developers who rely on the work. I recently caught up with Shai to discuss the inspiration behind Codename One, the realities of maintaining an open-source project, and his perspective on how the Java ecosystem has evolved over the years. Check it out below! 1. What is Codename One, and what originally inspired you to create it? Codename One is what we always wanted Java to be: write once, run anywhere, without accepting the lowest common denominator. It’s what you would get if Spring Boot, Flutter, Go, and Node.js had a baby that inherited the best qualities of each. You get Java’s mature ecosystem, Flutter-style control over the UI/portability, Go-like native binaries and footprint, and the full-stack client/server connection people associate with Node.js. With Codename One, one Java codebase can target iOS, Android, macOS, Windows on x64 and ARM64, Linux on x64 and ARM64 with either glibc or musl, and even the browser. A small self-contained desktop GUI application can be around 4–5 MB, with no JVM to install. Underneath, Codename One uses different pipelines for different targets. On our ahead-of-time compiled platforms, ParparVM analyzes the reachable Java bytecode, removes unused code, translates what remains to C, and hands the result to the platform’s native compiler. Codename One also provides the portability libraries, UI toolkit, tests, theming, and build system needed to handle the ugly platform-specific details. We don’t pretend every platform is identical. Our builder architecture assembles the native pieces each application needs, so developers can use platform-specific capabilities without fragmenting the entire codebase. The original inspiration was frustration with Java’s unfulfilled promise. Java gave us portability, but conventional Java application deployment became increasingly heavy. We wanted the language and ecosystem without requiring a large runtime or giving up native performance, platform access, or control of the UI. Now we’re taking that architecture to the backend. The goal is to bring the startup time, footprint, and native compilation work we developed for constrained devices into serverless and microservice workloads. I want Java developers to get the full-stack productivity they expect from Spring Boot, with deployment economics closer to Go and Node.js. 2. Codename One has been around for more than a decade. What has kept you motivated? The community is the biggest reason. People have built businesses and products on Codename One. When someone has trusted your work with something that important, walking away isn’t an option. For a long time, competing in this market was brutal. Our competitors were not normal startups. They were Google, Facebook, Microsoft, and companies with teams larger than our entire community. There was a limit to how much a small team could build, test, document, and support. Agentic development changed that equation. It lets us scale engineering work in a way that simply wasn’t possible a few years ago. We now maintain an enormous cross-platform test matrix and can move at a pace that is competitive with much larger organizations. On most feature dimensions, I would argue that we are already ahead of the big-name alternatives. We still have gaps. We’re a small team, and pretending otherwise would be ridiculous. But our size is also an advantage. The person reading a bug report can be the same person who changes the compiler. We can respond to developers in hours where a large company may need three teams, a quarterly roadmap, and permission from a product manager. The other reason is simpler: we love Java, and we use the competing tools. Every time we do, we run into one of the compromises Codename One was created to remove. 3. What’s one of the biggest misconceptions developers have about maintaining an open source project? That maintaining open source is mostly about accepting code contributions. A pull request looks like free work from the outside. For a maintainer, it can be the beginning of a large amount of work. Is the code correct? Does it meet the project’s standards? Does it work on every target? Who maintains it five years later? How do you tell a volunteer who spent a week on something that the design is wrong? How many rounds of changes can you reasonably ask them to make? We recently stopped accepting outside pull requests. AI wasn’t the reason. If anything, AI improved the baseline quality of many contributions. It also made some of them enormous, because generating another thousand lines became almost free while reviewing those thousand lines did not. The real problem was validation. Codename One currently has more than 60 GitHub Actions workflow files. A small framework change can affect Android, iOS, JavaScript, desktop, bytecode translation, native compilation, and screenshot baselines. That matrix represents roughly $15,000 worth of monthly CI capacity, which GitHub currently provides to us. At that point, validating and repairing an outside patch often took longer than implementing the confirmed change ourselves. So today we ask people to contribute through reproducible issues, benchmark counterexamples, design discussions, testing, and documentation feedback. Those contributions are often more valuable than code because they expose problems we cannot reproduce in our own environments. Also sorry for all the GitHub Actions downtime ;-) 4. You’ve watched Java and the broader development ecosystem evolve considerably. What changes have surprised you the most? The biggest surprise is that Java effectively surrendered the idea of being small. GraalVM is a remarkable piece of engineering, so we aren’t literally alone (although it's a very different project that hasn't really taken the same direction and hasn't taken it as far). But mainstream Java spent years treating bigness as evidence of seriousness: larger runtimes, larger frameworks, more dynamic machinery, longer warmups, and more memory. The industry began treating that weight as inevitable. I'm not against comprehensive/large; Codename One itself is HUGE... But not in runtime, just in API and scope. Most successful newer platforms moved in the opposite direction. Go made self-contained native binaries normal. Node.js made application deployment almost trivial. Mobile and serverless made startup time and memory consumption economically important again. Java largely responded by optimizing large systems rather than questioning why every system needed to be large. Modules are a good example. They help organize and encapsulate a large dependency graph. They do not eliminate that graph or turn it into a small, closed-world native program. Codename One puts a real Java GUI application or full server into a roughly 5 MB binary. What surprises me is not that this works. We have spent years proving that it works. What surprises me is how lonely that position still is. Java could be one of the best languages for small native applications and low-cost serverless workloads. Most of the ecosystem simply stopped trying to make that happen and let Go eat our lunch. 5. After spending so much of your career in technology, what do you enjoy doing when you completely step away from a screen? Completely step away from a screen? I’m probably the wrong person to ask. I just got back from Thailand with my kids. We had a wonderful time. I also had my laptop with me and was doing pull requests on the beach. This is what I enjoy doing. I don’t have an unwind button. Vacations often leave me more exhausted than work because sitting still doesn’t recharge me. If I have an idea and I can build it, fix it, or argue about it with another developer, that is how I unwind. Explore more of Shai's content here.

Without writing code, Salesforce makes it simple to modify a live system. It only takes a few minutes to create a new field, a validation rule, an updated flow, or a modified permission set. Many teams use the platform because of its speed, which also makes it challenging to securely modify many Salesforce organizations after a few years. On a previous project, I oversaw Copado deployment pipelines, including version control, conflict resolution, and automated regression testing, for a Salesforce environment. This article explains the methods that make declarative modifications as traceable as code changes, along with why they matter more as automation increases. Declarative Changes Are Still Changes The cultural transformation is the most significant. Logic is a flow. An access decision is a sharing regulation. A business rule is a validation rule. Users perceive any errors in production in the same manner as they would a defect in handwritten code. Every modification has a ticket, a branch, a reviewer, and a log of its progress when configuration is treated as code. In order for everybody to know what changed, why, and who approved it, we actually tracked work items in JIRA and connected each one to a user story in Copado. Version Control for Metadata Salesforce metadata, including objects, fields, flows, layouts, profiles, and permission sets, can be retrieved as files and stored in a Git repository. Copado manages that relationship. Each user story captures the metadata components that changed, commits them to a feature branch, and promotes them through environments such as development, integration testing, user acceptance testing, and production. Two habits made version control useful rather than ceremonial: Commit only what changed. Large commits that include whole profiles or layouts make reviews hard and conflicts likely. Selecting the specific components for a story keeps the history readable.Write the reason in the story. A commit that says "updated flow" helps nobody six months later. A story that says which business rule changed, and for whom, makes later investigations far shorter. Resolving Metadata Conflicts Conflicts are common when several people work in the same org. Two administrators may edit the same page layout, or two stories may change the same flow. In a Salesforce context, conflicts often hide inside large XML files such as profiles, where a small change in one place can overwrite a different change elsewhere. My approach was to look for conflicts early and resolve them deliberately: Promote often. Small, frequent promotions surface overlaps while both authors still remember their changes.Review the actual difference. Before accepting either version, compare the components line by line and confirm with both authors what the final behavior should be.Prefer permission sets to profile edits. Keeping profiles minimal and adding access through permission sets reduces the number of changes that collide in the same file. Each resolved conflict was recorded in the story, so the reasoning was available if the behavior was questioned later. Regression Testing for Automation Declarative automation can make mistakes that are easy to overlook in a cursory manual inspection. For a single record, a Record-Triggered Flow might function well, but for a bulk update, it might not. A current flow or integration could be blocked by a new validation rule. For one position, an approval procedure may route correctly, but for another, it may route wrongly. These problems were identified by the pipeline's automated regression tests before they affected customers. We concentrated our testing on business-critical activities, such as generating and updating important records, completing approval procedures from start to finish, and ensuring that automation operated appropriately when records were modified in large quantities. Custom code was covered by Apex tests, and if a test failed, the pipeline would not encourage a modification. Test results were just as important as the tests themselves. In order to ensure that a failed test indicated a true change in behavior rather than missing data, we maintained a small, known set of records in each testing environment that covered the roles, regions, and record kinds that the business utilized most frequently. Test records and anticipated outcomes were added to that set as part of the same tale when a new procedure went live. Not every click was intended to be covered. It was assurance that the operations on which the company relied continued to function with each release. Why This Matters More Now Salesforce orgs keep accumulating automation. Teams moving from Process Builder to Flow, as I have done in my current role, are consolidating logic that was previously scattered. AI-based features add new behavior that depends on the same data, permissions, and automation. The more an org does automatically, the more important it is to know exactly what was released and when. Release discipline is what keeps that growth manageable. It does not slow teams down for long. After the first few cycles, small stories, early conflict resolution, and automated tests make releases routine rather than stressful. A Short Checklist Link every change to a story and a reason. Configuration deserves the same traceability as code.Keep commits small and specific. Select only the components that changed.Promote often and resolve conflicts with both authors. Review the real differences, not just the file names.Test business-critical paths automatically. Include bulk updates, approvals, and integrations.Write release notes and root cause records. They turn production investigations into short tasks. Low-code platforms remove much of the effort of building software. They do not remove the need to manage change carefully. A pipeline that treats Salesforce configuration like code gives teams the speed of the platform without losing track of what they have built.

A coding agent opens a pull request for a background job that keeps retrying after the user cancels it. The test that caught the bug now passes, and the agent reports that all tests are green. But the diff changes two things: the retry handler and the test itself. Which change eliminated the failure, and would the test still catch the bug if it came back? I use coding agents to design integration tests, triage failures, and prepare fixes for back-end services, so this is a practical review question for me. Sometimes the test really is wrong, and changing it is the right fix. Sometimes the agent has just found the shortest way to a green build. In continuous integration, both look the same. Benchmarks show how far it can go. ImpossibleBench makes the tests contradict the task, so any passing solution has to cheat, and GPT-5 still passed 54% of its conflicting SWE-bench tasks. But tests aren't always right, either: OpenAI found issues with the tests or the task description in 59.4% of the 138 SWE-bench Verified problems it examined. So a test edit proves nothing by itself. Below, I'll show how to get evidence either way with a small Python example: one bug, three PRs that all pass CI, and a check that tells them apart in seconds. A Green Build Is a Statement About the Checks I find it useful to look at a test run as a pipeline. Tests are selected and set up, the application runs, the test observes and asserts, the harness reports the outcome, and a CI gate turns that outcome into a merge decision. Figure 1: A test run as a pipeline. A change to any stage other than the application changes what a green result means. Only the application stage can repair a defect. If a PR touches only the application, going from red to green tells you something about the code. If it also touches another stage, the green result may say more about the evidence than about the bug. One Bug, Three Pull Requests The Failing Test Here's the contract for the example. Once cancel() returns, no new attempt may start for that job. An attempt that's already running may still report its result, and a job that wasn't canceled must still be retried after a transient failure. The model below starts with one attempt already running and handles callbacks one at a time. run_retry() stands in for the scheduler starting a queued retry; there are no threads, network calls, or sleeps. Save it as retry_job.py: Python class RetryJob: def __init__(self): self.attempts = 1 self.canceled = False self.retry_pending = False def cancel(self): self.canceled = True self.retry_pending = False def on_transient_failure(self): self.retry_pending = True def run_retry(self): if self.retry_pending: self.retry_pending = False self.attempts += 1 The bug is about ordering. cancel() clears any queued retry, but the running attempt can report a transient failure afterward, and on_transient_failure() then queues another retry without checking whether the job was canceled. These two tests go in tests/test_retry_job.py: Python from retry_job import RetryJob def test_cancel_blocks_retry_after_late_failure(): job = RetryJob() job.cancel() job.on_transient_failure() job.run_retry() assert job.canceled is True assert job.attempts == 1 def test_uncanceled_job_retries(): job = RetryJob() job.on_transient_failure() job.run_retry() assert job.attempts == 2 Run python -m pytest -q from the repository root. The cancellation test fails because attempts is 2, and the ordinary retry test passes. Three Ways to Turn It Green The first PR leaves the application alone and loosens one assertion: diff - assert job.attempts == 1 + assert job.attempts >= 1 Now the test accepts a second attempt, and a third and a fourth. Its name still promises that cancellation blocks a retry, but the assertion no longer checks that promise. The second PR fixes the handler and leaves the tests alone: diff def on_transient_failure(self): + if self.canceled: + return self.retry_pending = True The cancellation test retains its meaning, and the ordinary retry test shows that the guard hasn't disabled retries for every job. The third PR does both. It's the easiest to miss: the application diff looks right because the application is right now, but the test can no longer catch the bug if a later change brings it back. Ordinary CI reports 2 passed for all three PRs. Run the Swap Check Ordinary CI runs exactly one combination: the PR's tests against the PR's code. The two combinations it skips answer the questions a reviewer actually has: Base tests on head code. Does the change still meet the original expectations? If an original test fails here, the PR changed a promise, and that needs a reason.Head tests on base code. Do the new tests still detect the bug? For a bug fix, at least one of them should fail here. If they all pass, they can't tell the buggy code from the fixed code. Figure 2 shows all four combinations for the three PRs: Figure 2: Swap matrices for the three pull requests. CI only shows the head code with head tests, which is green in all three. In an agent's PR, the tests are part of the change, so you have to swap them on purpose. Git worktrees make that cheap. These commands build the cell that matters most for a bug fix, the base code with the head's test files: Shell base=$(git rev-parse main) # commit IDs, because inside a worktree head=$(git rev-parse fix-and-weaken) # HEAD would mean something else tests=$(git diff --name-only --no-renames "$base" "$head" -- \ tests conftest.py pytest.ini) cell=$(mktemp -d) # base code + head tests git worktree add --quiet --detach "$cell" "$base" git -C "$cell" restore --source="$head" -- $tests (cd "$cell" && python -m pytest -q) git worktree remove --force "$cell" On the third PR, this prints 2 passed: the new tests can't see the bug. For the other cross cell, add the worktree at $head and restore the test files from $base. The PR that only loosens the assertion raises a flag in both cross cells, and the honest fix raises none. Whole-suite results are coarse, though. A head test can fail on the base code just because it imports something the fix added, and a test that already fails on the base hides the signal. In real use, compare outcomes test by test, as SWE-bench does with its FAIL_TO_PASS and PASS_TO_PASS lists. The list of test paths matters, too. It has to cover everything that defines a check, including fixtures, snapshots, and harness configuration such as pytest.ini, and files that mix application and test settings, such as pyproject.toml, need a human look. This conftest.py hook, for example, rewrites failed test reports as passed: Python import pytest @pytest.hookimpl(hookwrapper=True) def pytest_runtest_makereport(item, call): outcome = yield report = outcome.get_result() if report.failed: report.outcome = "passed" In a PR that adds only this hook, with only tests on the list, the swap check treats conftest.py as application code and reports a clean fix. Add conftest.py, and both warnings appear. Anthropic researchers describe this family of hacks, along with calling sys.exit(0) before any assertion runs. Treat the result as review evidence, not a verdict: a refactoring's new tests should pass on the old code, while a bug fix's reproducer should fail there. I'd run the swap check as a report-only CI job with the full Git history (fetch-depth: 0) and, because it runs the candidate's code, a read-only token that isn't left in the checkout (permissions: contents: read, persist-credentials: false). When Changing the Test Is the Right Fix Tests change all the time. In one comparison, human-written PRs modified existing tests more often than agent-assisted ones: 43.5% versus 20.6% in a manual sample. So the useful question is why a test changed. Four answers hold up: The requirement changed. If the service now lets one already-dispatched retry start after cancellation, the test has to say so, and someone has to have approved the new guarantee.The fixture was wrong. Maybe it labeled a permanent failure as transient. Fix it, but keep a transient-failure case so the retry path stays covered.The test looked too early. If cancellation is acknowledged asynchronously, wait for the acknowledgment or step a fake scheduler forward instead of adding a sleep.The test pinned an implementation detail. Move the check to the public boundary and show that it still catches a violation. This was the largest category in OpenAI's audit, at 35.5% of the examined problems. In none of these cases is the application's current output the reason for the new expectation. Ask for the contract, the corrected setup, or the reason the old observation was invalid, plus swap-check evidence that the revised test still fails on the bug. Guardrails In the Agent Loop An agent needs room to investigate failures, including bugs in tests, and a clear way to report a conflict between a requirement and an assertion rather than quietly resolving it. For the cancellation task, I'd commit the reproducer first, after checking that it fails for the right reason, and give the agent an instruction like this: Fix the retry behavior so that no new attempt starts after cancel() returns. Keep the late-failure reproducer and the ordinary retry case. If you think a test is wrong, stop, explain how it conflicts with the contract, and show any proposed test change separately. Report the commands you ran and any skipped, expected-failing, or unexecuted cases. To make tests read-only for an agent, Claude Code permission rules can deny edits to path patterns, and a PreToolUse hook can block a tool call by exiting with status 2; Cursor's preToolUse hook works the same way. These rules cover the agent's file tools, not every shell command, unless you also run Claude Code's sandbox with allowUnsandboxedCommands set to false. In my own setup, the agent that writes integration tests works only in test code, and its standing instructions forbid silencing or deleting a failing test to get a green run. Triage first classifies each failure as an application, environment, or test problem; application bugs become bug reports, not test edits. Changes to the agents' own instructions need a person's confirmation, and agents never merge. In the Repository Repository controls decide what gets merged. The useful ones keep the evidence for important behavior out of the candidate patch's reach, as Figure 3 shows: Figure 3: What the candidate patch can change, and what should be reviewed separately. Own the evidence. Assign test, fixture, harness, and CI paths to owners in CODEOWNERS, and turn on "Require review from Code Owners"; the file alone only requests reviews. GitHub reads it from the base branch, so a PR can't loosen its own owners. Give CODEOWNERS itself an owner, and require approval of the most recent push. Keep acceptance checks outside the patch. Keep a small acceptance suite for important contracts in a separate repository or at a pinned reference, and run it from a workflow the PR can't edit, such as a ruleset-required workflow on GitHub Enterprise Cloud or a pipeline execution policy on GitLab Ultimate. Run it from its own checkout with its own pytest.ini, because pytest searches parent directories for configuration and could pick up the candidate's conftest.py, and set PYTEST_DISABLE_PLUGIN_AUTOLOAD to block plugins the candidate's package might register. Prove that the check ran. Make the acceptance job unconditional: a required status check accepts a skipped result, and on GitHub "a job that is skipped will report its status as 'Success'." Then fail the job unless the required cases ran and passed. Any runner that writes JUnit XML, including pytest, Maven Surefire, and Gradle, can feed a short script that checks each required test by name. Don't count a test that passed only on a rerun; Surefire keeps the build green and records the failed attempt as a flakyFailure. Parse the report with defusedxml, since it's untrusted input. No single control catches everything. The required-tests check trusts a harness that lies, the acceptance run can't see a weakened test next to a real fix, and the swap check is only as good as its path list. Costs and Limits The swap check runs the suite four times, so for a large suite I'd limit it to the affected tests. It needs a base revision where the reproducer can run, flaky tests add noise to the cross cells, and each cell must import its own checkout rather than an installed package or a shared node_modules folder. These controls also target shortcuts rather than code written to defeat them. In a seventh variant of the example, retry_job.py patches pytest's report object when it's imported, and all three controls pass it. Where that matters, run acceptance tests against a built artifact or a running service, and keep reading the application diff. What to Ask for Before Approving Ask for evidence instead of a verdict. For the fixed example, a useful PR description would say: The late failure callback queued a retry after cancellation. The handler now ignores that callback when the job is canceled. The original cancellation test fails on the old code and passes on the repair; the ordinary retry test also passes. Removing the guard makes the cancellation test fail again. The assertion and event order are unchanged. For a real PR, I'd want to see five things before approving: The reproducer fails on the base code and passes on the head code.For a bug fix, at least one head test fails on the base code due to an assertion rather than an import.Every edited test has a stated reason: a changed requirement, a fixed fixture, a corrected observation point, or a removed implementation detail.Changes to selection, harness, or CI configuration were approved by their owners.The required tests ran and passed, by name, not just by exit code. Wrapping Up Reviewers have always had to ask why a test changed. Coding agents make the question come up more often, because they can rewrite code and tests in minutes and keep going until CI is green. Treat changes to tests, fixtures, the harness, and CI as changes to the evidence: swap tests, accept edits with a reason, and keep checks for important behavior out of the patch's reach. A passing run is only useful when you can explain why it passed.