DZone Spotlight

Software engineers often feel most comfortable with hard skills, such as writing code, studying frameworks, experimenting with databases, reviewing architectures, or building side projects. As engineers advance into leadership roles, technical depth becomes even more critical. Technical leaders must guide decisions that impact other engineers, teams, and sometimes entire platforms. To do this effectively, strong communication and influence are essential, but they must be grounded in sound technical judgment. Without sufficient technical depth, leadership can steer teams in the wrong direction. Open source is especially valuable in this context. Mature projects expose engineers to challenges rarely found in tutorials or new applications, such as software evolution, legacy modernization, backward compatibility, architectural trade-offs, design decisions, documentation, code reviews, and the internals of widely used frameworks. Open source also offers opportunities to learn from experienced engineers worldwide and to observe how complex technical decisions are made. This article will explore how open source helps Software Engineers develop the hard skills needed to become both better developers and stronger technical leaders. Why Technical Leaders Still Need Hard Skills Leadership in software engineering does not require knowing everything. The technology landscape is vast, systems are complex, and specialization is deep. However, a technical leader must have a solid technical foundation to exercise sound engineering judgment. They should be able to communicate effectively with engineers, understand core software design and architecture concepts, recognize key trade-offs, and know which questions to ask when solutions are unclear. Without this foundation, effective leadership is challenging. Discussions about scalability, consistency, coupling, performance, security, or maintainability can be difficult to follow if the leader does not understand the team's language. The goal is not to be the top specialist in every area, but to have enough context to distinguish meaningful concerns from unnecessary complexity, recognize when further investigation is needed, and help the team progress when discussions stall. Technical Judgment Helps Teams Move Forward Engineering discussions do not always converge naturally. Two experienced engineers may advocate different architectural approaches, each with valid arguments. A migration can involve several strategies. Teams may struggle to decide whether to introduce a new service, adopt a different database, or continue investing in the current system. A technical leader must help navigate these situations. This does not mean making decisions alone. Effective leadership involves asking insightful questions, clarifying assumptions, identifying missing information, running experiments, or helping the team focus on the most important trade-offs. Technical knowledge makes that possible. Without technical knowledge, a leader risks relying on the most confident voice in the room. The Cost of a Bad Decision Grows With Your Scope The higher you progress in a technical or executive career, the larger the potential consequences of your decisions become. A software engineer may make an implementation decision that affects a feature. A staff engineer can influence several teams. A principal engineer may shape a platform used organization-wide. A VP of Engineering or CTO can approve a technical direction that impacts hundreds of engineers and years of investment. As your scope increases, so does the cost of mistakes. Technical knowledge cannot eliminate failure. Architecture involves uncertainty, and even skilled engineers make decisions that later prove incorrect. However, stronger technical judgment reduces the likelihood of avoidable mistakes and helps leaders identify risks earlier. This is why technical depth remains important, even as you write less code. Technical Knowledge Keeps You Connected to the Team Hard skills also help leaders stay connected to the engineers doing the work. A technical leader should be able to join design discussions, understand why the team struggles with integration, follow the impact of legacy constraints, and recognize when tasks that seem simple at the management level are actually complex to implement. That connection matters. When engineers feel a leader understands their work, communication improves. It becomes easier to discuss risks, challenge unrealistic expectations, and translate technical constraints for other parts of the organization. This does not mean micromanaging implementation or overriding specialists. On the contrary, strong technical knowledge helps leaders know when to contribute, when to ask questions, and when to trust the experts. The goal of hard skills in leadership is not technical dominance, but better judgment, communication, and decision-making at scale. How Open Source Builds Technical Leadership Skills Open source is valuable for developing technical skills because it exposes engineers to real systems under real-world conditions, often evolving over many years with decisions made publicly. You see more than just the final code. Many projects allow you to review issues, discussions, rejected alternatives, pull requests, review comments, and the tests that justified changes. For those building technical leadership, this experience develops both implementation skills and sound judgment. Learn Design From the People Who Built the Technology A key advantage of open source is the ability to learn design decisions directly from the project’s creators. Rather than just reading how to use a framework, you can examine why abstractions exist, how APIs evolved, which alternatives were rejected, and what constraints shaped the design. Comparing the public API with its implementation reveals where convenience, performance, compatibility, and maintainability may conflict. This offers a deeper level of learning. You move beyond learning how to use the technology to understanding how its creators approach software design. And eventually, if you contribute long enough, you stop being only an observer and start participating in those decisions yourself. Learn How Software Survives for Decades Many engineers primarily work on relatively new systems. Mature open-source projects introduce a different challenge: evolving software that cannot be easily rewritten. Some open-source technologies have existed longer than many software companies. This involves managing backward compatibility, deprecated APIs, refactoring, migration paths, legacy design decisions, performance requirements, security issues, and users who rely on behaviors that were never intended to be permanent. Here, legacy modernization becomes a tangible challenge. You learn that modernization rarely means replacing everything with the latest architecture. It is typically about advancing a system without disrupting existing users. For technical leaders, this lesson is fundamental. Most architectural work occurs within existing systems, not on a blank slate. Learn Quality Through the Cost of Regression Open-source projects also make the cost of regression very visible. A seemingly minor change can disrupt another operating system, database, integration, or an unforeseen use case. This requires mature projects to be disciplined in testing, compatibility, review, and release processes. Participation in these projects leads to a deeper understanding of software quality. Tests are not there simply to increase coverage. They protect behavior. Code review is not mere bureaucracy. It helps prevent individual misunderstandings from causing widespread issues. Backward compatibility is not resistance to innovation; it is often a contractual obligation to users. These lessons translate directly into technical leadership because leaders are responsible not only for introducing change but also for understanding its consequences. Learn From Some of the Best Engineers in the Industry Open source also removes a significant barrier: organizational boundaries. Within a company, you typically learn from colleagues. In open source, you collaborate with contributors from companies, universities, foundations, and communities worldwide. This allows you to review code from highly experienced engineers, observe their problem-solving approaches, receive feedback, and sometimes discuss technical decisions directly with the creators of widely used technologies. Such access is rarely available elsewhere. Because discussions are often public, you can learn even without direct participation. Carefully reading complex design discussions can provide deeper architectural insights than studying lists of patterns. Understand the Internals of the Tools You Depend On There is a significant difference between knowing how to use a framework and understanding its inner workings. Examining a project's internals reveals the constraints that shape its behavior. Why does this API behave this way? Why is this operation expensive? Why does this abstraction leak under certain conditions? Why was a seemingly obvious feature rejected? This deeper knowledge enhances your ability to debug, design, and make architectural decisions. For technical leaders, this is important because decisions often occur at a level above the application code's abstraction. Understanding underlying mechanisms helps you assess when a framework is suitable, where its limitations lie, and when it is being misapplied. Open-Source Practices Can Scale Beyond Open Source Many organizations now actively seek to replicate these practices internally. There is even a term for it: InnerSource. The goal is to adopt practices from successful open-source communities within the company, such as transparent development, shared ownership, cross-team contributions, visible discussions, reusable components, documented decisions, and review processes that facilitate knowledge sharing. This is important because a major risk in software organizations is knowledge becoming isolated within organizational structures. A team may possess valuable knowledge, but if code, decisions, and practices remain confined, the broader organization cannot benefit. Open-source-style collaboration helps reduce this dependency. In some cases, an engineer contributing a few hours a week to a widely used project can impact more software systems than months of work within a single company. This is not necessarily due to greater skill, but because the work is visible, reusable, reviewable, and accessible to a larger community. This is a key leadership lesson: impact depends not only on the amount of work you do, but on how effectively your knowledge scales beyond yourself and your immediate team. Conclusion Hard skills are essential for technical leadership. As software engineers advance to roles such as staff engineer, principal engineer, software architect, or technology executive, their decisions have a broader organizational impact, making technical judgment critical. Open source offers an ideal setting to develop this judgment by exposing engineers to real-world software evolution, legacy modernization, architecture, design, testing, framework internals, code review, and the insights of experienced engineers worldwide. It teaches not only how to write software, but also how to evolve, protect, and guide it over time. Open source is often described as a philosophy of collaboration, freedom, transparency, and knowledge sharing. While these values are important, open source extends beyond philosophy. Mature open-source ecosystems are among the most advanced software engineering environments, where technologies used by millions are designed, reviewed, tested, maintained, and improved publicly. For software engineers seeking technical leadership, participating in this environment is an excellent way to strengthen hard skills, refine judgment, and learn how impactful software is built and maintained.

By Otavio Santana CORE

Executive Summary The Ampere® System Profiler (ASP) is a Python command-line interface utility that uses a set of Linux profiling tools to gather system-level performance metrics while running applications of interest. The system-level collectors run in parallel and provide detailed reporting on network, disk, CPU utilization, and top functions via perf during the sample period. This is helpful to determine system-level bottlenecks. One of the foundational features of the ASP is its easy-to-read HTML reports that provide a simple view of the collectors’ outputs. Additionally, all the raw data to generate reports are saved in logs should an end user need to explore system profiles in greater detail. Running this tool is simple and provides an easy-to-run command line with minimal overhead to accurately profile any benchmark. This tool is part of the Ampere Performance Toolkit (APT) and can be used by a performance engineer for a top-down approach to root-causing performance problems. What Is the Ampere System Profiler? The Ampere System Profiler (ASP) is comprised of multiple collectors that collect: numastatsocket powerCPU utilizationnetwork utilizationperf functions They run concurrently in the background while the user collects profiles of applications or benchmarks on Ampere systems. The ASP project can be found on Ampere’s GitHub page. The core of the ASP’s utility comes from open-source Linux profilers. Why Do We Need the Ampere System Profiler? Ampere System Profiler exists as part of the larger Ampere Performance Toolkit and is used as an application or benchmark performance analysis tool, and is particularly useful in identifying system-level bottlenecks to help identify sources of performance issues. It can be used to help understand: What system resources are undersaturated or are experiencing bottlenecksOS-level metrics like IRQ affinity and context switch rateThe amount of user and kernel time being spent on the CPUApplication-level functions that consume CPU cycles during the sample period When Do We Use the Ampere System Profiler? Understanding the APEX Framework Performance tuning is a process of systematic investigation, moving from a broad, system-wide view down to the specific interactions between code and hardware.  The Adaptive Profiling and Execution (APEX) Benchmarking and Optimization Funnel Performance optimization is as much art as it is science. The APEX framework uses tools and methodologies to add structure and rigor to the process and can bridge the gap between creative intuition and empirical fact. We propose applying the APEX (Adaptive Profiling and Execution) methodology to enable root cause analysis for solving performance problems. Follow the funnel above from top to bottom to effectively use the procedure. The methodology recommends starting with assessing platform health as a first step to ensure that the platform used for performance analysis is set up well. An unhealthy platform may mislead the performance analysis. Consider capturing initial performance metrics before tuning any system or application settings. This establishes a clear understanding of the current workload and identifies key scalability knobs. We recommend using Ampere’s PerfKit Benchmarker (APB), which supports many open-source applications, to create a reliable baseline for further analysis and tuning. Next is to assess system performance and any hardware or system bottlenecks while the code is running — this is where the Ampere System Profiler (ASP) is useful to eliminate any system or resource bottlenecks. The ASP can also be used to right-size the instance shape and ensure that the compute resources are efficiently consumed by the workload. One method is to use the APB’s automated benchmarking framework to start and stop ASP’s collectors during the run phase of a given APB benchmark. This ensures that the profile is collected while critical code paths are executed and a clear profile report is generated. Once system and resource bottlenecks are eliminated, if the performance issue persists and points to CPU cycles not being used efficiently, we propose going to the next step in the pyramid and using the Ampere PMU Profiler to root-cause the issue further. Finally, system benchmarking should be done after all bottlenecks are resolved or analyzed to effectively measure the system’s performance for the workload. Following this systematic APEX methodology ensures that we eliminate possible issues as part of a structured process to efficiently conduct root cause analysis. System-Level Analysis Goal: Understand the overall system health and identify the primary resource bottleneck. Is the application limited by CPU, Memory, Disk I/O, or Network?  Key Questions: Is the overall CPU utilization high?  Is it predominantly user time or system time (application or kernel)?Is the system swapping or under memory pressure? Is the application spending a lot of time waiting for I/O (iowait)?Are there system limitations? Is the network oversaturated?Is the CPU load evenly distributed?Are the top functions mostly spent in kernel? Common Tools: sarmpstatnumastatiostatsensorsperf The Ampere System Profiler utilizes all these collectors within a single command-line interface. Example Usage and Output: Plain Text “asp -n 20 -i 2 -N eboot0 –F 99 –o example” Let’s break down this command: “asp” is the CLI utility for invoking the tool. Passing “–n” is the number of samples a user wants to collect, and “-I" is the frequency in seconds to collect each sample. This is required to capture a network interface “-N”, which is capitalized, and tells the network profiler which interface to profile. Finally, “-F” indicates the frequency rate in Hz to collect its profile. The perf frequency rate will significantly impact file size; a lower rate reduces the overall file size, which is useful for longer-running profiles. The user can then pass “-o” to designate where they want data outputted. The above command collects: 20 samplesSets interval of 2 secondsRuns for a total of (samples x interval) - 40 secondsCollects network information (-N) on NiC labeled eboot0Uses perf record collection frequency of 99 Hz as the default sampling rateWrites logs to an output directory titled “example” Metrics reported by the Ampere-System-Profiler: metric namedescription CPU Utilization Percentage of CPU Utilization over Time. Includes percent of system time and percent of user time (application) Average Per Core Utilization Average User/System time per core during sample period CPU Frequency Average per-core frequency during sample period  Socket Power Shows CPU Socket Power over Time for CPU+IO  Numastat  Shows per-node memory statistics Disk I/O Outputs Disk Bandwidth over time during sample period Network I/O Shows network bandwidth during sample period  Perf Top Functions  Shows top perf record functions as percentage of cycles during sample period. Includes application code and kernel code Case Study: Redis Performance Regression Problem statement: A 55% performance regression was observed when running Redis in a virtual machine (VM). The ASP was used to help identify and mitigate two separate issues. First, the CPU profile indicated a large proportion of %soft IRQs being handled due to network saturation generated by the memtier traffic generation utility. Unbound IRQs accounted for up to 80% core utilization for %soft IRQs compared to 45% on a competitive platform. This finding led the team to choose tcp_stream as a simple reproducer to simulate the behavior of running Redis over the network to try and investigate the issue further. Pinning the IRQs to core 1 enabled the team to isolate the perf report generated by the ASP to compare hot functions running during the benchmark with the simple reproducer. The results concluded that a large proportion of the system time is being spent copying data from kernel space to user space during the critical period of the benchmark. This enabled the team to develop mitigations for reducing CPU time spent on this hot function. The second finding occurred while running tcp_stream in a lab environment, where performance did not align with what was observed, and performance observed in the cloud environment was not reproducible on bare-metal instances. However, a new problem was uncovered. After some configuration alignment, a system profile was performed again, showing an additional hot function where the host instance spends a significant time in spin locks. This provided clues to collect lock stat reports showing much higher wait times with the malformed NIC coalescing settings. This resulted in code fixes being made to kernel code and upstreamed to larger open-source communities. Example Report – Redis Network Bottleneck and High System Time Fig 1: CPU utilization over time The red line on the time series on the left indicates that a majority of CPU utilization is occurring because of high system time. A healthy application will spend the majority of its time in user space, where CPU time does the majority of the work in application code. This high amount of system time indicates that a lot of CPU time is spent outside of critical path code. Notice as well that the green line, which indicates %IOWait, is nearly 0%, indicating little to no IO operations. Fig 2: Network utilization over time The generated Network Utilization chart shows that on this system, the NIC is fully saturated and cannot handle any more network bandwidth being sent by the memtier load generator. Fig 3: Top CPU hotspots during sample period The perf report that generates the Top CPU Hotspots output shows that the redis-server is spending the majority of cycles servicing network-related mlx5e functions to process incoming network packets in kernel space. Conclusion The Ampere System Profiler (ASP) provides an efficient, system-level view of performance bottlenecks while an application or benchmark runs. By collecting CPU (user vs kernel), NUMA, disk and network utilization, socket power, and perf-based hotspot functions in parallel, ASP helps performance engineers quickly determine whether a workload is constrained by system resources or by inefficient CPU cycles in specific call paths. Following the APEX methodology, ASP is used first to eliminate platform and resource bottlenecks; if the issue persists, you can then proceed to deeper CPU root-cause analysis with PMU-based profiling and targeted instrumentation. The resulting HTML reports and raw logs enable both fast triage and deeper investigation when needed. We invite you to download and try the Ampere Performance Toolkit from the Ampere Performance Toolkit Repository. To learn more about our developer efforts and find best practices, visit Ampere’s Developer Center and join the conversation in the Ampere Developer Community. Check out the full Ampere article collection here.

Head-based sampling makes a decision the instant a trace starts, before anyone knows whether that trace is boring or the one you will spend Friday night chasing. That is the wrong time to decide. At that point the request has not failed yet, and the slow dependency call that will define it is still milliseconds away. Head sampling commits before any of that is visible, so it discards a random slice of exactly the traces you will later wish you had kept. Tail-based sampling flips the order. It buffers the spans of a trace until the trace is complete, then decides once the errors and timing are actually on the record. The OpenTelemetry Collector ships a tail_sampling processor that does this well. It also has one operational trap that most tutorials skip, and getting it wrong quietly corrupts every decision the processor makes. This walks through a policy set that keeps the traces worth keeping, and then through the trap. How the Processor Actually Works The tail_sampling processor groups incoming spans by trace ID and holds them in memory. It waits decision_wait seconds for more spans in the same trace to arrive, then evaluates the buffered trace against your policies. If the decision is to sample, the whole trace is exported. Otherwise it is dropped. A minimal configuration that keeps every error and a baseline of everything else: processors: tail_sampling: decision_wait: 10s num_traces: 100000 expected_new_traces_per_sec: 1000 policies: - name: errors type: status_code status_code: status_codes: [ERROR] - name: baseline type: probabilistic probabilistic: sampling_percentage: 5 One thing to internalize early: policies are not first-match-wins. By default, every policy votes, and if any policy votes to keep, the trace is kept. The config above does not mean "errors, otherwise 5 percent." It means "keep all errors, and independently keep 5 percent of everything (including errors)." That OR behavior is usually what you want, but it surprises people who read the list top-down like an if/else. The one exception is an inverted or drop policy, which votes to drop and overrides the keep votes, though none of the policies here use that. A Policy Set That Keeps What Matters The point of tail sampling is to encode "interesting" in policy. In practice, four categories cover most of it: errors, slow requests, business-critical paths, and a low baseline so healthy traffic is still visible. YAML processors: tail_sampling: decision_wait: 15s num_traces: 200000 expected_new_traces_per_sec: 10000 policies: - name: errors type: status_code status_code: status_codes: [ERROR] - name: slow type: latency latency: threshold_ms: 1000 - name: critical-routes type: string_attribute string_attribute: key: http.route values: - /api/v1/checkout - /api/v1/payment - name: baseline type: probabilistic probabilistic: sampling_percentage: 2 This keeps every errored trace, every trace slower than a second, every trace through checkout or payment, and 2 percent of the rest. You can go further with numeric_attribute (keep transactions over a value, or traces with more than N database calls, a cheap way to catch N+1 queries), span_count (keep unusually complex traces), and and composite policies when a single condition is too blunt, for example "slow AND in production AND an API call." Reach for the composite policy when a plain latency rule would sweep in noise from batch jobs or health checks. The Trap: A Trace Decided On Half Its Spans Here is the part that breaks silently. The processor can only make a correct decision if it can see the whole trace. A trace is not correct or incorrect in isolation; a checkout trace might have twenty spans across six services. If those spans are split across two collector instances, each instance sees a fragment, evaluates a fragment, and decides on a fragment. The instance that never received the errored span happily drops the trace. You do not get an error. You get a slow, steady loss of exactly the traces your policies were written to keep, and it looks like the policies are just not matching. A single collector sidesteps this, because it sees everything, but a single collector does not scale and is a single point of failure. The first time I ran into this, error traces started disappearing the day we scaled the sampling collector from one replica to three, and nothing alerted, because the fragments that survived still parsed as valid traces. The moment you run more than one tail_sampling instance, you have to guarantee that all spans of a given trace land on the same instance. The Collector solves this with a two-tier layout. A first tier receives spans and routes them by trace ID using the load_balancing exporter (older configs call it loadbalancing, now a deprecated alias). A second tier runs the actual tail_sampling processor. Tier one, the router: YAML exporters: load_balancing: routing_key: traceID protocol: otlp: tls: insecure: true resolver: dns: hostname: otel-sampling.observability.svc.cluster.local port: 4317 service: pipelines: traces: receivers: [otlp] exporters: [load_balancing] The routing_key: traceID setting is the whole point. It hashes on trace ID so every span with the same trace ID is sent to the same downstream instance. The DNS resolver watches a headless service and keeps the backend list current as sampling pods come and go, rehashing when the set changes. Tier two, the sampler, is a normal tail_sampling pipeline that receives the already-grouped spans and exports the survivors to your backend: YAML service: pipelines: traces: receivers: [otlp] processors: [tail_sampling, batch] exporters: [otlp/backend] Run tier two as a StatefulSet or a stable set of replicas behind that headless service. Put batch after tail_sampling, not before, so you are batching the survivors rather than shuffling spans ahead of the grouping. Sizing Decision_wait and Memory Two settings decide whether this is stable. decision_wait has to be longer than your slowest realistic trace, or you will evaluate traces before their tail spans arrive and drop good data. Rough starting points: 5 to 10 seconds for a monolith, 15 to 20 for microservices, 30 or more when traces cross regions. If you see "interesting" traces getting dropped, this is the first knob to turn. num_traces is the in-memory buffer, and memory is the constraint people hit. A workable estimate: YAML num_traces ≈ expected_new_traces_per_sec × decision_wait × 1.2 memory ≈ average_trace_size × num_traces At 10,000 traces per second, a 15-second wait, and 10 KB per trace, you are holding roughly 180,000 traces and around 1.8 GB before headroom. Size the pods for it and add 20 to 30 percent buffer, because an out-of-memory kill on a sampling collector drops whatever it was holding. Confirm It Is Actually Working Do not trust it because it started. The processor emits metrics that tell you the truth: otelcol_processor_tail_sampling_count_traces_sampled breaks down kept-versus-dropped by policy. If your errors policy is sampling almost nothing, either you have very few errors or your status codes are not set the way you think.otelcol_processor_tail_sampling_sampling_trace_removal_age is how old a trace is when it leaves the buffer. At steady state, it sits near decision_wait, and that is healthy: a trace waits, gets decided, and is removed. The warning sign is the opposite. If it drops well below decision_wait, the buffer is full, and traces are being evicted before they can be decided, so raise num_traces or add replicas.otelcol_processor_tail_sampling_sampling_decision_timer_latency shows how long decisions take. It is your early warning that the instance is overloaded. The end-to-end check that matters: trigger a known error and a known slow request in a test environment, then confirm both traces show up complete in your backend. If they arrive whole, your routing is correct. If they arrive missing spans, the load-balancing tier is not doing its job, and every decision above it is suspect. Tail sampling earns its place because it keeps the traces you will actually open: the failures and the slow paths, not a random 2 percent that probably misses both. But the processor is only as good as the traces it can see in one place. Get the trace-ID routing right first. The policies are the easy part.

This project creates a daily digest for sellers in an enterprise system. Each seller handles accounts at a set of companies and needs to know when something happens at one of them: a product launch, leadership change, new contract, or funding round. That news is often an opening for the next conversation. The pipeline reads the day's incoming news and sends each seller a short email with the handful of articles worth their time about the companies they cover. Every item in that email carries a thumbs-up and a thumbs-down button. At launch, we set a simple quality target. From historical user behavior on other surfaces, we knew that about 9% of served items would receive a user vote. Among those votes, we wanted 70% to be thumbs up. We expected that approval rate to show whether the pipeline was improving. A month later, the dashboard could report vote counts but not explain them. Every downvote triggered the same manual investigation: retrieve the document, retrieve the target profile, read both, and infer the cause. The possible causes belonged to different parts of the pipeline. A user may already have seen the story earlier that week. An article may concern the right parent company but the wrong business unit, such as iPad coverage sent to a seller responsible for iPhone accounts. A technically relevant article from a low-quality publisher may contain no useful signal. A stock-ticker recap may mention the company accurately and still offer nothing useful to the recipient. The feedback system reduced all of these outcomes to the same negative signal. The cost of this ambiguity became clear at production volume. The ingestion layer processed 700,000 to 1 million articles per day across more than 20,000 target entities. Candidate generation reduced that corpus to a bounded pool before any model call. To fix this, we changed the output contract between the judging model and the downstream pipeline. In a previous article, I described the system's three stages: cost-efficient triage, target-aware retrieval, and bounded LLM reasoning. Candidate generation controls most of the pipeline’s cost because it determines how many documents reach the model. This article focuses on Stage 3: the judgments the model should make, the structured verdict it returns, and how those verdicts make user feedback auditable. Judging vs Selecting The single most important design decision in Stage 3 is what job the LLM is not allowed to do: selection. For each target, Stage 2 supplies at most 100 candidates. The model evaluates each document independently and returns a verdict: whether it is relevant to the target, whether the event is new to that user's digest, and whether it should be sent in the email. A pool of 100 candidates usually produces 10 to 40 digest-worthy items. Keeping these jobs separate puts a hard limit on model work. Every increase to the Stage 2 pool cap increases the number of documents sent to the model, including documents that will later be rejected. Candidate-pool size is therefore a cost-control parameter owned upstream, where retrieval and ranking are cheaper and easier to inspect. The separation also changes the model’s task. Ranking requires the model to compare every candidate against every other candidate in the pool. Pointwise judgment asks a narrower question: is this document relevant and novel for this target? Stage 2 has already removed most of the distractors, so the model can evaluate each remaining document independently. Each verdict can then be stored and inspected later. Listwise LLM ranking also introduces position bias: a model can favor documents partly because of where they occur in the supplied list. An independent pointwise verdict avoids that failure mode because candidate order does not affect the task definition. A survey of LLM-as-a-judge biases documents this issue. Microsoft's UMBRELA evaluation work also found that LLM-based pointwise relevance assessments correlated strongly with human-derived rankings across five years of TREC data. This boundary makes missing coverage diagnosable. When a document is missing from a digest, there are exactly two possibilities: it never entered the pool (a Stage 2 problem with a deterministic, inspectable cause) or it entered the pool and the model judged it out (a Stage 3 problem with a recorded verdict). You never have to ask "did the model even see it?" The Output Contract Stage 3 returns a structured verdict for every candidate. Any field used by downstream code has a closed set of values. Each verdict contains a short rationale, relevance and novelty judgments, a reason code, and a one-sentence summary. rationale and summary are free text. The rationale records why the model reached its conclusion, and the summary is written for the recipient. Downstream services never branch on either field. They consume the enum values. The order of those fields is deliberate. Asking for labels before reasoning can hurt judgment quality because the model commits to an answer and then produces a justification for it. The EMNLP paper Let Me Speak Freely? describes this cost of constrained output formats. We put rationale first so the model can work through the document before returning the typed verdict. We made two deliberate choices in the code below. The model-facing schema and the stored record are separate classes: the model only sees ModelVerdict, while the pipeline wraps its output in Verdict and adds the document ID and status. extra="forbid turns a hallucinated field into a validation failure rather than silently accepting it: Python class VerdictStatus(str, Enum): VALID = "valid" CONTRACT_VIOLATION = "contract_violation" class Relevance(str, Enum): RELEVANT = "relevant" TANGENTIAL = "tangential" # mentions target, no actionable signal IRRELEVANT = "irrelevant" class Novelty(str, Enum): NEW = "new" UPDATE = "update" # known event, new material detail ALREADY_COVERED = "already_covered" class ReasonCode(str, Enum): NEW_CONTRACT_WIN = "new_contract_win" LEADERSHIP_CHANGE = "leadership_change" REGULATORY_ACTION = "regulatory_action" PRODUCT_LAUNCH = "product_launch" FINANCIAL_RESULTS = "financial_results" MARKET_MOVEMENT = "market_movement" OTHER = "other" # Watch this rate for vocabulary gaps class ModelVerdict(BaseModel): model_config = ConfigDict(extra="forbid") rationale: str # Free text, deliberately before enum fields relevance: Relevance novelty: Novelty reason_code: ReasonCode summary: str # Free text for the digest class Verdict(ModelVerdict): doc_id: str # Stamped by the pipeline, never by the model. This field is excluded # from the schema the model sees, so a verdict can't declare itself valid. status: VerdictStatus = VerdictStatus.VALID def parse_verdict(raw: str, doc_id: str) -> Verdict: try: model_verdict = ModelVerdict.model_validate_json(raw) return Verdict(doc_id=doc_id, **model_verdict.model_dump()) except ValidationError as first_error: repaired = repair_call(raw, error=str(first_error)) try: model_verdict = ModelVerdict.model_validate_json(repaired) return Verdict(doc_id=doc_id, **model_verdict.model_dump()) except ValidationError: metrics.increment("stage3.contract_violation") return Verdict( doc_id=doc_id, status=VerdictStatus.CONTRACT_VIOLATION, rationale="[verdict failed validation]", relevance=Relevance.IRRELEVANT, novelty=Novelty.ALREADY_COVERED, reason_code=ReasonCode.OTHER, summary="[verdict failed validation]", ) def belongs_in_digest(verdict: Verdict) -> bool: return ( verdict.status == VerdictStatus.VALID and verdict.relevance == Relevance.RELEVANT and verdict.novelty != Novelty.ALREADY_COVERED ) The contract earns its keep in three places: Ingestion is straightforward: Digest assembly, notification routing, and the frontend receive typed fields instead of free text that each service must interpret independently. Serving a document is the belongs_in_digest filter above: the verdict must be VALID, RELEVANT, and not ALREADY_COVERED. Checking status first ensures that a failed parse cannot masquerade as a model judgment.Debugging becomes a query, instead of an investigation: Every candidate has a stored verdict, with enum fields for filtering and aggregation plus the rationale for human review. Months later, an engineer can identify why a document was included or excluded without reconstructing the original model call.Monitoring is split into two signals: Calculate relevance, novelty, and reason-code distributions from VALID verdicts only. Track CONTRACT_VIOLATION separately. Mixing the two means a parser regression can look like a sudden change in model quality. The valid-verdict distributions are the first line of quality monitoring. A growing share of IRRELEVANT results for one vertical, an unusual increase in ALREADY_COVERED for one company, or an increase in OTHER each gives the team a place to start looking. There is a maintenance cost. New event types and business concepts eventually exceed the initial reason-code list. OTHER gives that gap a measurable home: if its rate rises among valid verdicts, the taxonomy needs review. This is a vocabulary-maintenance signal and not evidence that the model failed to return a valid response. Enums Close the Feedback Loop A thumbs-down on its own is almost useless. It says that a digest item disappointed the user, but not whether the problem was relevance, novelty, summary quality, source quality, or something earlier in the pipeline. The verdict schema lets the feedback UI ask a more specific question. Instead of a free-text "tell us more" box, the prompt can offer a small, closed set of answers that can be joined to the judge's structured verdict: Not relevant to this companyI already knew thisThe summary is inaccurateThe source was not useful The user is not asked to understand the pipeline or diagnose the model. They only identify what went wrong from their perspective. The response is stored beside the model's original verdict in a form that can be queried and aggregated. That makes feedback reconcilable. A user selecting "I already knew this" for an item the model labeled NEW is a novelty disagreement. Aggregate enough of those disagreements and the pattern starts to localize the fault: One company produces repeated staleness feedback: its recent-coverage window may be too short.One publisher produces repeated staleness feedback: the provider may be delivering articles days after the underlying event.Staleness rises across the whole system: the ingestion or digest schedule may be too slow. In the last two cases, the model may have judged the item correctly against the context it received. The defect is that the context did not contain enough recent coverage, or that the document arrived too late to be useful. One class of staleness should never reach the model. When a user has already received coverage of an event, follow-on articles about the same event should be removed from that user's candidate pool by a deterministic lookup against serve history. That belongs in Stage 2. It is cheaper and more reliable than asking Stage 3 to rediscover a fact the system already knows. We learned this the hard way. Our first version used near-duplicate cluster IDs for deduplication, but it did not retain user-level serve history. A story that remained in the news for several days kept resurfacing in the digest through different articles. It became one of the steadiest sources of “already knew this” feedback. The novelty verdict is for the cases that a lookup cannot resolve: a document covers an already-served event, but may contain a material update. A contract win reported on Monday and revisited on Thursday with a disclosed dollar amount is not a duplicate, even though the event is familiar. UPDATE gives the model and digest assembler a distinct outcome for that case. Relevance feedback points elsewhere. If a user marks an item “not relevant” when the model returned RELEVANT with MARKET_MOVEMENT, the document may be a stock-ticker recap that slipped past initial triage. This points to a Stage 1 triage gap; the Stage 3 relevance prompt is working as intended. The enums make those distinctions visible. Free-text feedback would leave a collection of dissatisfied users and an expensive investigation. A shared vocabulary turns recurring complaints into evidence about the pipeline stage that needs work. Develop on the Large Model, Serve on the Small One Model choice in Stage 3 is a tuning decision rather than an architectural one. Candidate generation bounds the number of calls, and the output contract bounds the work inside each call. That lets you change models without changing the rest of the pipeline. We developed the prompt and ran early production on a large-tier frontier model. At that point, the output contract, reason-code vocabulary, and feedback flow were still changing. We wanted one variable we did not have to question: model capability. Debugging a prompt and a model at the same time is miserable. When a verdict is wrong, the cause could be an ambiguous instruction, insufficient target context, a missing reason code, weak novelty context, or a model that cannot reliably follow the task. Starting with the stronger model removes one of those possibilities. Once the contract was settled, we built a gold set of human-labeled candidates and measured the large model against it. We then ran the cost-optimized small model over the same set. The switch was a measured decision: the small model had to preserve the verdict quality required for the product before it received production traffic. The bounded pool made that switch viable. Judging 50 to 100 pre-vetted candidates against a fixed enum contract is narrower than asking a large model to absorb retrieval, ranking, and summarization in an unstructured pipeline. Bound the task before reducing model cost. A cascade was the obvious alternative. Systems such as FrugalGPT send requests to a lower-cost model first and escalate uncertain cases to a stronger one. That pattern can preserve quality while reducing spend when requests vary widely in difficulty. We considered it and skipped it. After Stage 2 bounded the pool and Stage 3 reduced output to a small enum vocabulary, verdict difficulty was relatively uniform. A cascade would have added a confidence estimator, escalation policy, and second production path without much remaining cost to remove. The more immediate savings came from the workload shape: Prompt caching: The instructions, enum definitions, target profile, and recent-coverage context are shared across a target's candidate pool. A stable shared prefix makes prompt caching effective.Batch processing: Daily digest generation is not latency-sensitive. Discounted batch endpoints fit the workload better than synchronous calls, as long as the batch completes before the digest send window.Per-document records: Each candidate produces an independent verdict. Retries, failures, model comparisons, and later reprocessing can happen at the document level instead of rerunning an entire target pool. The order matters. First bound the pool and then define and stabilize the contract. Measure a stronger model against human labels. Only then test a smaller model on the same set. Cost optimization is much easier when the task, failure modes, and acceptance criteria are already known. Auditing Beats Labeling at Scale You cannot label your way to confidence at this scale, but you can audit. The pipeline processes too many candidate documents to build a comprehensive human-labeled corpus or to review every model verdict. A smaller gold set, maintained over time, is enough to calibrate the judge and catch meaningful regressions. The gold set should contain enough candidates to cover the major verdict classes, common edge cases, and the document types that matter most to the product. Each example receives the same fields the model produces: relevance, novelty, and reason code. Measure agreement per field; Cohen's kappa is useful when class imbalance makes raw accuracy look better than the system really is. Calibration decays, so you have to keep redoing it. Re-run the gold set whenever you change the model, the prompt, the reason-code vocabulary, the target-profile format, or the recent-coverage context. The tier-switch evaluation in the previous section is one example: it turned a model-cost decision into a measured comparison against a fixed baseline. Human labels are expensive, so we use the cheaper signals first. The verdict distributions described earlier often reveal a change before anyone reads an individual item: A rising OTHER rate can mean the reason-code vocabulary no longer fits the documents entering the system.A rising IRRELEVANT rate for one vertical can mean the Stage 2 retrieval query or entity aliases are pulling the wrong material.A sharp change in NEW, UPDATE, or ALREADY_COVERED for one company can point to a bad serve-history window, an ingestion delay, or a change in news volume.A rising CONTRACT_VIOLATION rate indicates a schema, prompting, or provider problem. It is not a model-quality signal and should remain separate from verdict distributions. Distribution drift does not prove what broke. It tells you which targets, document types, or pipeline stages deserve investigation. That is enough to direct limited human review where it has the highest value. Thumbs-down data provides another signal, but only for content that was served. That leaves a more dangerous failure mode: a target with little or no coverage. On a typical day, only about 6,000 of our 15,000 users received a digest. For the other 9,000, the system decided that nothing was worth sending. Usually that is correct. When it is wrong, no recipient has an item to downvote. We therefore audit low-coverage targets as well as high-complaint targets. A target that normally generates thirty useful documents a week but receives three may have a broken alias, a failed source feed, an overly strict retrieval threshold, or an upstream classifier rejecting valid material. Those failures are invisible in served-item feedback. The audit loop is deliberately small: Use verdict and feedback distributions to select suspicious targets, sources, and document types.Sample candidates from those pockets, including documents that entered the Stage 3 pool and documents Stage 2 excluded.Have human reviewers apply the same relevance, novelty, and reason-code contract.Compare their labels with model verdicts and upstream exclusion reasons.Fix the stage that owns the failure, then rerun the gold set before changing production behavior. LLM judges do not eliminate human assessment. They make it selective: the contract supplies the categories, distributions identify samples, and human reviewers determine whether the system is still making the judgments the product needs. Production Notes A few practices made this stage workable in production: Make every field that downstream code branches on an enum. Keep free text for the rationale and human-facing summary. Put the rationale before the enum fields so the model can reason before committing to a label.Keep contract failures visible. A response that fails validation should become a stored CONTRACT_VIOLATION record, never a silently dropped candidate or a fake IRRELEVANT judgment. Monitor that rate separately from model-quality metrics.Turn feedback into a debugging signal that points at the pipeline stage responsible. Ask recipients why an item was unhelpful in a small, structured vocabulary. Join that response to the original verdict and look for recurring disagreements by company, source, reason code, and pipeline stage.Stabilize the task before optimizing model cost. Develop the prompt and contract on a capable model, evaluate against a fixed gold set, then test a smaller model against the same set. Otherwise, prompt defects and model capability gaps look identical.Track distributions before reading individual documents. Relevance, novelty, and reason-code shifts identify where human review is most valuable. Keep OTHER under observation, a rising share among valid verdicts means the taxonomy is falling behind the domain.Audit low-coverage targets in addition to the ones with negative user feedback. Thumbs-down feedback exists only for items that were served. A target receiving suspiciously little coverage may have a retrieval, entity-resolution, source, or triage problem that no user can report. Stage 3 must be observable as well as accurate. A model verdict that cannot be stored, queried, compared, and challenged has limited value in a production decision pipeline. Conclusion The architecture in the previous article put the expensive model behind a bounded candidate pool. This article adds the other half: define the verdict before you tune the model. A pointwise judge over a fixed pool is easier to control than a model asked to retrieve, rank, and explain everything at once. An enum-based verdict gives downstream code stable inputs, gives operators something to monitor, and gives user feedback a route back to the stage that owns the problem. The LLM's job is to judge each candidate. The pipeline's job is to make that judgment inspectable: bounded by the pool, typed by the contract, and useful to the systems and people downstream.

Let's be honest about what happens when you give an entire engineering team AI coding assistants. You look at the sprint board, and tickets are moving to "In Review" faster than ever. Your developers are happy. They are writing boilerplate in seconds and generating entire component structures before their morning coffee gets cold. If you measure productivity by the sheer volume of code generated, your team has successfully turned into a factory. But then you look at your deployment frequency. It's flat. Depending on the week, it might actually be trending downward. How can a team be writing code twice as fast, but shipping to production at the exact same speed? Image 1: The AI Productivity Illusion The problem is that output increases, but throughput/deployments remains static or drops. The challenge is how to address this. The answer lies in basic systems engineering. If you optimize a step in a process that isn't the primary bottleneck, your overall throughput doesn't change. We just spent the last two years making typing faster. But typing was never the hardest part of software development. As engineering leaders, we are flying blind if we rely on legacy productivity metrics in an AI-assisted world. The bottleneck hasn't disappeared; it has simply shifted downstream. To actually measure and manage engineering velocity today, you need to abandon "lines of code" and start tracking the new friction points. Here are the three metrics you need to start watching immediately. 1. PR Cycle Time (and the "Rubber-Stamp" Ratio) AI tools are incredible at generating massive blocks of code. They are decidedly not incredible at explaining the architectural reasoning behind why they generated it. When a developer uses an LLM to build a feature, they often submit a massive Pull Request. For the author, it took ten minutes. For the senior engineer assigned to review it, it's a nightmare. Reviewing 500 lines of AI-generated code in a GitHub diff requires significantly more cognitive load than reviewing human-written code, because AI code often lacks a recognizable, human train of thought. The immediate result? Your PRs sit in the queue for days. What to track: Time to First Review / Time to Merge: If this metric is spiking while ticket completion is dropping, your bottleneck is purely code review. You need to enforce strict PR size limits. AI or not, a PR should rarely exceed 300-400 lines of logic. The Rubber-Stamp Ratio: Look for massive PRs that are approved in under five minutes with a simple "LGTM." This means your senior engineers are overwhelmed and are just pushing AI code through without reading it. That is a ticking time bomb for production outages. 2. CI/CD Pipeline Stability and P95 Build Times More code means more tests. AI is perfectly happy to generate fifty unit tests for a single utility function. On the surface, high test coverage looks great to upper management. But if those tests are poorly constructed, overly coupled to implementation details, or reliant on flaky assertions (especially in UI testing), your CI/CD pipeline is going to grind to a halt. We’ve all seen it: a developer merges a feature, and the build fails three times in a row because of a flaky, auto-generated E2E test that didn't properly await a DOM element. What to track: Build Failure Rate (specifically on the main branch): If your developers are moving fast but your pipeline is constantly turning red, your velocity is zero. P95 Build Duration: Is your pipeline taking 45 minutes to run because it's executing thousands of low-value, AI-generated tests? You might need to implement a "Build Gardener" rotation—assigning an engineer each sprint to aggressively prune useless tests, update assertions, and optimize pipeline caching. 3. The Code Churn Rate and Architectural Decay Think of an AI coding assistant as a highly enthusiastic, incredibly fast junior developer. It solves the immediate problem right in front of it. It does not look at the holistic architecture of your application and decide to refactor a base class for long-term scalability. If developers lean too heavily on AI for problem-solving, you end up with massive amounts of duplicated logic and hyper-localized fixes. The code works today, but the architecture rots tomorrow. What to track: Code Churn (Percentage of code rewritten within 3-4 weeks of being merged): High churn in an AI environment usually means developers are using trial-and-error via prompts rather than thinking through the system design. They merge a feature, realize it breaks an edge case, and have the AI write a messy patch the very next week. Tech Debt Ratio vs. Feature Delivery: You have to force developers to slow down and do architectural planning before they open their IDE. If you don't track the time spent paying down debt, the sheer volume of AI-generated code will eventually bankrupt your architecture. The Takeaway AI coding assistants are a massive net positive for the industry. But they are a hyper-efficient engine, and an engine is dangerously useless if you don't upgrade your brakes and steering at the same time. Stop looking at how fast your developers are closing tickets. Start looking at the health of your review culture, the speed of your pipelines, and the stability of your architecture. That is where the real engineering management happens today.

Ever since Swift Concurrency was introduced, its main mission has been clear: keep memory safe without making us write callback hell. But if we’re being honest, context switching-specifically thread hopping-has always been a bit of a head-scratcher. How many times have you marked an async function as nonisolated on a @MainActor class, only to watch it instantly jump off to the cooperative global pool for no obvious reason? Swift 6.2 addresses this head-on with Approachable Concurrency and its underlying flag, NonisolatedNonsendingByDefault. Let’s break down what actually changes under the hood, how @concurrent fits into the picture, and what this all looks like when stepping through real code. What Changes With NonisolatedNonsendingByDefault Before Swift 6.2 (or with Approachable Concurrency turned off), any nonisolated async function would immediately yield its execution to Swift’s global cooperative executor whenever you called await. That meant constant, often unnecessary thread switching. With Approachable Concurrency enabled (APPROACHABLE_CONCURRENCY = YES), that default behavior flips. Ordinary async methods now behave much like their synchronous counterparts. They stay on the caller’s executor by default instead of hopping away. A few quick rules to keep in mind: nonsending: The function isn’t bound to a specific actor’s isolation domain, but it keeps the execution context of whoever called it.@concurrent: The explicit opt-in attribute telling the compiler, “No, seriously, run this on the global concurrent pool.”Good to know: @concurrent automatically implies nonisolated, so writing both is redundant. Comparing the Flags: A Basic Test Let’s look at a straightforward example to see the difference in practice: Swift @MainActor class ViewModel { var title = "Hello" func updateData() { print("1:", Thread.isMain) } nonisolated func helperMethod() async { print("2:", Thread.isMain) } @concurrent func thirdMethod() async { print("3:", Thread.isMain) } } // Calling it from a MainActor context: Task { let viewModel = ViewModel() viewModel.updateData() await viewModel.helperMethod() await viewModel.thirdMethod() } Quick Compiler Tip: When testing thread execution across different isolation contexts, you might run into compiler warnings or errors when accessing Thread.isMainThread. To cleanly check the main thread without triggering actor isolation warnings, use a nonisolated helper extension: Swift extension Thread { static nonisolated var isMain: Bool { Thread.isMainThread } } Here’s what gets printed depending on your project settings: OutputAPPROACHABLE_CONCURRENCY = NOAPPROACHABLE_CONCURRENCY = YES1: updateData()truetrue2: helperMethod()false (Background)true (Main Thread)3: thirdMethod()false (Background)false (Background) What’s happening here? When set to NO: Calling helperMethod() drops off the main actor and executes on a background thread (false).When set to YES: helperMethod() isn’t isolated, but thanks to nonsending, it inherits the caller’s context. Since the calling Task runs on @MainActor, helperMethod() stays right there on the main thread.thirdMethod() is marked @concurrent, so it always hops to a background worker thread regardless of the build setting. Deep Dive: Following the Execution Chain To really see how thread hopping behaves during nested calls and returns, let’s trace a slightly more complex scenario involving a custom global actor: Swift @globalActor actor BackgroundActor { static let shared = BackgroundActor() } @MainActor class ViewModel { var name = "Swift 6" // 1. Synchronous isolated method func runTest() { print("1:", Thread.isMain) Task { await complexHelper() } } // 2. Async nonisolated helper nonisolated func complexHelper() async { print("2:", Thread.isMain) // Jumping over to our custom actor await BackgroundActor.shared.doWork { print("3:", Thread.isMain) } print("4:", Thread.isMain) // Calling a sync nonisolated helper syncHelper() } // 3. Synchronous nonisolated helper nonisolated func syncHelper() { print("5:", Thread.isMain) } } extension BackgroundActor { func doWork(_ operation: @Sendable () -> Void) async { operation() Task { print("6:", Thread.isMain) } } } Side-by-Side Execution Trace: StepAPPROACHABLE_CONCURRENCY = NOAPPROACHABLE_CONCURRENCY = YES1truetrue2falsetrue ← Stays on caller’s thread3falsefalse ← Hopped to BackgroundActor4falsetrue ← Returned to caller context5falsetrue ← Synchronous call from step 46falsefalse ← Task spawned inside BackgroundActor Why steps 2, 4, and 5 change in Swift 6.2: Step 2 (complexHelper): Because the caller is on @MainActor, complexHelperstarts executing on the main thread (true).Step 3 (doWork): We explicitly await a method on BackgroundActor, so execution correctly hops over to a background thread (false).Step 4 (After await doWork): Here’s the key difference. When doWorkfinishes, control resumes in complexHelper. Under Swift 6.2, the method remembers where it was called from, so it hops back to the Main Thread (true).Step 5 (syncHelper): This is a plain synchronous call made right after step 4, so it stays on the main thread (true). Wrapping Up Swift 6.2’s Approachable Concurrency makes writing async Swift feel a lot more natural: Fewer random context switches: Your app spends less time hopping back and forth across threads when it doesn’t need to.Predictable execution: Async code holds onto its caller’s context until you explicitly use @concurrent or call into a different actor.Easier mental model: Async methods now align much closer with how we expect synchronous code to flow, removing a big chunk of the concurrency learning curve.

For the past several years, technical debt was something that was easy to identify. It came in the form of outdated frameworks, missing documentation, messy databases, etc. It was something companies racked up by moving too fast, skipping the best course of action, and patching up old systems instead of improving them. But now, a new kind of technical debt is fast emerging. At first glance, it may not look broken. However, it may even work perfectly fine initially. The app loads just fine, the feature responds, the workflow runs, and the AI-generated module seems to pass the first round of testing. On the surface, everything looks good. But the problem starts later on, when someone asks just one simple but very crucial question: how did this actually work? That is where many modern software teams begin to really feel the weight that comes with this new reality. They get easy and prompt access to a lot of code that functions, but cannot be easily explained or trusted. This is the new technical debt. Working Code Is No Longer Enough In the past, code that worked was a major milestone for any team. If the product worked, that meant that the team could finally move ahead. If the feature could be deployed, that meant the sprint was successful. If the app did not crash, that was a cause for celebration. That mindset was not the best one to have. Modern software does not live or work in isolation. There are several programs that it’s connected to, ranging from CRMs to cloud platforms, third-party APIs, AI models, and more. Just a single feature on the software may involve five or ten systems behind the scenes. So, if the team making the product does not understand how the code works, they cannot confidently tell what went wrong if the product runs into a wall or something changes, A feature that is running perfectly today may become a liability tomorrow if no one understands how it really works. That is why explainability is becoming just as important as functionality. AI Has Made This Problem Bigger With the assistance of AI in software development, the speed at which software is being created has increased monumentally. Developers can now do their tasks, including generating functions, writing test cases, building interfaces, and troubleshooting issues, at a fraction of the time that it took before. But speed alone does not automatically improve software delivery. According to Google’s 2025 DORA report, AI can be an amplifier of an organization's existing strengths and weaknesses, which means it can improve disciplined teams while exposing weaker engineering processes. This is a major advantage when used responsibly. But it also creates a new risk. AI can help come up with code that looks clean, runs correctly, and solves the immediate problems at hand. But is your development team able to follow along with the logic behind what the AI is doing? Are they accepting the AI-generated solution just because it passes a test, or do they understand every trade-off that it makes? AI tools can help a non-technical founder build a working prototype without knowing how secure or scalable it is. Just like that, your team, though moving faster in the short term, is quietly creating systems that they cannot confidently own in the long term. This does not mean AI-generated code is bad. The issue is not the tool. It is blind adoption. This concern is already showing up in the industry. GitLab’s 2026 research found that 73% of respondents are concerned about the maintainability of AI-generated code, while 82% believe it risks creating a new form of technical debt that their organization is not prepared to manage. When you use AI to accelerate development without reviewing it or passing human judgment on it, you are not just building software faster. You are building uncertainty faster. Explainable Software Requires Better Discipline The solution that I see to this is not to slow down innovation, but to build with more clarity. The basis of explainable software comes with strong requirements. Software development teams need to know what a certain feature is supposed to do, what it’s supposed to not do, what systems it connects with, and what happens when something goes wrong. Without this context, despite how well written the code is, it can become dangerous. The next thing to keep in mind is that the architecture behind the software can and should be understood by more than one person. A strong software system should not depend on one developer’s memory. It needs to have a clear structure, documentation, test coverage, and decision record that help future teams easily understand the system. Code reviews should also evolve. You shouldn’t just be reviewing your code for syntax or performance. Teams need to start asking deeper questions like why was this specific approach chosen? What assumptions does it make? What happens if the API fails? Can another developer understand this system six months from now? These questions may seem simple, but they are what separate working software from dependable software. The New Standard: Code That Works and Can Be Explained The old standard was simple: does the code work? The new standard needs to be stronger. Can it be explained? Can it be maintained? Can it be tested? Can it be scaled? Can a new engineer understand it without reverse engineering the entire system? If the answer is no, then your company may unfortunately be a victim of technical debt. It has only managed to hide it behind a working interface. That hidden debt will eventually come due. With AI tools and automated development workflows becoming common, more and more businesses are capitalizing on the opportunities to build software quickly. I get that it’s exciting, but speed without understanding is not progress. The next generation of technical debt will not always look like bad code. It will look like code that works, until the day someone needs to scale it or explain it. And by then, the real cost will become clear.

Most engineering teams working on healthtech applications reach a point where someone asks a question that sounds simple but isn't: How do we make sure a developer testing a new feature can't accidentally access production patient data? The answer determines whether the architecture that follows will be auditable or not. Teams that answer it with process — "we have policies about that" — spend the next 18 months patching access-control gaps that reopen every time a new engineer joins or a new service gets wired in. Teams that answer it architecturally spend a week setting up AWS Organizations correctly and then largely stop thinking about it. This article covers the multi-account architecture pattern for HIPAA-compliant infrastructure — specifically, the account structure decisions that either enforce PHI workload isolation or make it a permanent source of audit findings. Why Single-Account PHI Isolation Fails at the Seams A single AWS account running production, staging, and development workloads creates a specific problem that IAM policies alone cannot fully solve. The issue is not that IAM is insufficient as a technology. IAM policies enforced within an account are only as reliable as the discipline of the people who manage them. A policy that restricts a developer's access to production RDS today can be modified tomorrow by anyone with sufficient IAM permissions. Nothing in the account structure itself prevents the boundary from being crossed. In practice, the gaps show up in predictable ways. A pipeline service role gets broad permissions during a sprint because scoping them properly would have taken an extra hour. An engineer copies an IAM role from staging to production because it was faster than creating a new one. A debugging session in production happens under an account that was supposed to be read-only. None of these are malicious decisions. They are the natural result of putting access control boundaries inside an environment where the people who need to cross them also have the permissions to do so. The access control problem that surfaces during security reviews is almost always this one — not a missing encryption setting or an unpatched vulnerability, but access boundaries that exist on paper and drift in practice. The Multi-Account Model: Enforcement at the Boundary AWS Organizations with a properly structured multi-account hierarchy solves this problem by moving the enforcement point outside the accounts being protected. The boundary is no longer an IAM policy that someone with IAM permissions can modify. It is an account boundary that the engineers inside those accounts cannot cross, enforced by Service Control Policies applied at the organizational unit level. The recommended structure has four organizational units under the root: a Security OU containing a Log Archive account and a Security Tooling account, a Production OU containing only the Production account where PHI workloads run, a Non-Production OU containing Staging and Development accounts, and a Shared Services OU containing the account used for CI/CD pipelines, DNS, and shared tooling. The Production OU sits under its own organizational unit with SCPs that restrict what can happen inside it, regardless of what IAM policies exist within the production account itself. An engineer whose IAM role in the development account grants broad permissions has those permissions scoped to the development account. Crossing into production requires a separate role, in a separate account, with a separate set of credentials. The architectural boundary is the enforcement mechanism, not the IAM policy. The Log Archive account under the Security OU serves a specific purpose: it is the only account to which CloudTrail logs from all other accounts are delivered, and it is an account to which production engineers have no write access. This means the evidence trail for PHI access events cannot be modified by the accounts generating those events - which is exactly what auditors verify when they ask about log integrity. Service Control Policies: What to Enforce at the OU Level SCPs applied to the Production OU are where the architectural enforcement becomes concrete. The first policy prevents anyone inside the production account from disabling CloudTrail, including account administrators: JSON { "Effect": "Deny", "Action": [ "cloudtrail:StopLogging", "cloudtrail:DeleteTrail", "cloudtrail:UpdateTrail" ], "Resource": "*" } CloudTrail continuity across the full audit period is not something that should depend on engineering discipline. It should be architecturally enforced. An account that can leave the organization can escape every SCP applied to it. This policy closes that path: JSON { "Effect": "Deny", "Action": "organizations:LeaveOrganization", "Resource": "*" } PHI that moves outside defined regions may fall outside data residency commitments. This policy locks the production account to specific regions: JSON { "Effect": "Deny", "Action": "*", "Resource": "*", "Condition": { "StringNotEquals": { "aws:RequestedRegion": ["us-east-1", "eu-west-1"] } }, "NotAction": [ "iam:*", "organizations:*", "route53:*", "budgets:*", "waf:*", "cloudfront:*", "globalaccelerator:*", "importexport:*", "support:*", "trustedadvisor:*" ] } EBS encryption is not enforced by default in all account configurations. This policy makes an unencrypted volume impossible to create in the production account: JSON { "Effect": "Deny", "Action": "ec2:RunInstances", "Resource": "arn:aws:ec2:*:*:volume/*", "Condition": { "Bool": { "ec2:Encrypted": "false" } } } Cross-Account Access: The Pattern That Doesn't Create New Gaps Multi-account architecture introduces a problem engineers feel immediately: how does anything talk to anything else? A CI/CD pipeline in the Shared Services account needs to deploy to production. A developer needs read access to production logs during an incident. A monitoring service needs metrics from all accounts. The answer is cross-account IAM roles with tightly scoped trust policies. A role created in the production account with minimum required permissions defines a trust policy that allows only specific principals from specific accounts to assume it, and only under specific conditions like MFA or an external ID: JSON { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::SHARED-SERVICES-ACCOUNT-ID:role/DeploymentRole" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "deployment-pipeline-prod" } } } ] } The deployment role in the Shared Services account can assume the deployment role in production - but only that role, only from that account, and only with the correct external ID. A developer's personal IAM credentials cannot assume it. An engineer who compromises the development account cannot use that foothold to pivot into production. This pattern creates cross-account access without creating a backdoor through the account boundary. The boundary holds because the trust relationship is explicit, narrow, and auditable through CloudTrail - every role assumption generates a log entry in both accounts. What This Architecture Makes Provable The operational argument for multi-account PHI isolation often focuses on security. The architectural argument that matters more for engineering teams dealing with audits and enterprise security reviews is about provability. In a single-account setup, proving that a developer did not touch production PHI during a given period requires auditing IAM policies, CloudTrail logs, and access history, and then arguing that the policies were correctly configured and consistently enforced throughout the period. There is always a gap between what the policy said and what actually happened, and that gap is what auditors probe. In a multi-account setup, the same question has a simpler answer. The developer's credentials are scoped to the development account. The development account has no access to the production account's resources. Access to production PHI requires a separate role assumption that is logged, requires separate credentials, and would appear immediately in CloudTrail. You are not arguing that the configuration was correct. You are pointing to an architectural boundary that makes the question moot. This shift from arguable to verifiable is what separates teams that sail through security reviews from teams that spend three weeks responding to follow-up questions. The Operational Overhead Is Smaller Than It Looks The most common objection to multi-account architecture from engineering teams is overhead. More accounts means more IAM configuration, more billing to reconcile, more consoles to log into. In practice, this friction is front-loaded and largely disappears once the structure is in place. AWS Control Tower reduces the account provisioning overhead significantly - new accounts inherit the correct SCP structure, logging configuration, and security baseline automatically. Account Vending Machine patterns built on top of Service Catalog or Terraform can provision a correctly configured new account in minutes. After the initial setup, adding a new account is not significantly more work than adding a new VPC. The billing concern is resolved through AWS Organizations consolidated billing, where all accounts roll up to a single payment method with unified cost visibility. The console switching concern is resolved through IAM Identity Center, which provides a single sign-on entry point across all accounts in the organization. The overhead that remains is real but small. The alternative - treating IAM policies inside a single account as the primary PHI protection mechanism - creates ongoing operational overhead that grows with the team and never fully goes away. Final Thoughts PHI workload isolation is an architectural problem, not a policy problem. IAM policies enforced inside an account are only as reliable as the operational discipline of the team maintaining them. Account boundaries enforced by SCPs at the organizational level are reliable by construction — they hold regardless of what happens inside the accounts they protect. The multi-account structure described here is not a compliance checkbox. It is the architecture that makes the access control claims in a security review actually true rather than approximately true with caveats. When an auditor asks how you prevent developer access to production PHI, the strongest answer available on AWS is an account boundary that the developer's credentials cannot cross. Building that boundary is a week of work. Not building it is a permanent source of audit findings.

Last spring, I had six small text features to build: flag filler phrases in a draft, score sentence-length variation, format a citation, check a document against a rubric. My first design put all six behind an API route that called a model. It worked in an afternoon. Then I priced it. Anthropic lists Claude Fable 5 at $10 per million input tokens and $50 per million output. A 700-word draft plus instructions runs about 1,500 input tokens, and users hit the button five or six times per session while they edit. The bill is survivable. The rest of the tradeoff is not. Every keystroke a user typed would leave their machine and land in someone else's logs. Every click added 900ms of round trip to something that should feel like a spellchecker. And two runs over identical input returned different advice, which turns "did my edit help?" into an unanswerable question. I rewrote all six as deterministic browser code. No API route, no server, no network. This is what that took, and where the approach breaks. What a Heuristic Actually Catches The honest framing is that heuristics and models solve different problems, and half the features people route to an LLM belong in the first category. A model is worth paying for when the task needs world knowledge or judgment: Is this argument coherent, does this paragraph follow from the last one, is this claim supported? A regular expression cannot do any of that. But "does this text contain the phrase in order to" is a lookup. "How much do sentence lengths vary" is arithmetic. "Should of be capitalized in this title" is a rule from a style manual, written down, unchanged since 2019. Sending those to a probabilistic system buys you latency and nondeterminism in exchange for nothing. The six tools I run in production all fall in the second category. They ship as static pages with inline scripts, no build-time secrets, and no runtime dependencies. Sentence Segmentation Without a Regex You Will Regret Every metric below needs sentence boundaries, so this is the piece to get right first. Splitting on /[.!?]+\s+/ collapses under real prose. Run it over four ordinary lines and watch: Code language: Text Plain Text IN : The file cost $3.50. It shipped on Jan. 5 anyway. naive: ["The file cost $3.50", "It shipped on Jan", "5 anyway."] IN : He said "stop." Then he left. naive: ["He said \"stop.\" Then he left."] One false split, one missed split, and the abbreviation list you are about to write will never end. The browser ships an ICU-backed segmenter instead: Code language: JavaScript JavaScript const SEG = new Intl.Segmenter('en', { granularity: 'sentence' }); const raw = (text) => [...SEG.segment(text)].map((s) => s.segment.trim()).filter(Boolean); ICU gets both of those cases right, along with 9 a.m., decimals and section numbers like 2.1. It has one failure I hit in production, and it is worth knowing before you ship: it breaks after title abbreviations. Code language: Text Plain Text IN : She met Dr. Chen last week. The draft grew by 3.5 pages. ICU : ["She met Dr.", "Chen last week.", "The draft grew by 3.5 pages."] The repair is a merge pass over the output rather than a rewrite of the splitter. If a segment ends in a known title, glue the next one onto it: Code language: JavaScript JavaScript const TITLE_END = /(^|\s)(Dr|Mr|Mrs|Ms|Prof|Sr|Jr|St|vs|Fig|No)\.$/i; function sentences(text) { return raw(text).reduce((out, part) => { const prev = out[out.length - 1]; if (prev && TITLE_END.test(prev)) out[out.length - 1] = `${prev} ${part}`; else out.push(part); return out; }, []); } Verified against the cases above: Code language: Text Plain Text ["Dr. Chen wrote 3.5 pages.", "She revised twice."] ["She met Dr. Chen last week.", "The draft grew by 3.5 pages."] ["The file cost $3.50.", "It shipped on Jan. 5 anyway."] ["We deployed at 9 a.m.", "Nobody noticed."] ["He said \"stop.\"", "Then he left."] ["Prof. Ada Lovelace vs. Mr. Babbage.", "Round one."] That is a twelve-entry list against the open-ended one the naive regex demands, because ICU already covers the numeric and punctuation cases that make abbreviation lists grow. Intl.Segmenter landed in Chrome 87, Safari 14.1 and Firefox 125, so a 2026 audience has it. It also does granularity: 'word', which matters the moment a user writes in Thai or Japanese, where whitespace tokenization returns one enormous token. Guard it if you support older embedded webviews: Code language: JavaScript JavaScript const hasSegmenter = typeof Intl !== 'undefined' && 'Segmenter' in Intl; Phrase Matching That Does Not Fire on Substrings The naive filler checker uses indexOf, then reports "just" inside "adjustment" and loses the user's trust in the first thirty seconds. Build one alternation with word boundaries, compile it once, and keep the phrase list in data rather than code: Code language: JavaScript JavaScript const FILLERS = [ 'in order to', 'it is important to note', 'at the end of the day', 'due to the fact that', 'a wide variety of', 'needless to say', ]; const escapeRe = (s) => s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); const FILLER_RE = new RegExp( '\\b(' + FILLERS.map(escapeRe).join('|') + ')\\b', 'gi' ); function findFillers(text) { return [...text.matchAll(FILLER_RE)].map((m) => ({ phrase: m[0], index: m.index, })); } Two details that cost me a rewrite. Compile the RegExp outside the function, because a global-flagged regex carries lastIndex state and rebuilding it per call hides that bug instead of fixing it. And use matchAll rather than a while (re.exec()) loop, which is where that state bites. The phrase list is the whole product here. Mine came from marking up 200 real drafts by hand, not from asking a model what filler looks like. Measuring Variation, and the Trap Next to It Uniform sentence length reads as flat prose. The metric is standard deviation over word counts: Code language: JavaScript JavaScript function rhythm(text) { const lens = sentences(text).map((s) => s.split(/\s+/).length); if (lens.length < 2) return null; const mean = lens.reduce((a, b) => a + b, 0) / lens.length; const variance = lens.reduce((a, n) => a + (n - mean) ** 2, 0) / lens.length; return { mean, sd: Math.sqrt(variance), count: lens.length }; } Low standard deviation is a useful writing signal. It is also, and this is where teams get into trouble, one of the two features commercial AI-text detectors lean on, alongside token-level perplexity. Do not ship it as one. A peer-reviewed study in Patterns tested seven commercial detectors and found they misclassified more than half of TOEFL essays written by non-native English speakers as machine-generated, while scoring near-perfect on native-speaker samples (full text). Steady sentence patterns are what a second-language writer produces under pressure. If your product tells that user their own writing looks synthetic, you have built a discrimination engine with a progress bar on it. Report the number as rhythm. Let the writer decide. Make "No Network" a Test, Not a Promise Claiming a tool runs locally is easy. Proving it survives the next dependency bump is the engineering. Two layers. Content Security Policy on the tool pages: Code language: HTML HTML <meta data-fr-http-equiv="Content-Security-Policy" content="default-src 'self'; connect-src 'none'; img-src 'self' data:;"> connect-src 'none' kills fetch, XMLHttpRequest, WebSocket and sendBeacon. If you run first-party analytics on the same origin, drop to connect-src 'self' and lean harder on the second layer. That second layer is a Playwright spec that fails the build if anything leaves the origin: Code language: JavaScript JavaScript test('clarity checker makes no offsite requests', async ({ page }) => { const offsite = []; page.on('request', (req) => { if (new URL(req.url()).origin !== BASE) offsite.push(req.url()); }); await page.goto(`${BASE}/tools/clarity-checker/`); await page.fill('#draft', 'In order to be clear, it is important to note this.'); await page.click('#analyze'); expect(offsite).toEqual([]); }); This caught a real regression for me: a font subset I added later pulled from a CDN, which meant the browser advertised the visitor's IP and user agent to a third party on a page whose whole selling point was that nothing left the device. The CSP would have blocked the request in a browser that enforced it. The test told me before a user did. The Comparison, With Numbers LLM API routeBrowser heuristicFirst response600–1,200 msunder 5 msMarginal cost~$0.001 per runzeroSame input, same outputnoyesUser text leaves deviceyesnoWorks offlinenoyesHandles novel phrasingyesnoJudges argument qualityyesnoShips without a backendnoyes The last row decided it for me. Six static pages on a CDN have no runtime to patch, no key to rotate, and no bill that scales with traffic. When to Call the Model Anyway I still reach for one, on three conditions. The task needs judgment rather than lookup. Restructuring an argument, catching a claim the writer never supported, spotting that paragraph four repeats paragraph two. No word list gets there. The user asked for it explicitly, with the data boundary stated in plain language on the button. Silent exfiltration dressed as a feature is how teams end up in a compliance review. And the output gets checked. For anything structured, constrain the response with a schema and validate it before it touches your UI, because a model that returns prose where your parser expects an object will do it on a Friday. Everything else stayed in the browser. Six features, roughly 400 lines of JavaScript total, zero infrastructure, and a p99 that is a rounding error. The default in 2026 is to reach for an API key first. Check whether the problem is a lookup before you do.

Ground Truth for AI-Written Code Session capture, per-line attribution, and selection-bias-free agent benchmarks, on top of the Git host you already use. A technical overview for engineers and engineering leaders evaluating how much of their codebase is now written by AI agents - and who is accountable for it. 1. The Problem: Git Blame No Longer Tells the Truth On most teams, AI agents now write a large share of new code. But the tools that record who wrote what were built for humans. When an agent edits files in your working tree and you commit them, git blame attributes every one of those lines to you. The prompt that produced them, the model that ran, the cost, the number of turns, and whether the code survived the next sprint — none of it is recorded anywhere. That gap has real consequences: Provenance – no answer to “which agent, from which prompt, wrote this line?” during review or an incident.Cost and efficiency – no ground truth on what a feature cost in tokens and dollars, or which agent got therein fewer turns.Quality – no measure of whether agent-written code survives, or gets reworked and reverted days later.Comparison – “which agent is better for us?” answered by vibes, because every naive comparison is poisoned by selection bias (the hard tasks go to the agent you already trust). Origin closes that gap. It captures the full agent session — prompt, diff, tokens, cost, tools, duration — attributes every surviving line back to an agent and a prompt using Git as the source of truth, and turns that data into honest, selection-bias-free comparisons between agents. It runs on top of GitHub or GitLab; there is nothing to migrate. 2. How Origin Captures an Agent Session Capture is deliberately boring and durable. A one-time origin enable registers the machine, auto-detects installed agents (Claude Code, Codex, Cursor, GitHub Copilot, Gemini, Aider, Devin, Antigravity, and more), and installs two kinds of listeners: Agent hooks – Origin hooks fire on the agent’s lifecycle events (session start, each user prompt, eachtool/file edit, and stop/end). They record the prompt text, the per-turn file diff, token and cost counters, tool calls, and the model.Transcript watchers – for agents that keep a durable on-disk transcript (e.g., Codex’s rollout logs, Devin’s local session DB), Origin reads that record directly instead of depending on hooks. The principle: if there is an authoritative transcript, read it; hooks are for context and policy. Capture is resilient by design. It writes locally first, retries on a durable queue when the network is down, resolves session end from heartbeat liveness rather than a fragile inactivity timer, and is aware of Git work trees so parallel sessions don’t collide. Sessions that never produced real work are swept so counts reflect reality. Figure 1. Every AI coding session Origin captured — agent, model, cost, tokens, branch, and review status. This is the raw material everything else is built on. 3. Attribution: First-Author Wins, With Git as the Source of Truth Recording a session is easy; attributing lines correctly is the hard part, and it is where Origin is opinionated. The model is first-author-wins: a line is credited to whoever introduced it, and later edits never reclaim it. For pushed commits, Git is the ground truth — Origin reconciles its capture against the committed diff rather than trusting a possibly lossy hook stream. A suite of invariants guards the accounting so numbers never drift: InvariantWhat it guaranteesFirst-author-winsA line counts once, for its original author - no double-credit when it’s later touched.Git-truth reconciliationPushed-commit line counts come from the real diff, not the (lossy) live hookstream.Hunk-aware countingAdd/remove tallies parse diff hunks correctly; content lines aren’t miscounted.Writes never claim linesA write/format/no-op operation cannot claim authorship it didn’t earn.Missing-commit self-healA commit-and-exit race is reconstructed at read time from the transcript-attested SHA. Figure 2. One session, decomposed: each prompt and its diff (committed vs uncommitted), the linked commit, and a 100%-AI verdict - the ground truth per-line blame is built from. The AI Blame tab drills to the line level. 4. Prompt-Level Time Travel Because Origin records the state before every prompt, each prompt becomes a restore point. You can undo an agent’s changes — the files revert — without rewriting or losing your commits. 5. Honest Benchmarking: The Agent Scorecard Once sessions are captured and attributed, Origin computes a per-agent scorecard — efficiency, outcome, and survival — for your real work. The point of difference is honesty: the scorecard refuses to draw conclusions the data can’t support. MetricDefinitionCost/taskMean cost per completed session for the agent.Tokens/produced lineToken spend normalized to lines that actually shipped.Median turnsHow many prompts it took to finish - lower is tighter.First-pass approvalShare of reviewed sessions approved without changes.Cost/merged PRDollars per PR that actually merged (outcome, not activity).Code survival @ 7/30dFraction of authored lines still present a week/month later.Rework rateThe inverse - how much of the agent’s output got reverted or rewritten. The guardrails matter as much as the metrics: Minimum sample size – agents aren’t ranked on a handful of sessions; below a threshold, a metric is shown as “not enough data,” not a misleading average.Confidence intervals – ratio metrics (e.g., tokens-per-line) carry a CI, so a noisy small sample can’t masquerade as a clear winner.Estimated tokens excluded – sessions whose token counts were estimated rather than reported are flagged and kept out of the money math.Line-weighted authorship – the AI-vs-human percentage is weighted by lines, not session count, so one giant human commit doesn’t get outvoted by many tiny agent ones. Figure 3. The agent scorecard — cost, tokens-per-line, median turns, approval, and survival per agent, with sample-size and confidence guardrails. 6. Bake-Offs: The Selection-Bias-Free Comparison The scorecard measures agents on the work you happened to give them — and you give the hard tasks to the agent you trust, which skews every comparison. A bake-off removes that bias by construction: it runs the same prompt through N agents, each in its own isolated Git work tree, and lets you compare the results side by side. Every arm gets identical work. Architecture: The Server Schedules, Your Machine Executes Coding agents run on your machine, with your keys — Origin’s cloud can never run them. So a bake-off is split cleanly in two: the server owns the queue and the schedule; a local runner daemon owns execution. Each arm branches from HEAD into bakeoff/<id>/<agent>, the agent works autonomously and commits, and Origin correlates the result back to the branch via normal session capture — nothing extra to wire up. The list nests each arm’s session inline (cost, tokens, lines, status), filters by status/repo/agent, pages ten at a time, and rolls up a head-to-head agent comparison across every bake-off you’ve run. Deliberately, Origin does not auto-declare a winner. It tints the cheapest and fewest-turns arms to help you scan, but “cheapest” and “best” are not the same thing - only a human reading the diff can decide. You pick the winner.Figure 4. Composing a bake-off — the same prompt, two or more agents, each running autonomously in its own git work tree. Results stream back as sessions and roll up into a head-to-head comparison. 7. For Teams: Governance Without a Second Source of Truth Everything above is per-developer value that also aggregates for a team. On top of it, Origin adds an org layer: typed policies enforced across review, PR checks, and CI; AI auto-review of agent sessions; secret and PII scanning on captured diffs; budgets and cost controls with per-agent visibility; role-based access; and an organization dashboard that shows what share of the codebase is AI-authored, by whom, at what cost - line-weighted, not guessed. Because attribution is per line and travels with the repo (prompts are carried in Git notes, and a dedicated sessions branch makes context portable across clones), the governance view is derived from the same ground truth developers see — not a parallel system that drifts. 8. Architecture, Privacy, and Getting Started Local-first capture – session data is recorded on your machine first. A fully standalone mode keeps everything in the repo with no account at all.Sits on your host – GitHub and GitLab, multiple connected accounts, native Windows/macOS/Linux CLI.No repo migration.Portable provenance – prompts live in Git notes; the origin-sessions branch is a zero-tooling vehicle, so a fresh clone still has the history.CLI-native – the CLI is a single Node binary distributed via signed GitHub releases; the platform API runs on a small, boring stack (Express + Prisma). Origin turns the invisible half of your codebase — the half an agent wrote — into something you can read, attribute, price, and compare. Solo, it’s your provenance and undo button. For a team, it’s the ground truth under every AI-code decision.