Ground Truth for AI-Written Code: Why Context Matters More Than Prompts

Ground Truth for AI-Written Code

Session capture, per-line attribution, and selection-bias-free agent benchmarks, on top of the Git host you already use.

A technical overview for engineers and engineering leaders evaluating how much of their codebase is now written by AI agents - and who is accountable for it.

1. The Problem: Git Blame No Longer Tells the Truth

On most teams, AI agents now write a large share of new code. But the tools that record who wrote what were built for humans. When an agent edits files in your working tree and you commit them, git blame attributes every one of those lines to you. The prompt that produced them, the model that ran, the cost, the number of turns, and whether the code survived the next sprint — none of it is recorded anywhere.

That gap has real consequences:

• Provenance – no answer to “which agent, from which prompt, wrote this line?” during review or an incident.
• Cost and efficiency – no ground truth on what a feature cost in tokens and dollars, or which agent got therein fewer turns.
• Quality – no measure of whether agent-written code survives, or gets reworked and reverted days later.
• Comparison – “which agent is better for us?” answered by vibes, because every naive comparison is poisoned by selection bias (the hard tasks go to the agent you already trust).

Origin closes that gap. It captures the full agent session — prompt, diff, tokens, cost, tools, duration — attributes every surviving line back to an agent and a prompt using Git as the source of truth, and turns that data into honest, selection-bias-free comparisons between agents. It runs on top of GitHub or GitLab; there is nothing to migrate.

2. How Origin Captures an Agent Session

Capture is deliberately boring and durable. A one-time origin enable registers the machine, auto-detects installed agents (Claude Code, Codex, Cursor, GitHub Copilot, Gemini, Aider, Devin, Antigravity, and more), and installs two kinds of listeners:

• Agent hooks – Origin hooks fire on the agent’s lifecycle events (session start, each user prompt, eachtool/file edit, and stop/end). They record the prompt text, the per-turn file diff, token and cost counters, tool calls, and the model.
• Transcript watchers – for agents that keep a durable on-disk transcript (e.g., Codex’s rollout logs, Devin’s local session DB), Origin reads that record directly instead of depending on hooks. The principle: if there is an authoritative transcript, read it; hooks are for context and policy.

Capture is resilient by design. It writes locally first, retries on a durable queue when the network is down, resolves session end from heartbeat liveness rather than a fragile inactivity timer, and is aware of Git work trees so parallel sessions don’t collide. Sessions that never produced real work are swept so counts reflect reality.

Figure 1. Every AI coding session Origin captured — agent, model, cost, tokens, branch, and review status. This is the raw material everything else is built on.

3. Attribution: First-Author Wins, With Git as the Source of Truth

Recording a session is easy; attributing lines correctly is the hard part, and it is where Origin is opinionated. The model is first-author-wins: a line is credited to whoever introduced it, and later edits never reclaim it. For pushed commits, Git is the ground truth — Origin reconciles its capture against the committed diff rather than trusting a possibly lossy hook stream.

A suite of invariants guards the accounting so numbers never drift:

Figure 2. One session, decomposed: each prompt and its diff (committed vs uncommitted), the linked commit, and a 100%-AI verdict - the ground truth per-line blame is built from. The AI Blame tab drills to the line level.

4. Prompt-Level Time Travel

Because Origin records the state before every prompt, each prompt becomes a restore point. You can undo an agent’s changes — the files revert — without rewriting or losing your commits.

5. Honest Benchmarking: The Agent Scorecard

Once sessions are captured and attributed, Origin computes a per-agent scorecard — efficiency, outcome, and survival — for your real work. The point of difference is honesty: the scorecard refuses to draw conclusions the data can’t support.

The guardrails matter as much as the metrics:

• Minimum sample size – agents aren’t ranked on a handful of sessions; below a threshold, a metric is shown as “not enough data,” not a misleading average.
• Confidence intervals – ratio metrics (e.g., tokens-per-line) carry a CI, so a noisy small sample can’t masquerade as a clear winner.
• Estimated tokens excluded – sessions whose token counts were estimated rather than reported are flagged and kept out of the money math.
• Line-weighted authorship – the AI-vs-human percentage is weighted by lines, not session count, so one giant human commit doesn’t get outvoted by many tiny agent ones.

Figure 3. The agent scorecard — cost, tokens-per-line, median turns, approval, and survival per agent, with sample-size and confidence guardrails.

6. Bake-Offs: The Selection-Bias-Free Comparison

The scorecard measures agents on the work you happened to give them — and you give the hard tasks to the agent you trust, which skews every comparison. A bake-off removes that bias by construction: it runs the same prompt through N agents, each in its own isolated Git work tree, and lets you compare the results side by side. Every arm gets identical work.

Architecture: The Server Schedules, Your Machine Executes

Coding agents run on your machine, with your keys — Origin’s cloud can never run them. So a bake-off is split cleanly in two: the server owns the queue and the schedule; a local runner daemon owns execution.

Each arm branches from HEAD into bakeoff/<id>/<agent>, the agent works autonomously and commits, and Origin correlates the result back to the branch via normal session capture — nothing extra to wire up. The list nests each arm’s session inline (cost, tokens, lines, status), filters by status/repo/agent, pages ten at a time, and rolls up a head-to-head agent comparison across every bake-off you’ve run.

Deliberately, Origin does not auto-declare a winner. It tints the cheapest and fewest-turns arms to help you scan, but “cheapest” and “best” are not the same thing - only a human reading the diff can decide. You pick the winner.
Figure 4. Composing a bake-off — the same prompt, two or more agents, each running autonomously in its own git work tree. Results stream back as sessions and roll up into a head-to-head comparison.

7. For Teams: Governance Without a Second Source of Truth 

Everything above is per-developer value that also aggregates for a team. On top of it, Origin adds an org layer: typed policies enforced across review, PR checks, and CI; AI auto-review of agent sessions; secret and PII scanning on captured diffs; budgets and cost controls with per-agent visibility; role-based access; and an organization dashboard that shows what share of the codebase is AI-authored, by whom, at what cost - line-weighted, not guessed.

Because attribution is per line and travels with the repo (prompts are carried in Git notes, and a dedicated sessions branch makes context portable across clones), the governance view is derived from the same ground truth developers see — not a parallel system that drifts.

8. Architecture, Privacy, and Getting Started 

• Local-first capture – session data is recorded on your machine first. A fully standalone mode keeps everything in the repo with no account at all.
• Sits on your host – GitHub and GitLab, multiple connected accounts, native Windows/macOS/Linux CLI.No repo migration.
• Portable provenance – prompts live in Git notes; the origin-sessions branch is a zero-tooling vehicle, so a fresh clone still has the history.
• CLI-native – the CLI is a single Node binary distributed via signed GitHub releases; the platform API runs on a small, boring stack (Express + Prisma). 

Origin turns the invisible half of your codebase — the half an agent wrote — into something you can read, attribute, price, and compare. Solo, it’s your provenance and undo button. For a team, it’s the ground truth under every AI-code decision.