LLM Judgment for Document Pipelines: Bounded Pools and Typed Verdicts

This project creates a daily digest for sellers in an enterprise system. Each seller handles accounts at a set of companies and needs to know when something happens at one of them: a product launch, leadership change, new contract, or funding round. That news is often an opening for the next conversation. The pipeline reads the day's incoming news and sends each seller a short email with the handful of articles worth their time about the companies they cover. Every item in that email carries a thumbs-up and a thumbs-down button.

At launch, we set a simple quality target. From historical user behavior on other surfaces, we knew that about 9% of served items would receive a user vote. Among those votes, we wanted 70% to be thumbs up. We expected that approval rate to show whether the pipeline was improving.

A month later, the dashboard could report vote counts but not explain them. Every downvote triggered the same manual investigation: retrieve the document, retrieve the target profile, read both, and infer the cause.

The possible causes belonged to different parts of the pipeline. A user may already have seen the story earlier that week. An article may concern the right parent company but the wrong business unit, such as iPad coverage sent to a seller responsible for iPhone accounts. A technically relevant article from a low-quality publisher may contain no useful signal. A stock-ticker recap may mention the company accurately and still offer nothing useful to the recipient. The feedback system reduced all of these outcomes to the same negative signal.

The cost of this ambiguity became clear at production volume. The ingestion layer processed 700,000 to 1 million articles per day across more than 20,000 target entities. Candidate generation reduced that corpus to a bounded pool before any model call. To fix this, we changed the output contract between the judging model and the downstream pipeline.

In a previous article, I described the system's three stages: cost-efficient triage, target-aware retrieval, and bounded LLM reasoning. Candidate generation controls most of the pipeline’s cost because it determines how many documents reach the model. This article focuses on Stage 3: the judgments the model should make, the structured verdict it returns, and how those verdicts make user feedback auditable.

Judging vs Selecting

The single most important design decision in Stage 3 is what job the LLM is not allowed to do: selection.

For each target, Stage 2 supplies at most 100 candidates. The model evaluates each document independently and returns a verdict: whether it is relevant to the target, whether the event is new to that user's digest, and whether it should be sent in the email. A pool of 100 candidates usually produces 10 to 40 digest-worthy items.

Keeping these jobs separate puts a hard limit on model work. Every increase to the Stage 2 pool cap increases the number of documents sent to the model, including documents that will later be rejected. Candidate-pool size is therefore a cost-control parameter owned upstream, where retrieval and ranking are cheaper and easier to inspect.

The separation also changes the model’s task. Ranking requires the model to compare every candidate against every other candidate in the pool. Pointwise judgment asks a narrower question: is this document relevant and novel for this target? Stage 2 has already removed most of the distractors, so the model can evaluate each remaining document independently. Each verdict can then be stored and inspected later.

Listwise LLM ranking also introduces position bias: a model can favor documents partly because of where they occur in the supplied list. An independent pointwise verdict avoids that failure mode because candidate order does not affect the task definition. A survey of LLM-as-a-judge biases documents this issue. Microsoft's UMBRELA evaluation work also found that LLM-based pointwise relevance assessments correlated strongly with human-derived rankings across five years of TREC data.

This boundary makes missing coverage diagnosable. When a document is missing from a digest, there are exactly two possibilities: it never entered the pool (a Stage 2 problem with a deterministic, inspectable cause) or it entered the pool and the model judged it out (a Stage 3 problem with a recorded verdict). You never have to ask "did the model even see it?"

The Output Contract

Stage 3 returns a structured verdict for every candidate. Any field used by downstream code has a closed set of values.

Each verdict contains a short rationale, relevance and novelty judgments, a reason code, and a one-sentence summary. rationale and summary are free text. The rationale records why the model reached its conclusion, and the summary is written for the recipient. Downstream services never branch on either field. They consume the enum values.

The order of those fields is deliberate. Asking for labels before reasoning can hurt judgment quality because the model commits to an answer and then produces a justification for it. The EMNLP paper Let Me Speak Freely? describes this cost of constrained output formats. We put rationale first so the model can work through the document before returning the typed verdict.

We made two deliberate choices in the code below. The model-facing schema and the stored record are separate classes: the model only sees ModelVerdict, while the pipeline wraps its output in Verdict and adds the document ID and status. extra="forbid turns a hallucinated field into a validation failure rather than silently accepting it:

class VerdictStatus(str, Enum):
    VALID = "valid"
    CONTRACT_VIOLATION = "contract_violation"

class Relevance(str, Enum):
    RELEVANT = "relevant"
    TANGENTIAL = "tangential"        # mentions target, no actionable signal
    IRRELEVANT = "irrelevant"

class Novelty(str, Enum):
    NEW = "new"
    UPDATE = "update"                # known event, new material detail
    ALREADY_COVERED = "already_covered"

class ReasonCode(str, Enum):
    NEW_CONTRACT_WIN = "new_contract_win"
    LEADERSHIP_CHANGE = "leadership_change"
    REGULATORY_ACTION = "regulatory_action"
    PRODUCT_LAUNCH = "product_launch"
    FINANCIAL_RESULTS = "financial_results"
    MARKET_MOVEMENT = "market_movement"
    OTHER = "other"                  # Watch this rate for vocabulary gaps

class ModelVerdict(BaseModel):
    model_config = ConfigDict(extra="forbid")

    rationale: str                   # Free text, deliberately before enum fields
    relevance: Relevance
    novelty: Novelty
    reason_code: ReasonCode
    summary: str                     # Free text for the digest

class Verdict(ModelVerdict):
    doc_id: str
    # Stamped by the pipeline, never by the model. This field is excluded
    # from the schema the model sees, so a verdict can't declare itself valid.
    status: VerdictStatus = VerdictStatus.VALID

def parse_verdict(raw: str, doc_id: str) -> Verdict:
    try:
        model_verdict = ModelVerdict.model_validate_json(raw)
        return Verdict(doc_id=doc_id, **model_verdict.model_dump())
    except ValidationError as first_error:
        repaired = repair_call(raw, error=str(first_error))
        try:
            model_verdict = ModelVerdict.model_validate_json(repaired)
            return Verdict(doc_id=doc_id, **model_verdict.model_dump())
        except ValidationError:
            metrics.increment("stage3.contract_violation")
            return Verdict(
                doc_id=doc_id,
                status=VerdictStatus.CONTRACT_VIOLATION,
                rationale="[verdict failed validation]",
                relevance=Relevance.IRRELEVANT,
                novelty=Novelty.ALREADY_COVERED,
                reason_code=ReasonCode.OTHER,
                summary="[verdict failed validation]",
            )

def belongs_in_digest(verdict: Verdict) -> bool:
    return (
        verdict.status == VerdictStatus.VALID
        and verdict.relevance == Relevance.RELEVANT
        and verdict.novelty != Novelty.ALREADY_COVERED
    )

The contract earns its keep in three places:

• Ingestion is straightforward: Digest assembly, notification routing, and the frontend receive typed fields instead of free text that each service must interpret independently. Serving a document is the belongs_in_digest filter above: the verdict must be VALID, RELEVANT, and not ALREADY_COVERED. Checking status first ensures that a failed parse cannot masquerade as a model judgment.
• Debugging becomes a query, instead of an investigation: Every candidate has a stored verdict, with enum fields for filtering and aggregation plus the rationale for human review. Months later, an engineer can identify why a document was included or excluded without reconstructing the original model call.
• Monitoring is split into two signals: Calculate relevance, novelty, and reason-code distributions from VALID verdicts only. Track CONTRACT_VIOLATION separately. Mixing the two means a parser regression can look like a sudden change in model quality.

The valid-verdict distributions are the first line of quality monitoring. A growing share of IRRELEVANT results for one vertical, an unusual increase in ALREADY_COVERED for one company, or an increase in OTHER each gives the team a place to start looking.

There is a maintenance cost. New event types and business concepts eventually exceed the initial reason-code list. OTHER gives that gap a measurable home: if its rate rises among valid verdicts, the taxonomy needs review. This is a vocabulary-maintenance signal and not evidence that the model failed to return a valid response.

Enums Close the Feedback Loop

A thumbs-down on its own is almost useless. It says that a digest item disappointed the user, but not whether the problem was relevance, novelty, summary quality, source quality, or something earlier in the pipeline.

The verdict schema lets the feedback UI ask a more specific question. Instead of a free-text "tell us more"
box, the prompt can offer a small, closed set of answers that can be joined to the judge's structured verdict:

• Not relevant to this company
• I already knew this
• The summary is inaccurate
• The source was not useful

The user is not asked to understand the pipeline or diagnose the model. They only identify what went wrong from their perspective. The response is stored beside the model's original verdict in a form that can be queried and aggregated.

That makes feedback reconcilable. A user selecting "I already knew this" for an item the model labeled NEW is a novelty disagreement. Aggregate enough of those disagreements and the pattern starts to localize the fault:

• One company produces repeated staleness feedback: its recent-coverage window may be too short.
• One publisher produces repeated staleness feedback: the provider may be delivering articles days after the underlying event.
• Staleness rises across the whole system: the ingestion or digest schedule may be too slow.

In the last two cases, the model may have judged the item correctly against the context it received. The defect is that the context did not contain enough recent coverage, or that the document arrived too late to be useful.

One class of staleness should never reach the model. When a user has already received coverage of an event, follow-on articles about the same event should be removed from that user's candidate pool by a deterministic lookup against serve history. That belongs in Stage 2. It is cheaper and more reliable than asking Stage 3 to rediscover a fact the system already knows.

We learned this the hard way. Our first version used near-duplicate cluster IDs for deduplication, but it did not retain user-level serve history. A story that remained in the news for several days kept resurfacing in the digest through different articles. It became one of the steadiest sources of “already knew this” feedback.

The novelty verdict is for the cases that a lookup cannot resolve: a document covers an already-served event, but may contain a material update. A contract win reported on Monday and revisited on Thursday with a disclosed dollar amount is not a duplicate, even though the event is familiar. UPDATE gives the model and digest assembler a distinct outcome for that case.

Relevance feedback points elsewhere. If a user marks an item “not relevant” when the model returned RELEVANT with MARKET_MOVEMENT, the document may be a stock-ticker recap that slipped past initial triage. This points to a Stage 1 triage gap; the Stage 3 relevance prompt is working as intended.

The enums make those distinctions visible. Free-text feedback would leave a collection of dissatisfied users and an expensive investigation. A shared vocabulary turns recurring complaints into evidence about the pipeline stage that needs work.

Develop on the Large Model, Serve on the Small One

Model choice in Stage 3 is a tuning decision rather than an architectural one. Candidate generation bounds the number of calls, and the output contract bounds the work inside each call. That lets you change models without changing the rest of the pipeline.

We developed the prompt and ran early production on a large-tier frontier model. At that point, the output contract, reason-code vocabulary, and feedback flow were still changing. We wanted one variable we did not have to question: model capability.

Debugging a prompt and a model at the same time is miserable. When a verdict is wrong, the cause could be an ambiguous instruction, insufficient target context, a missing reason code, weak novelty context, or a model that cannot reliably follow the task. Starting with the stronger model removes one of those possibilities.

Once the contract was settled, we built a gold set of human-labeled candidates and measured the large model against it. We then ran the cost-optimized small model over the same set. The switch was a measured decision: the small model had to preserve the verdict quality required for the product before it received production traffic.

The bounded pool made that switch viable. Judging 50 to 100 pre-vetted candidates against a fixed enum contract is narrower than asking a large model to absorb retrieval, ranking, and summarization in an unstructured pipeline. Bound the task before reducing model cost.

A cascade was the obvious alternative. Systems such as FrugalGPT send requests to a lower-cost model first and escalate uncertain cases to a stronger one. That pattern can preserve quality while reducing spend when requests vary widely in difficulty.

We considered it and skipped it. After Stage 2 bounded the pool and Stage 3 reduced output to a small enum vocabulary, verdict difficulty was relatively uniform. A cascade would have added a confidence estimator, escalation policy, and second production path without much remaining cost to remove.

The more immediate savings came from the workload shape:

• Prompt caching: The instructions, enum definitions, target profile, and recent-coverage context are shared across a target's candidate pool. A stable shared prefix makes prompt caching effective.
• Batch processing: Daily digest generation is not latency-sensitive. Discounted batch endpoints fit the workload better than synchronous calls, as long as the batch completes before the digest send window.
• Per-document records: Each candidate produces an independent verdict. Retries, failures, model comparisons, and later reprocessing can happen at the document level instead of rerunning an entire target pool.

The order matters. First bound the pool and then define and stabilize the contract. Measure a stronger model against human labels. Only then test a smaller model on the same set. Cost optimization is much easier when the task, failure modes, and acceptance criteria are already known.

Auditing Beats Labeling at Scale

You cannot label your way to confidence at this scale, but you can audit. The pipeline processes too many candidate documents to build a comprehensive human-labeled corpus or to review every model verdict. A smaller gold set, maintained over time, is enough to calibrate the judge and catch meaningful regressions.

The gold set should contain enough candidates to cover the major verdict classes, common edge cases, and the document types that matter most to the product. Each example receives the same fields the model produces: relevance, novelty, and reason code. Measure agreement per field; Cohen's kappa is useful when class imbalance makes raw accuracy look better than the system really is.

Calibration decays, so you have to keep redoing it. Re-run the gold set whenever you change the model, the prompt, the reason-code vocabulary, the target-profile format, or the recent-coverage context. The tier-switch evaluation in the previous section is one example: it turned a model-cost decision into a measured comparison against a fixed baseline.

Human labels are expensive, so we use the cheaper signals first. The verdict distributions described earlier often reveal a change before anyone reads an individual item:

• A rising OTHER rate can mean the reason-code vocabulary no longer fits the documents entering the system.
• A rising IRRELEVANT rate for one vertical can mean the Stage 2 retrieval query or entity aliases are pulling the wrong material.
• A sharp change in NEW, UPDATE, or ALREADY_COVERED for one company can point to a bad serve-history window, an ingestion delay, or a change in news volume.
• A rising CONTRACT_VIOLATION rate indicates a schema, prompting, or provider problem. It is not a model-quality signal and should remain separate from verdict distributions.

Distribution drift does not prove what broke. It tells you which targets, document types, or pipeline stages deserve investigation. That is enough to direct limited human review where it has the highest value.

Thumbs-down data provides another signal, but only for content that was served. That leaves a more dangerous failure mode: a target with little or no coverage. On a typical day, only about 6,000 of our 15,000 users received a digest. For the other 9,000, the system decided that nothing was worth sending. Usually that is correct. When it is wrong, no recipient has an item to downvote.

We therefore audit low-coverage targets as well as high-complaint targets. A target that normally generates thirty useful documents a week but receives three may have a broken alias, a failed source feed, an overly strict retrieval threshold, or an upstream classifier rejecting valid material. Those failures are invisible in served-item feedback.

The audit loop is deliberately small:

• Use verdict and feedback distributions to select suspicious targets, sources, and document types.
• Sample candidates from those pockets, including documents that entered the Stage 3 pool and documents Stage 2 excluded.
• Have human reviewers apply the same relevance, novelty, and reason-code contract.
• Compare their labels with model verdicts and upstream exclusion reasons.
• Fix the stage that owns the failure, then rerun the gold set before changing production behavior.

LLM judges do not eliminate human assessment. They make it selective: the contract supplies the categories, distributions identify samples, and human reviewers determine whether the system is still making the judgments the product needs.

Production Notes

A few practices made this stage workable in production:

1. Make every field that downstream code branches on an enum. Keep free text for the rationale and human-facing summary. Put the rationale before the enum fields so the model can reason before committing to a label.
2. Keep contract failures visible. A response that fails validation should become a stored CONTRACT_VIOLATION record, never a silently dropped candidate or a fake IRRELEVANT judgment. Monitor that rate separately from model-quality metrics.
3. Turn feedback into a debugging signal that points at the pipeline stage responsible. Ask recipients why an item was unhelpful in a small, structured vocabulary. Join that response to the original verdict and look for recurring disagreements by company, source, reason code, and pipeline stage.
4. Stabilize the task before optimizing model cost. Develop the prompt and contract on a capable model, evaluate against a fixed gold set, then test a smaller model against the same set. Otherwise, prompt defects and model capability gaps look identical.
5. Track distributions before reading individual documents. Relevance, novelty, and reason-code shifts identify where human review is most valuable. Keep OTHER under observation, a rising share among valid verdicts means the taxonomy is falling behind the domain.
6. Audit low-coverage targets in addition to the ones with negative user feedback. Thumbs-down feedback exists only for items that were served. A target receiving suspiciously little coverage may have a retrieval, entity-resolution, source, or triage problem that no user can report.

Stage 3 must be observable as well as accurate. A model verdict that cannot be stored, queried, compared, and challenged has limited value in a production decision pipeline.

Conclusion

The architecture in the previous article put the expensive model behind a bounded candidate pool. This article adds the other half: define the verdict before you tune the model.

A pointwise judge over a fixed pool is easier to control than a model asked to retrieve, rank, and explain everything at once. An enum-based verdict gives downstream code stable inputs, gives operators something to monitor, and gives user feedback a route back to the stage that owns the problem.

The LLM's job is to judge each candidate. The pipeline's job is to make that judgment inspectable: bounded by the pool, typed by the contract, and useful to the systems and people downstream.