Candidate Generation Decides Your Pipeline's Cost, Not the LLM

When the Most Capable Model Is the Wrong Starting Point

The fastest way to exceed a document pipeline budget is to let an LLM inspect every document before you have performed lightweight filtering. This sounds obvious, but the bottleneck is invisible at the prototype stage. A single model call is cheap, and it works well on the 20 documents in your test set. Then you hit production traffic.

The failure mode is usually pretty similar across teams: tens of thousands of LLM calls per day, tens of millions of tokens, and a monthly bill that drifts past the assigned budget. No candidate generation. No triage. Raw corpus straight to the model. The cost compounds because the corpus does not shrink without an upstream triage. A more capable model just gives you a more expensive way to process noise.

The Bottleneck Is Candidate Generation

Summarization is the easy half. Given a good document and a clear target, almost any capable model produces a passable summary. The hard part is deciding which documents are good and which ones match a given target at all. In a large-scale pipeline, most documents are irrelevant to any particular target entity. A company monitoring its own products, competitors, regulatory activity, industry and market signals might care about a small fraction of incoming documents. The system has to find those without missing too many, and without spending inference budget on the overwhelming majority it should never have touched.

Bad candidate pools produce confident summaries of the wrong material. Once the candidate pool is good, the LLM step becomes something you can afford to run. If you solve only summarization, you get a pipeline that is both unaffordable to run and unreliable in what it produces at scale.

The Three-Stage Pipeline

The pipeline ingests a high-volume stream of documents such as web articles, news feeds, financial filings, or any large text corpus and delivers a curated candidate set for each target. Targets might be companies, products, people, regulatory topics, threat actors, or research areas — potentially thousands of them. Each target has a profile defining what it cares about. The pipeline's output is a per-target digest: a scored and summarized shortlist of deduplicated documents.

The pipeline runs in three stages. Each stage is more expensive per document than the one before it, and each shrinks the volume the next stage sees. Figure 1 shows the end-to-end flow.

Stage 1: Cost-Efficient Triage

The triage stage ingests every incoming document through lightweight classifiers and filters. This filtering catches ads, paywalled stubs, spam, malformed pages, and auto-generated fragments. Purpose-built classifiers for topic, region, language, and content type label the document. These are small supervised classifiers trained for specific domain detection. Named entity recognition (NER), knowledge graph entities, industry verticals, and salient term extraction transform the document into a structured feature vector. Near duplicate detection clusters documents with near-identical text and picks a canonical document to represent the event. 

One of the highest-value filters here is less obvious. A large fraction of incoming news is stock-ticker recap: an article whose entire body is that a company's share price moved some percent, with the company name and ticker symbol repeated throughout. On pure entity overlap, it looks maximally relevant to that company. For almost any downstream consumer, it carries nothing actionable. A naive pipeline scores it high and spends a model call on it. We trained a small classifier specifically to catch this pattern and tuned it on human-annotated examples, because the failure mode is subtle: the document is not spam, it is well-formed, and it is genuinely about the target — it is just useless. Generic junk filters miss it for exactly that reason.

This stage uses no LLMs. Classical ML, static rules, regex, bloom filters, and blocking indexes are enough to process millions of documents a day. The output is a canonical document with a feature vector attached. Not a summary. Not an embedding. Teams sometimes try to add semantic richness here and end up with an ingestion stage that is computationally expensive and difficult to debug. Save that work for downstream stages. The stream coming out is far smaller than the stream going in — most of the firehose never survives triage — but the exact survival rate depends entirely on the corpus.

Stage 2: Target-Aware Retrieval

Stage 2 processes the features emitted by Stage 1 and matches them against target profiles to produce a bounded candidate set per target. The retrieval problem is not a full cross-join of documents against targets. That does not scale. Instead, you build an inverted index over the document feature vectors and use blocking strategies such as entity mentions and domain signals to constrain which target profiles each document is evaluated against. This is closer to classic information retrieval index construction than to brute-force semantic search.

A common instinct is to reach for dense vector similarity as the primary retrieval layer. Push back on that. Embeddings are useful, but they are often not effective when entity aliases, regulated terminology, and taxonomy labels are what actually define a match. A pharmaceutical company named in a document by its subsidiary's trade name will not reliably surface in a cosine-similarity search against a profile built on the parent brand. Inverted indexes on extracted entities and taxonomy codes handle that case directly. Dense retrieval can still participate as a secondary ranking signal, but it is not where to start.

Entity matching has the opposite failure too. Some of the most relevant documents name no target entity at all. An industry or regulatory development can matter to a whole set of targets without mentioning any of them by name — a rule change affecting a sector, a shift that hits every company in a category. Strict entity overlap drops these on the floor. We handle them with a separate classifier that matches on industry and taxonomy signals rather than entity mentions, then routes the document to every target whose profile sits in that category. This is why the scorer combines taxonomy agreement alongside entity overlap rather than treating entity match as the only path in: the entity-free relevant document is real, and a pipeline that only does entity matching never sees it.

For each document-target pair that survives blocking — whether it got there by entity match or taxonomy signal — a lightweight scorer combines entity overlap, keyword overlap, taxonomy agreement, and source reputation into a match score. This step is fast, deterministic, and easy to interpret. Each target accumulates a bounded pool, capped at roughly 50 to 100 documents. This pool is the only set of documents the LLM ever processes, and the cap is what makes Stage 3 cost predictable rather than a function of corpus size.

Here is the scoring logic in compact form:

def stage2_score(doc_features, target_profile):
    """Calculates a deterministic match score, bypassing heavy model inference."""
    # Blocking: Fast rejection using set intersections
    shared_entities = doc_features.entities & target_profile.entities
    shared_topics = doc_features.topics & target_profile.topics

    if not (shared_entities or shared_topics):
        return None
    # Weighted match signal — pure computation, zero model calls
    score = sum([
        WEIGHTS.ENTITY * jaccard(doc_features.entities, target_profile.entities),
        WEIGHTS.KEYWORD * keyword_overlap(doc_features, target_profile),
        WEIGHTS.TAXONOMY * taxonomy_agreement(doc_features, target_profile),
        WEIGHTS.REPUTATION * source_reputation(doc_features.source)
    ])
    return score if score >= target_profile.threshold else None


def build_candidate_pools(documents, target_index):
    """Maps documents to target profiles, returning bounded candidate sets."""
    pools = defaultdict(list)

    for doc in documents:
        # retrieve_candidate_targets acts as our inverted index lookup
        for target in retrieve_candidate_targets(doc, target_index):
            score = stage2_score(doc.features, target.profile)

            if score is not None:
                pools[target.id].append((doc, score))

    # Cap each pool at K so Stage 3 cost is bounded, not corpus-dependent
    return {target_id: top_k(candidates, k=100) for target_id, candidates in pools.items()}-

Blocking eliminates the cross-join. Scoring is a deterministic linear combination so that engineers can reason about why a document scored where it did. The top-K trim caps downstream cost regardless of how many candidates passed the threshold.

Stage 3: Bounded LLM Reasoning

By the time the LLM is invoked, it is operating over the bounded pool from Stage 2 — at most 100 documents per target, not the entire corpus. Of that pool, the model typically marks 10 to 50 documents as relevant for the target's digest. That reduction is a relevance verdict, not another trim — the trimming already happened in Stage 2, which is the whole point: the expensive model judges; it does not select. That is the difference between something that looks good in a prototype and something that survives production. Tasks at this stage are the final relevance judgment, novelty detection, concise summarization, theme extraction, and reason code generation. These tasks genuinely need a capable model because they require judgment and contextual synthesis that rule-based scorers cannot provide. If you think of this as retrieval-augmented generation, the retrieval side is doing most of the operational work. The RAG survey literature makes that retrieval/generation split explicit.

By this stage, the LLM is judging a curated shortlist rather than searching the corpus, and that changes two things. The cost argument is the obvious one: token usage starts only after the pool is bounded. The quality argument matters as much. A model handed 10 vetted candidates isn't competing with 9,990 distractors for attention — usually improves the quality of the relevance judgment. 

An End-to-End Concrete Example

To see how these three stages interact in practice, let's trace a single document from raw ingestion to final LLM reasoning. Suppose a standard press release arrives at 7:14 AM. The headline mentions a company called "Pomfrey Health Solutions" announcing a new contract with a hospital network called "Mount Avery Medical Center".

Stage 1 clears the junk filter, classifies the document as healthcare/industry news, and runs NER to extract "Pomfrey Health Solutions" and "Mount Avery Medical Center". Here is where alias handling matters. The entity database knows that "Pomfrey Health Solutions" is a wholly owned subsidiary of "Pomfrey Corp," a monitored company. Without that mapping, the document exits Stage 1 without matching anything and is never seen again. With it, the NER output gets enriched with the parent entity ID before the feature vector is indexed.

Stage 2 looks up the Pomfrey Corp entity ID and finds two active target profiles: an investment research team tracking the company and a competitor tracker watching the hospital software market. Heuristic scoring clears both thresholds. The document enters both candidate pools. No embedding lookup. No LLM call. Under single-digit milliseconds on a warm in-memory index.

Stage 3 (runs at 7:20 AM). The LLM receives the bounded pool for Pomfrey Corp — 72 candidates, this press release among them. The model judges the release relevant to Pomfrey Corp and confirms it's genuinely new — nothing in the recent window already covered this contract. It attaches a one-sentence summary and the reason code new_contract_win. Token usage starts only after the candidate pool is bounded. A naive pipeline spends tokens before it knows whether the document matters.

Where This Pattern Reappears

Nothing in this architecture is specific to news articles. The three-stage shape, namely cost-efficient triage, target-aware retrieval and bounded LLM reasoning, appears anywhere the document/corpus volume is high and the relevance question is specific.

Legal discovery is the clearest parallel. Documents are case filings, deposition transcripts, contracts, internal emails, and chat logs. Targets are legal matters or custodians. Stage 1 handles format normalization, OCR, deduplication, and junk removal. Stage 2 matches documents to matters using legal entity names, date ranges, case codes, and custodian aliases — the same alias problem the "Pomfrey" example showed, but with legal and compliance risk at stake. Stage 3 produces relevance and privilege flags for each document, plus short excerpts and timeline metadata for the ones that survive. Candidate pool discipline matters more here than in news. A missed document is not a missed news story. It is a discovery failure for the case.

Enterprise security has the same shape with a different texture. Alerts and threat intelligence reports are the corpus. Monitored assets and known threat actors are the targets. At Stage 3, you get a ranked list of alerts with a short triage note instead of a news digest. Corpus, prompts, and target profiles all differ, but the underlying architecture is virtually the same.

Trade-offs to Make Explicit

Any production system is a set of explicit choices. The main ones in this architecture: