Evaluating RAG in Production — What the Numbers Actually Revealed

I was working on a content extraction pipeline for a RAG implementation that ingests regulatory filings and legal documents to create a knowledge repository. The pipeline processes ingested files, runs them through an LLM, and produces structured summaries and extracts key insights. The output looked well-formatted and presented in a professional tone. Everything looked great to pass the unit test or integration test.

Then I decided to measure the output. 

The median F1 score across our initial test set was 0.611, which is well below our 0.80 target. The summaries read correctly. They just weren't complete. The system was confidently presenting a polished version of half the truth. 

This article is about what I learned building the evaluation approach that uncovered this, which metrics actually matter, what they miss, and the single biggest insight that changed how I think about RAG pipelines.

The Failure Mode 

Here's what our initial numbers looked like:

• Precision: 0.844 (median) — high. What the model said was almost always correct.
• Recall: 0.481 (median) — low. It was missing more than half the important content.
• F1: 0.611 — the harmonic mean told the real story.

A system with high Precision but low Recall is the most dangerous failure mode in regulated domains. It never said anything wrong; it just doesn't say enough. Stakeholders see a well-written summary and assume completeness. They don't know what's missing because they didn't read the 160-page original.

In my experience, this pattern is common in summarization pipelines that haven't been properly evaluated. If your team hasn't measured Recall independently from Precision, you're probably in the same boat.

Picking the Right Metrics

I tried every metric in the toolkit. Here's what actually proved useful:

What Worked

What Disappointed Us

Automated Metrics Pitfall

None of them catch:

• Logical errors: Correct facts in the wrong relationship
• Attribution/Citation mistakes: "Article 7 states..." when it was actually Article 12
• Temporal confusion: Mixing up current provisions with repealed ones
• Important omissions that aren't in the reference either

What I recommend now: Use F1 + ROUGE-1 + BERTScore as automated gates. Then human evaluation on 20–30 samples before any production decision. There's no shortcut past that second step.

How Document Length Impacts Scores

This was the finding that made everything else make sense.

We ran evaluation across documents ranging from 10 to 162 pages. Here's what the data showed:

That last row surprised me the most. Precision of 0.89 with Recall of 0.04. The system produced an accurate summary of pages 1–5 and completely ignored the other 157 pages.

When I ran correlation analysis:

• Recall vs. document length: r = -0.71 (p < 0.05)
• F1 vs. document length: r = -0.78 (p < 0.05)
• Precision vs. document length: r = +0.05 (not significant)

The pattern was consistent: as documents got longer, the system's coverage collapsed while its confidence stayed high. It became a confident partial summarizer, which is never wrong, just incomplete.

Why This Happens

My initial design was to process the first ~5 pages of each document. This was done to reduce token count and keep the cost in control. Usually, you will find the abstract, purpose, and high-level detail of a regulation in most of the regulatory content for any domain. So I thought this would work well, and all stakeholders agreed on this approach. This worked well during demos and the initial testing phase. We tested for a 10-page document; the first 5 pages here mean 50% coverage, which is very good. However, for a 162-page document, that's just 3%. The system literally could not see 97% of the content.

If you only test on short documents (which most teams do during development), this never shows up. The evaluation looks great, and we implement it. Then a compliance team relies on a summary that silently drops the critical clause on page 47.

How I Fixed It

What I Tried

• Hierarchical summarization: We split the full document into segments or chunks, summarize each one, then produce a final summary from the segment summaries. This guarantees every section gets seen. This improved Recall from 0.48 to ~0.65 on our test set. Not perfect, but a massive improvement for one architectural change.
• Contextual segmentation: Instead of fixed-page splits, we explored semantic segmentation. Embedding sentences and splitting where topic similarity drops. Better preservation of logical sections, but more complex to implement and tune.
• Adaptive chunk sizing: Make the chunk size proportional to document length so the entire document always gets processed. Simple, and ensures nothing is skipped.

What I'd Recommend Starting With

Hierarchical summarization or Summary or Summaries. It's the fastest fix with the highest impact. You can implement it in a day, split into 10-page segments, summarize each, store it in your index, then summarize the summaries. It won't solve everything, but it eliminates the catastrophic "invisible 97%" failure mode on long documents.

Building an Evaluation Approach That Actually Works

Here's the process I'd set up again if starting from scratch:

1. Get a Stratified Golden Dataset

Include documents at 2x, 5x, and 10x your median document length. This is where we failed initially. We tested on short documents and missed the length problem entirely.

Minimum 30 document summary pairs. Stratify by length, domain, and complexity.

Here's what a single entry looked like in ours:

{

    "doc_id": "REG_2019_1156",

    "source_file": "regulation_2019_1156.pdf",

    "pages": 12,

    "domain": "environmental_compliance",

    "reference_summary": "This regulation establishes emission standards for...",

    "key_provisions": [

        "Article 4: Emission limits for Category 3 machinery",

        "Article 7: Penalties up to $10M for non-compliance",

        "Article 12: Transitional period ending 2024-01-01"

    ],

    "difficulty": "medium",

    "notes": "Critical penalty clause is on page 9 - good test for coverage"

}

If you notice the field key_provisions, this field was the most valuable addition. It lets us measure not just vocabulary overlap but whether specific critical facts/insights were captured. If Article 7's penalty clause doesn't appear in the summary, that's a material failure regardless of what ROUGE says.

I implemented a scoring pipeline that ran nightly: a Lambda function pulled new summaries, computed F1 and BERTScore against the golden references, and pushed results to CloudWatch with alerts on threshold breaches.

2. Score at Multiple Levels

Don't just compute one number. Break it down:

• By document length: This is where you'll find the cliff
• By document type: Some content is inherently harder
• By section position: Are conclusions (typically at the end) being missed?

3. Set Honest Thresholds

4. Don't Skip Human Evaluation

I've seen cases where ROUGE was 0.45 (acceptable) and the summary was completely wrong because it used the right vocabulary to say the wrong things. Automated metrics are necessary but not sufficient without human supervision. I know it's expensive, and it might slow things down, but don’t skip human evaluation.

For faithfulness specifically, checking whether the summary only says things that are actually in the source, we used an LLM-as-judge with this prompt pattern:

“You are evaluating a summary for faithfulness to its source document.

SOURCE DOCUMENT (or relevant excerpt):
{source_text}

GENERATED SUMMARY:
{generated_summary}

For each factual claim in the summary:
1. State the claim
2. Is it SUPPORTED, PARTIALLY SUPPORTED, or NOT SUPPORTED by the source?
3. If supported, quote the relevant passage.

Then give an overall faithfulness score (0.0 - 1.0):
- 1.0 = every claim is directly supported
- 0.5 = some claims lack source backing
- 0.0 = the summary fabricates or contradicts the source

Output your assessment as structured JSON.”

This caught things ROUGE never could. Cases where the summary stated the penalty was "$5M" when the regulation actually said "$10M." The words overlapped; the meaning was wrong.

Sample 20–30 summaries. Have domain experts rate completeness and accuracy on a 1–5 scale. This is your production readiness gate.

Common Mistakes to Avoid

• Testing only on short documents: Our development set was mostly 10–20 page documents. Everything looked great. The length problem was invisible until we deliberately tested on 100+ page documents.
• Trusting Precision alone: When we found "Precision: 0.84", I thought we were in good shape. I should have immediately asked, "and what's the Recall?" High Precision with unknown Recall means nothing.
• No baseline comparison: We didn't know if our pipeline was better than simply taking the first 3 paragraphs. Turns out, for short documents, "first 3 paragraphs" was a surprisingly strong baseline.
• Evaluating once, not continuously: The first evaluation revealed the length problem. We fixed it, but we didn't set up ongoing monitoring. Six months later, new document types crept in that the pipeline struggled with, and nobody noticed until a stakeholder complained.

Key Takeaways

Measure Recall independently. If your Precision is high but you haven't checked Recall, you might have a confident liar.

• Test on long documents: Your pipeline's real weakness only appears at 5x your median document length. If you're not testing there, you don't know your system.
• The length/recall correlation is real: r = -0.71 in our case. If you're using any fixed-size processing approach, expect the same pattern.
• Fix coverage first: If Precision is already 0.80+, your problem is almost certainly Recall. Hierarchical summarization is the fastest fix.
• Automated metrics get you 80%. The last 20% requires human eyes. 
• Evaluate continuously: A continuous automated evaluation pipeline ensures the system remains reliable over time. 

The goal isn't a perfect score, it's knowing exactly where your pipeline fails so you can make informed decisions about what to deploy, and what to hold back. Build the evaluation pipeline before you need it, because by the time a compliance team finds the gap, you've already shipped a confident liar.