RAG Evaluation Guide: Measuring Retrieval and Generation as Separate Problems

Introduction

A retrieval augmented generation (RAG) system fails, and you see the wrong answer. The system retrieved documents from a knowledge base, a large language model generated a response from that context, and somewhere in that pipeline, something broke. Was the retrieval bad? Did the model hallucinate despite getting the right documents? If you're evaluating the final response as a single score, you can't tell.

This is the default state of RAG evaluation at most organizations. A 2025 study on facet-level hallucination tracing by Elchafei et al. found that Evidence Override — the model ignoring correct retrieved context — accounted for 28.4% of hallucinations in medical QA and 42.3% in HotpotQA. Evidence Failure, where retrieval actually returned the wrong documents, caused problems 4 to 7 times less often. The generation side was the dominant failure mode, not retrieval.

McKinsey's 2025 State of AI survey found that 88% of organizations now use AI in at least one function, but inaccuracy remains the most commonly reported negative consequence. Only 39% of respondents attribute any EBIT impact to AI. The gap between deployment and value is, in many cases, an evaluation gap. Teams that cannot pinpoint where quality breaks down cannot fix it systematically.

This guide walks through building a RAG evaluation pipeline that separates retrieval from generation. It covers what to measure at each stage, when to use LLM judges versus human reviewers, and why metric selection has to match your specific domain.

‍

Why Do Most RAG Evaluations Fail to Diagnose the Actual Problem?

The instinct to evaluate RAG end-to-end is understandable. You ask a question, the system answers, you check whether the answer is correct.

The problem is that "incorrect" has at least two distinct causes, and they require different fixes. If retrieval returned irrelevant documents, you need to change your chunking strategy or embedding model. If retrieval was fine but the model hallucinated anyway, the fix is in your prompt or grounding constraints. An end-to-end score registers the failure without localizing it.

The Facet-Level Tracing study from Johannes Kepler University decomposed hallucination causes across two datasets into Evidence Failure (retrieval problems) and Evidence Override (generation ignoring correct context). Evidence Override dominated in both. The retrieval worked; the model just didn't use it.

The RGB benchmark, accepted at AAAI 2024, tested LLMs on four fundamental RAG abilities: noise robustness, negative rejection, information integration, and counterfactual robustness. Models exhibited some noise robustness but struggled with negative rejection — knowing when to say "I don't have enough information" — and with integrating information across multiple retrieved passages. These are generation-side failures that retrieval metrics will never detect.

Mindful-RAG from Arizona State University identified eight critical failure points in knowledge-graph-based RAG systems, categorized into reasoning failures and topology challenges. Several of these failures occur even when the right facts are present in the knowledge graph. The model fails to reason over them correctly.

The pattern across all three studies: retrieval quality and generation quality fail independently, through different mechanisms, at different rates. Evaluating them together hides the signal you need to improve either one.

What Should You Measure at the Retrieval Stage?

Retrieval evaluation borrows from decades of information retrieval research. When a user query hits the retrieval component — pulling from external knowledge sources like a vector database or document index — the question is straightforward: did the system return the right documents, and did it rank them well?

The Auepora survey by Yu et al., published through Springer, maps retrieval metrics to specific RAG components. The key metrics split into two categories.

Relevance and coverage metrics tell you whether the retrieved set contains what it should. Context Precision measures the proportion of relevant chunks in the retrieved set. Context Recall measures context coverage — how much of the information the model needs actually made it into the retrieved context. These two metrics, formalized in the RAGAS framework, are reference-free — they don't require ground-truth annotations to compute.

Ranking metrics tell you whether relevant documents appear where they should in the ordered list. NDCG@k (Normalized Discounted Cumulative Gain) weights relevant documents logarithmically by position: a relevant document at position 1 counts more than one at position 5. MRR (Mean Reciprocal Rank) measures the reciprocal rank of the first relevant document, answering: how far does the model have to look before finding something useful?

For most RAG applications, a practical starting point is Context Recall above 0.8 (you're retrieving most of the relevant information) and Context Precision above 0.6 (more than half of what you retrieve is actually useful). These thresholds shift by domain and by chunk size — smaller chunks improve precision but risk splitting relevant information across boundaries, while larger chunks boost recall at the cost of noise. In medical or legal RAG, you likely want higher recall to avoid missing critical information, even at the cost of lower precision.

What Should You Measure at the Generation Stage?

Generation evaluation asks different questions: given the retrieved context, did the model produce a faithful, complete, relevant response?

Faithfulness is the most critical generation metric — a direct measure of factual accuracy relative to the retrieved context. It checks whether every claim in the generated response is supported by the retrieved documents. The RAGAS framework computes this by extracting claims from the response and checking each one against the provided context.

RAGChecker, presented at NeurIPS 2024, decomposes responses into atomic verifiable claims and scores each one independently. Claim-level scoring correlates significantly better with human judgments than holistic methods. When a response contains five claims and three are faithful but two are fabricated, claim-level decomposition catches the two bad ones. A holistic score might average them into a misleading "mostly correct."

Hallucination detection requires its own evaluation layer. FaithJudge from Vectara, accepted at EMNLP 2025, benchmarks LLM faithfulness across summarization, QA, and data-to-text tasks. Their core finding: LLMs frequently introduce unsupported information even when provided with relevant context. An LLM-as-judge approach using curated human-annotated hallucination examples outperformed zero-shot prompting for catching these failures.

Answer relevance measures whether the response generated actually addresses the user query. A response can be entirely faithful to the retrieved documents and still miss the point of the question. When a reference answer exists, answer correctness can be computed directly through semantic similarity — but most production RAG systems lack reference answers at scale, making reference-free relevance metrics the practical default.

These generation metrics operate independently from retrieval metrics. A perfect retrieval score with poor faithfulness means your model is the bottleneck. Poor retrieval with high faithfulness on the few documents that were retrieved means your retrieval pipeline needs work. Measuring both stages separately is how you localize the failure.

How Reliable Are LLM Judges for RAG Evaluation?

Using LLMs to evaluate LLM outputs sounds circular. The data says otherwise, with caveats.

The TREC 2024 RAG Track, a NIST-sponsored evaluation accepted at SIGIR 2025, compared GPT-4o judgments against human judges on support assessment. Agreement rates were 56% when both judged from scratch and 72% when humans post-edited LLM judgments. An independent third human judge correlated better with GPT-4o's scores than with the other human judge's scores. Human-human agreement isn't the gold standard people assume it is.

ARES, published at NAACL 2024 by Stanford researchers, takes a different approach: it finetunes lightweight language models as domain-specific judges. ARES requires only a few hundred human annotations to calibrate its judges via prediction-powered inference. This keeps human evaluation in the loop as a calibration mechanism rather than trying to replace it entirely.

The limits are real, though. Automated judges perform well on straightforward factual queries but degrade on multi-hop reasoning, numerical analysis, and domain-specific logic. The Cleanlab benchmarking study tested five hallucination detection methods across four RAG datasets. No single method was reliable across all scenarios. RAGAS Faithfulness proved moderately effective for simple search-like queries but produced an 83.5% null-result rate on FinanceBench, which requires numerical reasoning over financial documents.

LLM judges are viable for high-volume, first-pass evaluation. Human review remains necessary for calibration, for complex reasoning tasks, and for establishing the ground-truth baselines that automated judges are measured against.

Why Does No Single RAG Evaluation Metric Work Across Domains?

RAGAS Faithfulness works when "faithfulness" means "the response restates facts from the context." It fails when faithfulness requires computation — adding numbers from a table, calculating year-over-year changes, or synthesizing across multiple documents with conflicting information.

Different RAG use cases stress different failure modes. A customer support RAG needs high answer relevance and low hallucination. A legal research RAG needs exhaustive retrieval recall and precise citation. A medical QA RAG needs negative rejection — the ability to say "the available evidence doesn't support an answer" rather than generating a plausible-sounding response from insufficient context.

This means your evaluation criteria are a design decision, not a default. Your metric stack should reflect the specific ways your RAG system can fail in your domain, and that often means defining custom metrics beyond standard frameworks.

Factual question answering (support, documentation): prioritize Context Recall, Faithfulness, Answer Relevance.

Numerical or analytical RAG (finance, data analysis): add claim-level decomposition (RAGChecker-style), since holistic faithfulness scores miss computational errors.

High-stakes domains (medical, legal, compliance): prioritize Context Recall (miss nothing), Negative Rejection (refuse when unsure), and use human evaluation on a meaningful sample of edge cases.

Multi-document synthesis (research, competitive analysis): measure Information Integration — whether the model combines facts from multiple retrieved passages into a coherent response rather than echoing a single source.

How Do You Build a Two-Stage RAG Evaluation Pipeline?

Separating retrieval from generation evaluation is a design principle. Implementing it requires a concrete workflow.

Stage 1: Retrieval evaluation. For each query in your evaluation dataset, run retrieval independently and capture the retrieved document set before it reaches the generator. Score this set against your ground-truth relevant documents using Context Precision, Context Recall, and NDCG@k. If retrieval scores fall below your thresholds, no amount of generation tuning will fix the output. Fix retrieval first.

Stage 2: Generation evaluation. Take the queries where retrieval passed your thresholds and evaluate the generated responses. Apply Faithfulness scoring at the claim level. Measure Answer Relevance against the original query. For domain-specific applications, add the failure-mode tests relevant to your use case — negative rejection, information integration, numerical accuracy.

The human calibration layer. Both stages need periodic human evaluation. For retrieval, a domain expert reviews a sample of retrieved sets to verify that automated relevance scores match actual relevance. For generation, human reviewers evaluate response quality on dimensions that automated metrics miss: nuance, completeness, tone, regulatory compliance. ARES's model of using a few hundred human annotations for calibration is practical. Human evaluation validates automated metrics; it doesn't replace them.

What this looks like in practice. Take a RAG system that drafts messages for customer-facing advisors. Retrieval accuracy and response quality both matter, but for different reasons.

The retrieval evaluation rubric has three core jobs:

• RESPONSE_CORRECTNESS is a binary classification — pass or fail — with a conditional comment field that fires when the response fails.
• SOURCE_MATCH checks whether the response relies on the correct source document, with a required justification when it doesn't.
• SPAN_FEEDBACK lets evaluators highlight specific passages in the response that need correction, turning vague "this is wrong" into pinpointed feedback.

The generation evaluation rubric adds dimensions that retrieval metrics can't touch.

• TONE scores whether the message reads appropriately for an advisor-to-customer interaction.
• USABILITY scores whether the advisor can send the response as-is or needs to rewrite it. These are inherently subjective, which is why consensus scoring — multiple evaluators independently rating the same response, with disagreements surfaced for review — matters here more than on the retrieval side.

Each evaluation stage runs as a separate project with its own rubric, its own evaluator pool, and its own quality thresholds. A multi-step workflow routes assets from initial evaluation through review, with configurable sampling rates controlling how much of the evaluated set gets a second pass. Step separation ensures no evaluator reviews their own work.

Retrieval accuracy and response quality are different evaluation problems, scored by different rubrics, often by different people. Separate evaluation workflows turn vague quality complaints into specific, fixable diagnoses.

Conclusion

The central mistake in RAG evaluation is treating a two-component system as a single unit. Retrieval and generation fail independently, through different mechanisms, at different rates. Measuring the final output tells you something is wrong. Measuring each stage tells you what to fix.

The tools exist. RAGAS, ARES, RAGChecker, and FaithJudge provide automated metrics for both stages. LLM judges are reliable enough for first-pass evaluation when calibrated against human baselines. The harder part is organizational: building evaluation into the pipeline as a permanent capability, selecting metrics that match your domain's failure modes, and maintaining the human calibration layer that keeps automated scores honest.

The 39% of enterprises that McKinsey found actually attributing EBIT impact to AI are not running different models. They have the infrastructure to know where quality breaks down and to fix it before it reaches users. Measuring RAG performance starts with evaluating retrieval and generation as the separate problems they are.

Resources

Evaluation Frameworks

• RAGAS: Automated Evaluation of Retrieval Augmented Generation – Reference-free framework separating retrieval and generation metrics; processes 5M+ evaluations monthly
  • https://arxiv.org/abs/2309.15217
• ARES: Automated Evaluation Framework for RAG Systems – Finetuned lightweight LM judges requiring only a few hundred human annotations; published at NAACL 2024
  • https://arxiv.org/abs/2311.09476
• RAGChecker: Fine-Grained Diagnostic Framework for RAG – Claim-level decomposition with diagnostic metrics for retrieval and generation; NeurIPS 2024
  • https://arxiv.org/abs/2408.08067

Benchmarks and Empirical Studies

• RGB: Benchmarking LLMs in Retrieval-Augmented Generation – Tests noise robustness, negative rejection, information integration, and counterfactual robustness; AAAI 2024
  • https://arxiv.org/abs/2309.01431
• TREC 2024 RAG Track: Human vs. LLM Judges – Compares GPT-4o and human judge agreement on support assessment; accepted at SIGIR 2025
  • https://arxiv.org/abs/2504.15205
• FaithJudge: Benchmarking LLM Faithfulness in RAG – Curated hallucination examples for LLM-as-judge evaluation across task types; EMNLP 2025 Industry Track
  • https://arxiv.org/abs/2505.04847
• Benchmarking Hallucination Detection Methods in RAG – Comparison of five detection methods across four datasets; shows domain-dependent reliability
  • https://cleanlab.ai/blog/rag-tlm-hallucination-benchmarking/

Surveys and Failure Analysis

• Evaluation of Retrieval-Augmented Generation: A Survey – Unified evaluation process mapping metrics to RAG components; published via Springer
  • https://arxiv.org/abs/2405.07437
• Facet-Level Tracing of Evidence Uncertainty and Hallucination in RAG – Demonstrates Evidence Override causes 4–7x more hallucinations than Evidence Failure
  • https://arxiv.org/abs/2604.09174
• Mindful-RAG: Points of Failure in Retrieval Augmented Generation – Identifies 8 critical failure points in knowledge-graph-based RAG
  • https://arxiv.org/abs/2407.12216

Reference Resources

• EvalScope RAG Evaluation Documentation – Overview of standard IR metrics (NDCG@k, MRR) applied to RAG evaluation
  • https://evalscope.readthedocs.io/en/latest/blog/RAG/RAG_Evaluation.html

Enterprise Context

• McKinsey: The State of AI in 2025 – Survey of 1,993 respondents; 88% of organizations use AI but inaccuracy is the top negative consequence
  • https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai

‍

FAQ

What is RAG evaluation?

‍RAG evaluation is the process of measuring how well a retrieval augmented generation system answers questions using external knowledge sources. Rather than treating the system as a black box, effective RAG evaluation separates the pipeline into two stages — retrieval (did the system pull the right documents from the knowledge base?) and generation (did the large language model produce a faithful, relevant response from that context?). Measuring each stage independently is what lets you diagnose whether a failure is a retrieval problem or a generation problem.

Which metrics matter most for RAG retrieval evaluation?

‍The core retrieval metrics are Context Precision (proportion of relevant chunks in the retrieved set), Context Recall (how much of the needed information made it into the context), NDCG@k (ranking quality weighted by position), and MRR (how far the model looks before finding the first relevant document). A practical starting point is Context Recall above 0.8 and Context Precision above 0.6, though thresholds shift by domain and chunk size.

How do you measure RAG generation quality?

‍Generation evaluation centers on Faithfulness — whether every claim in the response is supported by the retrieved context. Claim-level decomposition, as used in the RAGChecker framework, catches partial hallucinations that holistic scores miss. Beyond faithfulness, Answer Relevance measures whether the response addresses the user query, and domain-specific metrics like negative rejection or tone scoring capture failure modes that standard frameworks don't cover.

Can LLM judges replace human evaluation for RAG?

‍LLM judges are reliable for high-volume, first-pass evaluation — the TREC 2024 RAG Track found GPT-4o matched human judges at 56–72% agreement, and in some cases correlated better with a third human judge than the human judges correlated with each other. But automated judges degrade on multi-hop reasoning, numerical analysis, and domain-specific logic. Human review remains necessary for calibration, complex reasoning tasks, and establishing ground-truth baselines. The practical model is human evaluation as a calibration layer, not a replacement for automation or vice versa.

Why doesn't a single RAG evaluation metric work across all use cases?

‍Different RAG applications stress different failure modes. RAGAS Faithfulness works for straightforward factual QA but produced an 83.5% null-result rate on FinanceBench, which requires numerical reasoning. A customer support RAG needs high answer relevance; a legal research RAG needs exhaustive retrieval recall; a medical QA RAG needs negative rejection. Your evaluation criteria are a design decision — the metric stack should reflect the specific ways your system can fail in your domain.

How does Kili support RAG evaluation workflows?

‍Kili's platform maps directly to the two-stage evaluation architecture described in this guide. You can configure separate project steps for retrieval evaluation and generation evaluation, each with its own rubric — classification jobs for binary pass/fail scoring, optional fields for conditional feedback, and named-entity recognition for span-level annotations. Consensus scoring surfaces evaluator disagreements on subjective dimensions like tone or usability, and configurable sampling rates control how much of the evaluated set gets a second review pass. Step separation ensures no evaluator reviews their own work, which matters for audit trails in regulated industries.