Measuring What a RAG System Retrieves, Not Just What It Answers

One scoreboard, two instruments

Retrieval-augmented generation was introduced as a combination of two distinct components — a parametric generator and a non-parametric retriever consulting an external index — precisely because the two were expected to fail differently and to be improved separately ^[1]. Measurement practice has not kept pace with that architectural distinction. A retrieval-augmented generation system is routinely reported as a single number: some accuracy, some pass rate, some percentage of questions answered correctly. That number is comfortable, and it is also a merger of two measurements that were taken with different instruments, on different objects, at different points in the pipeline. One instrument scores a ranking function against a labelled set of relevant documents and never looks at a single word the model writes. The other scores the model’s written output against whatever passages the retriever happened to hand it, and never checks whether those passages were the right ones to hand over.

Collapse the two into one end-to-end score and a specific, repeatable confusion follows. A system can score well on answer quality while its retrieval barely functions, because the underlying language model already knew the answer from its own training and would have written the same sentence with an empty context window. A system can score badly on answer quality while its retrieval works perfectly, because the generator hedges, misreads, or contradicts a passage that was in fact exactly the right one. Both systems can report the same headline accuracy. Only one of them has a retrieval problem, and the headline number cannot say which.

This article works through the two measurement objects in turn — what a retrieval metric actually scores and what a generation metric actually scores — then the automated LLM-as-judge frameworks now used to approximate both at scale, the structural reason the two can move independently, the human evaluation protocols built to check the automated approximations, and the contamination and leakage risks that quietly stop a benchmark from measuring what it claims to measure.

What a retrieval metric actually scores

A retrieval metric is computed from three things only: a query, a ranked list of documents or passages returned for it, and a set of relevance judgements — a labelled mapping from each query to the documents that count as relevant, usually built by human assessors ahead of time. None of the three requires a generator. This is the methodological point worth holding onto before any of the specific metrics: recall at $k$, precision at $k$, mean reciprocal rank and normalised discounted cumulative gain are all properties of a ranking function evaluated against ground truth, and they can be computed the moment the retriever returns its list, with no language model ever invoked.

Recall at $k$ is the simplest of the four: the fraction of relevant documents that appear anywhere in the top $k$ results. It answers one question only — did the evidence make it into the candidate set at all — and says nothing about where within that set it landed. Mean reciprocal rank answers the positional question for the single best hit:

averaged over a query set $Q$, where $\mathrm{rank}_q$ is the position of the first relevant result for query $q$. Mean reciprocal rank was formalised as the primary scoring metric for the TREC-8 Question Answering track, the first large-scale evaluation of domain-independent question answering systems, where it assigned a value of $1/r$ to a ranked list of candidate answers based on the rank $r$ of the one judged correct ^[3]. It rewards getting the single most useful passage near the top and is blind to everything else in the list, which is exactly the property a downstream generator that reads only the first few passages needs measured.

Normalised discounted cumulative gain generalises this to graded relevance and to the whole list rather than only the first hit:

where $\mathrm{rel}_i$ is the graded relevance of the result at position $i$ and $\mathrm{IDCG@}k$ is the DCG of the ideal ordering, so the ratio is bounded near one regardless of how many relevant documents exist for a given query. The logarithmic discount encodes a specific judgement about attention: a relevant document at position one is worth far more than the same document at position ten, and nDCG is the standard way the information-retrieval literature makes that judgement quantitative. Heterogeneous retrieval benchmarks built to compare retrievers across many domains at once — BEIR evaluated ten lexical, sparse, dense, late-interaction and re-ranking systems across eighteen public datasets and found BM25 a robust baseline that dense retrievers frequently underperformed out of domain — rely on exactly these rank-based metrics to make that comparison possible without ever generating an answer ^[2].

The practical consequence is that a retrieval-only evaluation is cheap to build once and reusable forever. A set of queries paired with the identifiers of the passages that answer them, scored by recall at $k$ and nDCG, survives every later change to the generator, the prompt template, or the model vendor, because none of those changes touch the object being scored.

What an answer-quality metric actually scores

A generation metric is a different kind of object entirely. It does not consult the relevance judgements at all. Instead it takes the model’s written response, the specific context that was actually handed to the generator, and sometimes the original question, and scores some relationship among those three — typically whether the response is supported by the context, whether it addresses the question, and whether it stays inside the boundaries of what the context actually says.

Ragas is a widely used reference-free evaluation framework built explicitly around this separation: it scores retrieval effectiveness, the faithfulness with which a model uses retrieved passages, and generation quality, without requiring a human-written ground-truth answer for every query ^[4]. Its faithfulness metric is defined mechanically rather than as a vague notion of “sounding grounded”: a response is decomposed into individual factual claims, each claim is checked against the retrieved context to see whether it can be inferred from it, and the score is the resulting ratio, as the framework’s own documentation states ^[5]:

Notice what this formula does and does not depend on. It does not depend on whether the retrieved context was the right context — a response can be perfectly faithful to a passage that answers a different question than the one asked, and the faithfulness score will not fall. It depends only on the relationship between the written output and whatever was placed in front of the model. Answer relevance and groundedness metrics, which the same documentation lists alongside faithfulness, complete the picture from the other side: relevance checks whether the response actually addresses the question rather than merely being consistent with the context, and groundedness-style checks look for unsupported additions the model introduced beyond what any retrieved passage stated ^[5].

Put the two families of metric side by side and the division of labour is exact. A retrieval metric asks whether the right evidence was in the candidate set. A generation metric asks whether the written answer is a faithful, relevant argument from whatever evidence it was actually given. Neither question can stand in for the other, because each is computed from a disjoint set of inputs — one from a ranked list and a label set, the other from a response and a context — and a system can score at either extreme on one axis while sitting anywhere at all on the other.

Automated judges: what RAGAS and ARES actually buy

Faithfulness and answer relevance are cheap to define and expensive to compute by hand at scale, because deciding whether a claim is entailed by a passage is itself a judgement call that traditionally required a trained human reader. The dominant solution in current practice is to have another large language model make that call — an LLM-as-judge — and the two evaluation frameworks most cited for RAG specifically, Ragas and ARES, both build on this move while trying to control its cost in different ways.

Ragas prompts a judge model to extract claims and verify each one against the retrieved context directly, and its documentation also describes an alternative that swaps the judge model for a small purpose-trained classifier for the verification step, offered as a lower-cost substitute for the LLM call ^[5] — a framework design choice that should be read as exactly that, a stated engineering trade-off from the tool’s own documentation, not an independently verified claim about which approach is more accurate.

ARES takes calibration further. It fine-tunes lightweight language-model judges on synthetic training data to score context relevance, answer faithfulness and answer relevance, then corrects the resulting predictions using prediction-powered inference against a small set of human-annotated data points, rather than trusting the fine-tuned judge outright. Its authors report that this combination remained accurate across eight knowledge-intensive tasks spanning KILT, SuperGLUE and AIS, using only a few hundred human annotations during evaluation, and that the judges stayed reliable even when the query and document distributions shifted from what they were calibrated on ^[6]. The human-annotated subset is not a formality; it is the mechanism by which a cheap automated judge earns the right to be trusted at scale rather than merely assumed to be.

How much should that trust extend? The most cited evidence on LLM-as-judge reliability comes from outside RAG specifically. Zheng and colleagues validated GPT-4 as a judge of open-ended chat responses against both controlled and crowdsourced human preferences on MT-Bench and Chatbot Arena, and found agreement exceeding 80 percent — comparable to the level of agreement between two human annotators — while also identifying and proposing mitigations for position bias, verbosity bias and self-enhancement bias in judge models ^[7]. That is a strong result, and it is worth being precise about what it does and does not establish for a RAG faithfulness judge specifically. It was measured on open-ended chat preference judgements, not on claim-by-claim entailment against a retrieved passage; carrying an 80-percent figure from one judging task to a structurally different one is an assumption a RAG evaluation pipeline makes, not a transfer the cited study itself demonstrates. The named biases are the more directly relevant finding: a verbosity-biased judge will tend to reward a longer, more hedged answer regardless of whether it is more faithful, which is precisely the failure mode a faithfulness metric exists to catch.

Why a high answer score can hide a bad retriever, and the reverse

The measurement structure explains the divergence directly. Let $R$ be the event that the retriever’s top-$k$ results contain sufficient evidence to answer the query, and let “correct” mean the generated answer is judged faithful and relevant. Total correctness decomposes as

An end-to-end accuracy score measures the left-hand side only. The two terms on the right are exactly what generation metrics and retrieval metrics separately illuminate, and the second term is the one an aggregate score cannot see at all. $P(\text{correct} \mid \lnot R)$ is the probability the model answers correctly despite the retriever having failed — because the underlying language model already carried the answer parametrically, because the question was easy enough to guess, or, in the worst case, because the benchmark’s own answer had leaked into the model’s training data. Whenever that term is large, an end-to-end score can look excellent while the retrieval component the system was supposedly built around contributed nothing to the result.

The reverse failure sits inside the first term. $P(\text{correct} \mid R)$ can be low even when $R$ holds — the right evidence was retrieved and still the generator wrote an unfaithful, evasive, or contradicted answer — and a retrieval-only evaluation, which never inspects generation at all, cannot detect this. Chen and colleagues benchmarked RAG systems along four separated axes — noise robustness, negative rejection, information integration, and counterfactual robustness — and found that while models tolerate some retrieved noise, they struggle badly at declining to answer when the retrieved context does not actually contain the answer, at integrating information across multiple retrieved documents, and at resisting counterfactual content placed in the context ^[8]. Negative rejection failure is the second term in the equation made concrete from the opposite direction: a system that answers confidently when it should abstain is a generation failure that a retrieval metric, scored only against relevance judgements, never sees.

CRAG, a factual question-answering benchmark built specifically to represent the dynamic, long-tail conditions real retrieval systems face, reported that advanced language models alone answered no more than 34 percent of its questions correctly, that adding retrieval in a straightforward way raised this only to 44 percent, and that the strongest industry RAG systems evaluated answered 63 percent of questions without producing any hallucination ^[9]. Those are the benchmark authors’ own reported figures rather than an independent replication, and they should be read as one dynamic, adversarially designed benchmark’s result rather than a universal ceiling on RAG accuracy — but the gap between “retrieval added” and “retrieval that actually closes the question” is exactly the gap this article’s decomposition predicts, and it is a gap no single end-to-end number distinguishes from a gap in the underlying language model’s reasoning.

Human evaluation protocols: what a person still has to judge

Automated judges are validated against human judgement, not a substitute that eliminates the need for it, and the largest standing effort to formalise that human layer for RAG specifically is the TREC RAG track. Its 2024 guidelines split evaluation into a retrieval task, an augmented-generation task that receives a fixed set of retrieved segments and must generate a grounded, cited answer from them, and a full retrieval-augmented-generation task combining both, with participants free to build their own chunking strategy for the full task ^[11]. Crucially, the track does not score the retrieval and generation tasks with one instrument: it applies nugget evaluation, which measures how many of the essential facts an answer covers; support evaluation, which checks whether each part of the generated answer is actually grounded in the segments it cites; fluency evaluation of the writing itself; and a separate retrieval evaluation of the ranked segment lists — with an initial pass of LLM-generated relevance and nugget assessments subsequently curated by NIST human assessors rather than accepted automatically ^[11]. The structure mirrors the argument of this article almost exactly: even a track explicitly named for retrieval-augmented generation refuses to reduce its scoring to one number, and keeps a human curation step over the machine-generated first pass rather than trusting it outright.

RAGTruth approaches the same problem from the annotation side rather than the track-design side. It is a corpus of nearly eighteen thousand naturally generated RAG responses across multiple large language models, with hallucinations annotated by humans at both the case level and the individual word level, together with a rated intensity for each instance ^[10]. Word-level annotation is a materially finer instrument than a single faithful-or-not label per response: a response can be mostly faithful with one fabricated clause embedded in it, and a case-level label would either miss the fabrication or unfairly condemn an otherwise well-grounded answer. That granularity is expensive — it is exactly the kind of annotation effort an automated judge is built to approximate — but it is also the calibration target the approximation is ultimately answerable to. An LLM-as-judge faithfulness score that has never been checked against word-level human annotation on data resembling its actual deployment is an unvalidated instrument, however plausible its formula looks on the page.

Contamination and leakage: when the reference set stops measuring anything

Every retrieval metric in this article depends on one artefact staying trustworthy: the labelled query set and its relevance judgements. That trust has a specific failure mode particular to the era of models trained on enormous, loosely audited web crawls — the reference set itself, or the answers associated with it, ending up inside a model’s training data before it is ever used to evaluate that model.

The mechanism is not exotic. A benchmark released as a public dataset, a paper, or a web page is exactly the kind of content large-scale pretraining crawls are built to ingest. A survey of benchmark data contamination in large language models lays out the resulting problem directly: leading models trained on web-scale corpora can inadvertently incorporate benchmark data into their training sets, which inflates measured performance in ways that do not reflect genuine capability, and the survey catalogues detection methods and alternative assessment strategies developed specifically to counter it ^[12]. For RAG evaluation the consequence is sharper than for a plain language-model benchmark, because contamination corrupts exactly the term this article has been isolating: a contaminated model can produce a correct, fluent answer with the retriever disabled entirely, which means $P(\text{correct} \mid \lnot R)$ is inflated by memorisation rather than by any generalisable capability, and every end-to-end score computed against that reference set overstates what the retrieval component is doing.

Two design responses follow, and both are visible in the benchmarks already discussed. The first is temporal: CRAG was explicitly built around questions with varying temporal dynamism, from entities that change over years down to ones that change by the second, so that a model’s memorised prior is worth less the more recent and volatile the fact is — freshness is a defence against leakage almost by construction ^[9]. The second is procedural: keep reference sets closed, rotate them, and never publish a fixed evaluation set as a static, permanently reusable artefact once frontier models are known to train on broad web crawls. Neither response is free. A reference set that never leaks is also a reference set no outside party can independently audit, and the tension between reproducibility and contamination resistance is not solved by any of the frameworks covered here — it is a standing trade-off any team building a RAG evaluation pipeline has to choose a side of, deliberately, rather than by default.

Predictions, with the observations that would falsify them

These are forecasts, clearly separated from the sourced analysis above. Horizon: 11 August 2028.

One. Reported RAG evaluations will increasingly disclose retrieval-only and generation-only sub-scores as standard practice rather than an end-to-end number alone, because the decomposition argued for here is already visible in the frameworks and tracks cited. Disconfirmed if mainstream RAG evaluation tooling and leaderboards in 2028 still report a single blended accuracy figure as their primary published metric.

Two. Prediction-powered calibration of automated judges against a small human-labelled subset, as ARES demonstrates, will become the default pattern for production LLM-as-judge pipelines rather than a research technique. Disconfirmed if production RAG evaluation pipelines in 2028 typically deploy an LLM-as-judge with no held-out human-labelled calibration set at all.

Three. Benchmark contamination concerns will push RAG-specific benchmarks toward rolling or time-boxed reference sets rather than fixed, permanently published ones, following the pattern CRAG’s temporal-dynamism design already points toward. Disconfirmed if the RAG benchmarks in widest use in 2028 are still the same static, fully public reference sets in circulation today, unrotated and unrefreshed.

Four. Word-level or claim-level human annotation, in the style RAGTruth uses, will remain the calibration standard automated faithfulness judges are checked against, rather than being replaced by fully automated verification with no human layer at all. Disconfirmed if a leading RAG evaluation framework in 2028 ships with published evidence that it no longer requires any human-annotated calibration data to be trusted.

None of these requires a capability discontinuity. They follow from the structure already visible: two measurement objects that use disjoint inputs, an automated approximation that is only as good as its last calibration against a human layer, and a reference set whose value decays the moment it becomes part of what a model was trained to predict.

What to take away

A retrieval metric and a generation metric are not two views of the same fact. They are computed from different inputs, answer different questions, and can point in opposite directions at once. Recall at $k$, MRR and nDCG score a ranking function against labelled relevance judgements and never see the model’s written words. Faithfulness, answer relevance and groundedness score the written words against whatever context the retriever handed over and never check whether that context was the right one to hand over. An end-to-end accuracy figure blends both into a single number that cannot be decomposed after the fact, and the decomposition in this article shows precisely what gets lost: a system can answer correctly with no useful retrieval at all, and a system can retrieve perfectly and still fail to write a faithful answer, and the aggregate score treats both as the same outcome.

Ask, before trusting any reported RAG number, which of the two instruments produced it, whether the automated judge behind it has ever been checked against a human-labelled subset resembling the deployment it is scoring, and whether the reference set it was computed against is recent enough that a frontier model could not simply have memorised the answer. A system that can answer those three questions has a measurement pipeline. A system that reports one accuracy number has a scoreboard nobody has taken apart.