Retrieval Is an Evidence System, Not a Memory

The word that does the damage

Ask an engineering team what their retrieval layer is for and most will answer, in some phrasing, that it gives the model memory. The assistant “remembers” the handbook. It “remembers” last quarter’s incidents. It “remembers” the customer. The description is comfortable, and it is wrong in a way that generates specific, repeatable, expensive failures.

A memory is a store a system can consult at will, whose contents it knows it holds, and whose absence it can notice. Retrieval is none of these. A retrieval-augmented system does not know what is in its corpus. It cannot tell that the document it needed was never indexed, or that the paragraph it needed was cut in half by a chunker eighteen months ago. It has a query, a ranking function, and whatever that ranking function happened to return. What lands in the context window is not recollection. It is evidence submitted in support of an answer that has not been written yet — and like all evidence it can be incomplete, mis-selected, merely adjacent, or planted.

A closed-stack library is the honest physical analogue. The reader never enters the store. A slip goes down the tube, and some minutes later a volume arrives on the counter. The reader does not remember the library’s holdings; the reader reasons from what the slip fetched. If the slip was wrong, the reasoning proceeds confidently from the wrong book, and nothing in the reading room signals the error.

This article works through the mechanism in that order: what the original formulation claimed, how the two families of retriever fail in different directions, why chunking is a commitment made before any question exists, why position in the window changes whether evidence is used at all, why a longer window does not subsume retrieval, what separates a supported claim from an adjacent one, how to measure the two halves apart, and why retrieved text is untrusted input arriving at a privileged interpreter.

What the original formulation actually claimed

The framing that “the model knows things” has a respectable origin. Petroni and colleagues showed that pretrained language models recover a surprising amount of relational knowledge under cloze-style probing without any fine-tuning, competitive with conventional extraction pipelines ^[3]. That result is real, and it is also the source of the confusion: it establishes that parameters store something knowledge-like, which invites the inference that a retrieval layer is simply more of the same, bolted on.

The retrieval literature said the opposite. Lewis and colleagues introduced RAG explicitly as a combination of parametric and non-parametric memory, and the architectural point is that the second is a different kind of object, not an extension of the first ^[1]. Their formulation treats the retrieved passage as a latent variable to be marginalised over. Writing $x$ for the query, $y$ for the output, and $\mathcal{Z}_k(x)$ for the top $k$ passages returned by a retriever with parameters $\eta$:

Read the structure rather than the arithmetic. The generator is never conditioned on the corpus. It is conditioned on $z$ — one span, or a handful — and its output distribution is a weighted account of what those spans support. Everything outside $\mathcal{Z}_k(x)$ has probability zero of influencing the answer, and the system has no representation of what it excluded. REALM made the same commitment on the training side, learning a latent retriever end-to-end so that the model attends over documents drawn from a corpus at pretraining, fine-tuning and inference time, and reporting 4–16% absolute gains on open-domain question answering over prior methods ^[2].

Two consequences follow immediately, and both are routinely violated in practice. First, the retriever is a component of the model’s likelihood, not a preprocessing step; its errors are not recoverable downstream. Second, the object supplied is evidential, so an honest system’s answer should be legible as an argument from specific spans — which is why attribution, treated later, is not a nice-to-have reporting feature but the natural output type of the architecture.

Two retrievers, two vocabularies, one corpus

Lexical retrieval scores a document by term overlap with the query, weighted by term rarity and normalised for document length; the BM25 family is the canonical instance. Embedding retrieval encodes query and passage independently into fixed-length vectors and ranks by inner product. Karpukhin and colleagues established the modern dense form, a simple dual encoder trained on a modest number of question–passage pairs, and reported 9–19% absolute improvement in top-20 passage retrieval accuracy over a Lucene BM25 baseline ^[4].

That result is often cited as though it settled the matter. It did not, and the reason is structural rather than empirical. Luan and colleagues analysed the capacity of fixed-length dual encoders and drew out the relationship between encoding dimension, the margin separating a gold document from lower-ranked ones, and document length: a fixed vector has a bounded ability to preserve the distinctions needed for precise retrieval as documents grow, which sparse bag-of-words representations do not share. They proposed combining the two precisely to recover sparse retrieval’s precision ^[5].

The generalisation problem is the second half. BEIR evaluated ten retrieval systems across eighteen heterogeneous datasets and found that dense retrievers, strong in-domain, often underperform out of domain, while BM25 remains a robust baseline and re-ranking or late-interaction models achieve the best zero-shot results at materially higher compute cost ^[6]. This is exactly what one expects from a learned function evaluated off its training distribution — and every production corpus is off-distribution relative to the retriever’s training data.

So the two families fail in opposite directions. Lexical retrieval fails on vocabulary mismatch: the query says “cannot log in”, the document says “authentication failure”, and overlap is nil. Embedding retrieval fails on precision: it returns passages that are topically adjacent and factually irrelevant, and it silently loses rare identifiers — a part number, a version string, an error code — because those tokens carry little semantic mass in a learned space and enormous discriminative weight in a lexical one.

Hybrid wins because the failures are close to independent. The simplest fusion, reciprocal rank fusion, ignores scores entirely and combines rank positions across retrievers $R$ with a smoothing constant $k$:

Cormack and colleagues introduced it and showed it consistently beat both the individual systems and Condorcet fusion ^[7]. Its virtue is that it needs no score normalisation, which matters because a BM25 score and a cosine similarity are not commensurable quantities. Its cost is that it discards score magnitude. Bruch and colleagues analysed fusion functions directly and found convex combination of normalised scores outperformed reciprocal rank fusion both in-domain and out-of-domain, and — contrary to the folklore — that reciprocal rank fusion is in fact sensitive to its parameter ^[8]. The practical reading: hybrid is not optional, and the fusion function is a tuned component rather than a default to be copied from a tutorial.

Chunking is a decision made before the question exists

Every retrieval system commits, at index time, to a unit of evidence. That commitment is made once, for all future queries, by someone who does not know what any of them will be. It is the least examined and most consequential decision in the stack.

The loss is twofold. A chunk severed from its document loses the referents that made it interpretable: the section it sat under, the product it described, the date it applied to, the negation three paragraphs earlier that reversed its meaning. Günther and colleagues state the problem plainly — chunk embeddings produced by encoding segments in isolation lose contextual information from surrounding chunks, yielding sub-optimal representations — and propose late chunking, applying segmentation after a long-context encoder has processed the whole text so that each chunk vector carries the document’s context ^[9].

The second loss is retrieval-side rather than representation-side. If the answer to a question spans a chunk boundary, no top-$k$ over those chunks can supply it in one piece; the system must either retrieve both halves and hope the generator reassembles them, or fail. Overlap windows mitigate this and do not solve it, because the correct overlap depends on the query.

The vendor evidence points the same way and should be read as a vendor assertion rather than an independent finding. Anthropic reports that prepending model-generated context to each chunk before embedding reduced the top-20-chunk retrieval failure rate by 35% (5.7% to 3.7%); that adding contextual BM25 took the reduction to 49% (to 2.9%); and that adding a reranking stage reached 67% (to 1.9%) ^[10]. These are the publisher’s own numbers on their own evaluation. The shape of the result nonetheless corroborates the peer-reviewed literature on two points: context restored to chunks helps, and lexical retrieval fused with dense retrieval helps again on top of it.

The design conclusion is uncomfortable but clear. Chunking is not a parameter to tune once; it is a lossy encoding of your corpus, and the loss is query-dependent. Systems that treat it as a fixed 512-token slice with 50 tokens of overlap have made an assumption about every future question and recorded it nowhere.

Where the evidence lands changes whether it is used

Suppose retrieval succeeds. The relevant passage is in $\mathcal{Z}_k(x)$, correctly chunked, in the prompt. The system may still fail, for a reason that has nothing to do with retrieval quality.

Liu and colleagues varied the position of the relevant document within a long input and found performance highest when it appeared at the beginning or the end, degrading significantly when the model had to use information in the middle — including in models explicitly built for long contexts ^[11]. This is a positional bias in usage, not in retrieval: the evidence is present and is not used.

The consequence for system design is that the ordering of retrieved spans is a live parameter, and the intuitive ordering is wrong. Ranking descending by relevance places the second- and third-best evidence exactly where it will be attended to worst. A fold that places top-ranked spans at both ends of the retrieved block is a cheap intervention that most stacks do not make.

Two adjacent results compound this. Shi and colleagues showed that adding irrelevant context to arithmetic reasoning problems dramatically decreased accuracy, with self-consistency decoding and explicit instructions to ignore irrelevant material serving as partial mitigations ^[12]. Chen and colleagues benchmarked RAG systems along four axes — noise robustness, negative rejection, information integration and counterfactual robustness — and found that while models show some tolerance for noise, they struggle badly at declining to answer when the corpus does not contain the answer, at integrating across documents, and at handling false retrieved information ^[18].

Read together, these say something specific about the top-$k$ parameter. Increasing $k$ raises the probability that the answer is present and simultaneously raises the volume of distractors and the chance that the answer sits mid-block. Recall and usability move in opposite directions, so $k$ has an interior optimum that must be measured rather than assumed, and the optimum is corpus-specific.

Why a longer window does not replace retrieval

The recurring proposal is to skip retrieval and put the corpus in the prompt. Three separate objections apply, and none is about cost alone.

The first is that advertised context length is not usable context length. RULER extended needle-retrieval testing into multi-hop tracing and aggregation and found that although all evaluated models claimed context sizes of 32K tokens or more, only half maintained satisfactory performance at 32K, with large drops as length grew ^[13]. A window is a capacity claim; effective length is an empirical quantity, and the two diverge.

The second is that the positional problem does not disappear at scale — it is a property of how long inputs are used, and the “lost in the middle” degradation was observed in long-context models specifically ^[11]. Extending the window extends the middle.

The third is economic and is the only one where the long-context case is genuinely strong. Li and colleagues compared the two approaches across public datasets and three then-current models and found that when resourced sufficiently, long-context processing consistently outperformed retrieval on average performance, while retrieval remained significantly cheaper. Their proposal, Self-Route, routes each query to one path or the other based on the model’s own assessment of whether it can answer, achieving comparable performance at substantially reduced cost ^[14].

That finding is worth stating precisely, because it is frequently mangled in both directions. It does not show that retrieval is obsolete; it shows that on benchmark corpora that fit in a window, spending the tokens buys accuracy. Most production corpora do not fit in any window, are updated continuously, are subject to per-user access control, and must produce citations. Each of those is an independent reason retrieval survives regardless of window length: retrieval is not a compression hack for small windows, it is the mechanism by which a system is restricted to evidence it is permitted and able to justify.

Supported, or merely adjacent

Here is the distinction that separates a system whose answers can be trusted from one whose answers merely look sourced.

Rashkin and colleagues formalised the standard with Attributable to Identified Sources: a statement is attributable to a source if the source supports it, assessed through a two-stage annotation pipeline, with the goal of establishing a common framework for whether model-generated statements are supported by underlying sources ^[15]. The operative word is “supports”. A retrieved span that mentions the same entity, or discusses the same topic, or contains the same numbers in a different relation, is adjacent. Adjacency is what an embedding retriever optimises for and what a careless attribution check accepts.

The empirical picture is not reassuring. Gao and colleagues built ALCE to evaluate citation quality automatically along fluency, correctness and citation dimensions, and found that even the best systems left claims without complete citation support half the time on the ELI5 subset ^[16]. That is a system producing citation markup at a rate substantially higher than its rate of producing citation validity, which is the worst possible arrangement: the visible signal of groundedness decouples from the property it signals, and a reader who checks the presence of a citation rather than its content is worse off than one with no citations at all.

The practical test is mechanical and should be automated. For each claim in the output, ask whether the cited span, read alone and without the rest of the document, entails the claim. Not “is consistent with”, not “mentions” — entails. Claims failing that test should be surfaced, not silently emitted with a superscript beside them.

Measure the two halves apart

Because retrieval and generation are separable, an end-to-end score is nearly uninformative about which one broke. Decompose it. Let $\mathcal{Z}_k(x)$ be the retrieved set and $S(x)$ the set of spans that would suffice to answer $x$:

The first factor is a pure retrieval quantity: recall at $k$, measurable against relevance judgements with no generator involved, using the established information-retrieval apparatus that BEIR standardised ^[6]. The second is a pure generation quantity: given that sufficient evidence was present, did the model use it faithfully? Es and colleagues built RAGAS to score exactly this separation — retrieval effectiveness, the faithfulness with which the model uses retrieved passages, and generation quality — as a reference-free evaluation that does not require ground-truth answers, enabling faster iteration ^[17].

Three rules follow, and they are the practical content of this section.

Never report only end-to-end accuracy. A drop of five points is a different engineering task depending on which factor moved, and the aggregate cannot tell you.

Build a retrieval-only test set first. Queries paired with the identifiers of spans that answer them, scored with recall at $k$ and nDCG. It is cheaper to build than an answer-labelled set, it is reusable across generator changes, and it is the only artefact that survives a model swap.

Measure abstention as a first-class outcome. The negative-rejection weakness reported by Chen and colleagues means that a system evaluated only on answerable queries is measuring the easy half of its behaviour ^[18]. Include queries whose answers are absent from the corpus and score silence as success.

The returns trolley: retrieved text is untrusted input

The final consequence of the evidence framing is a security consequence, and it is the one that has generated real incidents rather than merely disappointing benchmarks.

A retrieval pipeline takes content of arbitrary provenance — web pages, uploaded files, ticket bodies, email, wiki edits — and inserts it into a prompt that an agent will act on. Greshake and colleagues named this indirect prompt injection and demonstrated it against deployed systems, arguing that processing retrieved prompts can amount to arbitrary code execution: it can manipulate application functionality and control how and whether other APIs are called, enabling data theft, self-propagating attacks and ecosystem contamination ^[19]. OWASP’s Gen AI Security Project classifies prompt injection as LLM01 and defines the indirect variant as external content that, when interpreted by the model, alters its behaviour in unintended ways, recommending that untrusted content be separated and clearly denoted to limit its influence ^[20].

The structural point is that retrieval erases provenance. A span from a peer-reviewed internal specification and a span from a comment field an anonymous user edited last night arrive in the same block, in the same format, with the same implicit authority. The generator has no channel by which to distinguish them, because the prompt is a flat sequence and the trust boundary was never encoded in it.

If retrieval is memory, this is inexplicable — nobody expects their own memories to be adversarial. If retrieval is evidence, it is obvious: evidence has provenance, provenance determines weight, and a court that admitted anonymous documents at equal weight to authenticated ones would not be a court. The engineering corollaries are the standard ones for untrusted input, applied at the right boundary: mark trust level per span and carry it into the prompt; never let a retrieved span’s instructions widen the tool permissions of the turn that retrieved it; require that any consequential action be justified by a span whose provenance is at least as trusted as the action is dangerous.

Predictions, with the observations that would falsify them

These are forecasts, separated from the sourced analysis above. Horizon: 8 August 2028.

One. Retrieval-only evaluation sets — queries paired with sufficient-span identifiers, scored independently of any generator — will become a standard deliverable of production RAG projects rather than a research artefact. Disconfirmed if mainstream RAG tooling in 2028 still ships end-to-end answer scoring as its only first-class evaluation.

Two. Hybrid lexical-plus-dense retrieval with a tuned fusion function will remain the default in production, and pure-dense stacks will continue to be the common cause of “it cannot find the part number” bugs. Disconfirmed if a dense or learned-sparse retriever demonstrates BM25-level robustness across an out-of-domain benchmark of BEIR’s breadth without per-corpus tuning.

Three. Trust labelling of retrieved spans — provenance carried into the prompt and used to gate tool permissions — will move from bespoke engineering into framework defaults, driven by incidents rather than by research. Disconfirmed if the major agent frameworks in 2028 still concatenate retrieved content into prompts with no per-span provenance channel.

Four. Context windows will keep growing and retrieval will not shrink in importance, because access control, freshness, cost and citability are orthogonal to window size. Disconfirmed if systems handling permissioned, continuously updated corpora move predominantly to whole-corpus prompting.

None of these requires a capability discontinuity. They follow from the structure already visible: a selection step whose errors are unrecoverable, a generator that argues only from what it was given, and an input channel that anyone can write to.

What to take away

Retrieval supplies evidence for a claim. It does not give the system memory, and every failure catalogued above is what happens when a team builds the second thing while believing they built the first.

Say the corpus was never entered, the slip alone decides what arrives, and what arrives is the whole basis for the answer. Then the design questions ask themselves. What unit of evidence did the index commit to, and what did that commitment destroy? Which retriever’s blind spot is my corpus shaped like, and what is fused against it? Where in the window does the best span land? Does each claim’s cited span entail it, or merely sit near it? Which factor moved when the score dropped? And who, exactly, is allowed to write into the store that my privileged interpreter reads from?

A system that can answer those six questions has an evidence pipeline. A system that answers “it remembers our documentation” has a ranking function nobody has measured, feeding a generator nobody has constrained, from a corpus nobody has bounded.