From BM25 to Agentic Retrieval: A History of Retrieval-Augmented Generation

A term coined for an old idea

Ask when retrieval-augmented generation was invented and most people give a single date: 2020, the year Patrick Lewis and colleagues at Facebook AI Research published the paper that gave the mechanism its name ^[8]. The date is correct and the impression it leaves is not. What that paper named — condition a generator’s output on a small set of passages fetched by a separate ranking function — is not a 2020 idea. It is the joining of two older lineages: decades of work on ranking documents by relevance to a query, and a shorter but still pre-2020 line of work on using retrieval to answer open-domain questions. The 2020 paper did not invent retrieve-then-generate. It gave an existing pattern a differentiable, end-to-end trained form, and, not incidentally, a name that stuck.

This article follows the lineage in the order it was actually built: sparse ranking functions from classical information retrieval, the retrieve-then-read pipelines that used them for question answering, the arrival of learned dense retrievers, the paper that coined the term, the shift toward feeding a generator many passages at once, the return of sparse methods as one half of a hybrid, and the recent move toward retrieval that is interleaved with reasoning rather than performed once up front. Each link below is dated and sourced to the paper that reported it; where a claim below concerns “the first” system to do something, it goes no further than what that paper’s own authors claimed.

Ranking by geometry: the vector space model

Before any of this involved learning, it involved geometry. Salton, Wong, and Yang proposed representing each document as a vector of weighted index terms and ranking documents by the similarity of their vectors to a query vector, arguing that a well-separated document space — one where unrelated documents sit far apart — should correspond to better retrieval performance than a densely packed one ^[1]. Their paper is worth reading in the original rather than through summary, because the term-weighting scheme it specifies is exactly the ancestor of what every later retriever, sparse or dense, still does: score a term by how often it occurs locally and how rare it is globally. They define the inverse document frequency of a term $k$, for a collection of $n$ documents in which $k$ appears in $d_k$ of them, as

and combine it multiplicatively with raw term frequency so that a term scores highest when it occurs often in one document but rarely across the collection. Evaluated on three test collections in aerodynamics, medicine, and world affairs, replacing raw term-frequency weighting with this scheme, together with a term-discrimination-value model for phrase and thesaurus construction, improved average recall-precision by 17 to 50 percent depending on the collection ^[1].

What survived from this paper is not the specific formula — modern systems use different weighting entirely — but the framing: relevance as proximity in a vector space, computed once at index time and compared at query time. Every dense retriever discussed later in this history is, structurally, still doing that. What changed is what the vector’s coordinates mean.

BM25 and probabilistic weighting at TREC

The Text REtrieval Conference, run annually by the U.S. National Institute of Standards and Technology from 1992 onward, gave information retrieval something the field had mostly lacked: a shared, blind evaluation on a common document set, repeated every year with published results. Robertson, Walker, and colleagues at City University London entered TREC-3 in 1994 with the Okapi system, reporting a term-weighting approach within a probabilistic relevance framework and applying it, among other extensions, to phrase weighting and to query expansion using terms drawn from an initial pilot search ^[2]. Over that and the following TREC rounds, the Okapi team’s tuning of term-frequency saturation and document-length normalization converged on the specific scoring function that the field came to call BM25 — “Best Match 25,” after its position in a numbered sequence of variants the group had tried. Written in its now-standard form, a document $D$ scores against a query $Q$ as

where $f(q_i, D)$ is the frequency of query term $q_i$ in $D$, $|D|$ is the document’s length, $\mathrm{avgdl}$ is the average document length in the collection, and $k_1$ and $b$ are tuned constants controlling term-frequency saturation and length normalization respectively. The saturation term is the substantive advance over a raw term-frequency-times-IDF score: a term’s contribution grows quickly at first and then flattens, so a document that happens to repeat a query word fifty times does not dominate one that uses it three times in the right place. Decades later, this same function remains the default first-stage ranking method built into widely used open-source search engines, which is a strong claim to make about any piece of 1990s software and is offered here as an observation about longevity rather than as evidence that nothing since has improved on it.

Retrieve then read: open-domain question answering before neural retrievers

The specific pipeline shape that retrieval-augmented generation would later formalize — retrieve candidate text, then condition an answer on it — was already standard practice in open-domain question answering before any learned retriever existed. Chen, Fisch, Weston, and Bordes built DrQA to answer factoid questions against the whole of English Wikipedia by combining two components: a document retriever using bigram hashing and TF-IDF matching to narrow millions of articles down to a handful of candidates, and a multi-layer recurrent reading-comprehension model trained to locate the answer span within those candidates ^[3]. Both components were evaluated separately and reported to perform competitively on their own, and multitask learning across several QA datasets with distant supervision produced an effective combined system.

Two things about DrQA are worth stating plainly, because they are exactly the two things that later work would each independently change. The retriever was unlearned — the same TF-IDF-family scoring already described, with no training signal specific to the question-answering task — and the retrieval step ran once, before reading began, with no mechanism for the reader to ask for anything further. Every subsequent milestone in this history is a change to one or the other of those two properties: making the retriever learned, or making the retrieval step something other than a single fixed pass.

Learning where to look: dense retrieval arrives

The first property gave way first. Lee, Chang, and Toutanova proposed treating evidence retrieval from all of Wikipedia as a latent variable and training retriever and reader jointly from question-answer pairs alone, with no annotated evidence and no separately built IR system, pretraining the retriever with an Inverse Cloze Task rather than supervised relevance labels; they reported outperforming BM25 by up to 19 points of exact match on datasets where questioners were genuinely seeking an answer they did not already know, while noting that conventional retrieval remained perfectly adequate on datasets where the answer was already known to whoever wrote the question ^[4]. That caveat is easy to miss and matters: even the paper introducing learned retrieval reported that learning it was not a universal win.

Karpukhin and colleagues then showed that end-to-end joint training was not actually required to beat sparse retrieval. Their Dense Passage Retriever trains a simple dual encoder — two BERT-based networks, one for questions and one for passages, scoring a pair by the inner product of their embeddings — directly on a modest number of question-passage pairs with in-batch negative sampling, no specialized pretraining objective at all, and reported a 9-to-19-point absolute improvement in top-20 passage retrieval accuracy over a Lucene BM25 baseline, translating into new state-of-the-art results across several open-domain QA benchmarks ^[6]. Guu and colleagues took the third route, folding retrieval into pretraining itself: REALM retrieves and attends over documents from a large corpus during pretraining, fine-tuning, and inference alike, trained without labelled data by using masked-language-model prediction as the retrieval training signal, and reported large gains in interpretability and modularity alongside accuracy on open-domain QA ^[7]. All three replace symbolic term overlap with a learned embedding space; none of the three, on its own, changed whether retrieval happened once or repeatedly.

The paper that named the mechanism

By the time Lewis and colleagues published in 2020, learned dense retrieval and large pretrained generators both already existed as separate lines of work. The paper’s specific technical contribution was neither — it was making the whole pipeline trainable end to end through a single differentiable objective, treating the retrieved passage as a latent variable to be marginalized over rather than a fixed input handed to a frozen reader ^[8]. Writing $x$ for the input, $y$ for the output, $z$ for a retrieved passage, and $\eta$ and $\theta$ for the retriever’s and generator’s parameters respectively, the RAG-Sequence variant approximates

holding a single retrieved passage fixed across the whole generated sequence, while the RAG-Token variant allows the marginalization to be recomputed at every output token, letting different passages inform different parts of the answer. The retriever itself was a DPR-style dual encoder and the generator was BART; nothing about either component was new. What was new was the training signal running backward through the whole thing, so that the retriever’s parameters could be nudged by whether the passages it surfaced actually helped the generator, rather than being trained once, upstream, and frozen. Evaluated on knowledge-intensive tasks, the resulting models achieved state-of-the-art results and generated language the authors described as more specific, diverse, and factual than comparable parametric-only baselines ^[8]. The name retrieval-augmented generation describes this specific technical move — differentiable retrieval as part of a generation objective — not the broader retrieve-then-read pattern DrQA had already established three years earlier.

Feeding the generator more than one passage

RAG’s marginalization approach treats each retrieved passage as an alternative hypothesis to be weighted and summed. A different, and in practice more scalable, answer to the same problem is to let the generator read several passages at once and combine evidence across them internally. Izacard and Grave proposed Fusion-in-Decoder: encode each retrieved passage independently and in parallel with a shared encoder, concatenate the resulting representations, and let a single decoder attend jointly across all of them when generating the answer. Because passage encoding is independent and only concatenation happens before decoding, the cost of adding more passages grows far more gently than it would if every passage pair had to be jointly encoded, and the paper reported that performance improved as the number of retrieved passages increased, reaching state-of-the-art results on the Natural Questions and TriviaQA benchmarks ^[9].

Izacard and colleagues extended the same fusion-in-decoder idea into Atlas, training retriever and generator jointly in a few-shot regime with an index that could be updated after training rather than frozen at the vocabulary the model saw during pretraining. The headline result is arresting on its own terms: Atlas reached over 42 percent accuracy on Natural Questions using only 64 training examples, outperforming a 540-billion-parameter model despite having 50 times fewer parameters ^[10]. Read carefully, that is not a claim that Atlas’s language model is better than a model fifty times its size. It is a claim about where the knowledge lived — in an updatable external index rather than in weights that had to be trained to memorize it — which is the same structural argument the whole retrieval-augmented lineage has been making since Salton’s vector space, stated at a scale where it became hard to ignore.

Hybrid retrieval and reranking

Dense retrieval did not retire sparse retrieval; it revealed where each one fails, and production systems responded by combining both rather than choosing one. The empirical case is direct: Thakur and colleagues built BEIR, an 18-dataset zero-shot benchmark spanning ten retrieval systems, and found that dense retrievers, strong on their training distribution, often underperform out of domain, that BM25 remains what they call a robust baseline across the whole suite, and that reranking and late-interaction models achieve the best zero-shot results on average, at substantially higher computational cost ^[13]. That is a direct empirical explanation for why hybrid architecture, not any single retriever family, became standard practice.

Two further pieces filled out the hybrid stack. Combining a sparse ranked list with a dense one requires fusing two scores that are not on comparable scales; Cormack, Clarke, and Buettcher had already solved a version of this problem in 2009, years before dense retrieval existed, by proposing Reciprocal Rank Fusion — combining ranked lists purely by rank position rather than by score, shown to consistently outperform both individual systems and Condorcet fusion on the ranked-list-combination task it was designed for ^[12]. The method predates the hybrid-retrieval problem it is now most commonly used to solve, and its adoption there is a reuse of prior art rather than a claim its original authors made about dense retrieval. Reranking, the second piece, trades first-stage recall for second-stage precision: Nogueira and Cho applied BERT to jointly encode a query and a candidate passage as a single pair and score the pair directly, a cross-encoder architecture too slow to run over a full corpus but precise enough to rerank a short first-stage shortlist, reporting a 27 percent relative improvement in MRR@10 on the MS MARCO passage-ranking leaderboard ^[5]. Khattab and Zaharia proposed ColBERT as a middle path between full cross-encoders and single-vector dual encoders: encode query and document tokens separately, as dense retrieval does, but defer their interaction to a cheap late-stage maximum-similarity computation over individual token embeddings rather than compressing everything into one vector in advance, reporting effectiveness competitive with BERT-based rerankers at two orders of magnitude lower latency and four orders of magnitude fewer FLOPs per query ^[11].

From retrieve-once to retrieve-and-reason

Every system discussed so far, whatever ranking function it used, shared DrQA’s second unexamined property: retrieval ran once, before generation began, and the generator had no way to ask for more. The most recent shift in this history is the removal of that constraint.

Nakano and colleagues took the first clear step with WebGPT, fine-tuning GPT-3 to answer questions by issuing search queries and navigating a text-based browsing environment across multiple steps, collecting references in support of its evolving answer, trained first by imitating human demonstrations and then refined with reinforcement learning from human feedback; on the ELI5 dataset, human evaluators preferred its best model’s answers to human demonstrators’ 56 percent of the time and to the highest-voted Reddit answer 69 percent of the time ^[14]. Retrieval here is an action the model itself takes repeatedly, not a pipeline stage that runs before the model sees anything.

Trivedi, Balasubramanian, Khot, and Sabharwal made the coupling between reasoning and retrieval explicit for multi-hop questions with IRCoT, interleaving chain-of-thought generation with retrieval steps on the observation that what to retrieve next depends on what has already been derived — a dependency a single fixed retrieval pass structurally cannot satisfy. Applied across four multi-hop QA benchmarks with GPT-3, the method improved retrieval by up to 21 points and downstream QA by up to 15 points, reduced hallucinated reasoning steps, and worked even with much smaller models such as Flan-T5-large without any additional training ^[15]. Asai, Wu, Wang, Sil, and Hajishirzi moved the decision of whether to retrieve at all inside the model’s own trained behavior with Self-RAG, using special reflection tokens generated alongside ordinary output so the model can decide adaptively whether a given segment needs retrieved support, and can critique the relevance and factual support of what comes back rather than accepting it uncritically; their 7- and 13-billion-parameter models outperformed larger retrieval-augmented and proprietary baselines on open-domain QA, reasoning, and fact-verification tasks, with particular gains in citation accuracy on long-form generation ^[16].

The most recent link trains this behavior directly rather than eliciting it through supervised demonstrations or hand-designed reflection tokens. Jin and colleagues introduced Search-R1, extending reinforcement-learning-based reasoning training so a model learns, through outcome-based reward alone, when to generate a search query, when it has retrieved enough, and how to fold results back into its reasoning trajectory across multiple turns, using retrieved-token masking to keep training stable; across seven question-answering datasets they reported gains of 41 percent for a 7-billion-parameter model and 20 percent for a 3-billion-parameter model over conventional RAG baselines ^[17]. Where WebGPT’s browsing policy and Self-RAG’s reflection tokens were each, in different ways, shaped by human demonstration or hand-authored supervision, Search-R1’s retrieval policy is learned from whether the final answer was correct — the same reinforcement-learning-against-verifiable-outcomes pattern reshaping post-training more broadly, applied here specifically to the question of when to look something up.

What the name obscures: three transitions, not one

Collect the history and a habit of speech turns out to be hiding structure. Practitioners talk about “using RAG” or “not using RAG” as though it names one property a system either has or lacks. The lineage above shows three separable axes that moved independently, on different schedules, driven by different papers.

The first axis is what does the ranking: symbolic term overlap, as in the vector space model and BM25, a learned embedding space, as in DPR and REALM, or a fusion of both, as BEIR’s results argue production systems generally need. The second axis is how much of the pipeline is trained jointly: DrQA’s retriever was fixed and untrained; Lewis and colleagues’ specific contribution was making retriever and generator trainable together through one differentiable objective. The third axis is how many times retrieval happens and who decides: a single fixed pass before generation starts, as in every system through Atlas, or an adaptive, model-directed loop interleaved with reasoning, as in WebGPT, IRCoT, Self-RAG, and Search-R1.

Positioned against those three axes, the 2020 RAG paper sits specifically on the second — trainability — not on the first, since its retriever was already established dense retrieval, and not on the third, since RAG-Sequence and RAG-Token both perform retrieval once per generated output, exactly as DrQA had. What practitioners increasingly mean when they say “agentic RAG” today has moved furthest on the third axis, and the paper that gave the whole family its name never made a claim about that axis at all. Conflating the three is not merely an imprecision of speech; it produces real design confusion, because a system can be weak on one axis and strong on another, and the fix for each is a different piece of engineering.

Predictions, with the observations that would falsify them

These are forecasts, clearly separated from the sourced history above. Horizon: 11 August 2029.

One. The share of new production retrieval systems that perform a single fixed retrieval pass per generated answer will keep shrinking, replaced by policies — trained, as in the Search-R1 lineage, or heuristic — that decide per query whether and how many times to retrieve. Disconfirmed if a survey of production RAG deployments in 2029 finds a majority still issuing exactly one retrieval call per generation.

Two. Sparse lexical scoring, whether classical BM25 or a learned-sparse descendant, will remain present as one signal inside most production retrieval stacks, because the out-of-domain generalization gap BEIR documented for dense-only retrieval will not have closed. Disconfirmed if a single retriever architecture demonstrates BM25-level robustness across a BEIR-scale, multi-domain benchmark without per-corpus tuning, and hybrid stacks are broadly abandoned as a result.

Three. Reinforcement-learning-trained retrieval and search policies will extend from question-answering benchmarks into general-purpose agent tool use, and evaluation practice will lag that deployment — failure modes of trained retrieval policies will surface first through production incidents rather than through benchmark leaderboards. Disconfirmed if, by the horizon, standard tooling for training such policies ships with agreed-upon safety and reliability evaluation as a required default rather than bespoke per-team practice.

None of these requires a capability discontinuity. They are extrapolations of a pattern already visible across three decades: a fixed pipeline stage becomes a decision, a decision becomes a trained policy, and the training signal moves from human demonstration toward verifiable outcomes.

What to take away

Retrieval-augmented generation is not a 2020 invention wearing a 2020 name. It is the current position of a lineage that starts with ranking documents by vector geometry, runs through a probabilistic weighting scheme tuned inside a NIST evaluation series, formalizes into a retrieve-then-read pattern for open-domain question answering, replaces symbolic term matching with a learned embedding space, becomes trainable end to end under a specific differentiable objective in 2020, is scaled by feeding generators many passages at once, is corrected by the return of sparse methods as one half of a tuned hybrid, and is now being rebuilt around a generator that decides for itself, repeatedly, whether it has looked in the right place. Ask what a given system does on each of the three axes — how it ranks, how much of the pipeline is trained jointly, and how many times it is willing to look again — and the vague question of whether something “uses RAG” resolves into three separate, answerable, and separately dated ones.