What Actually Happens Between a Query and an Answer in RAG

A query is not answered; it is routed

Ask what a retrieval-augmented system does with an incoming question and most descriptions collapse three or four separate operations into one word: “retrieve.” That word is doing far too much work. Between the moment a user’s text arrives and the moment a generator begins producing tokens, a production RAG stack typically performs six distinct operations, built by different research communities, tuned on different objectives, and failing in different ways. A companion piece in this publication argued that retrieval should be understood as an evidence mechanism rather than a memory. This one takes that mechanism apart and looks at what happens on the workbench: query rewriting and expansion, two parallel first-stage searches built on incompatible premises, fusion of their disagreeing rankings onto one scale, an expensive rescoring pass over a deliberately short list, assembly of what survives into an ordered and budgeted context, and finally a generation step conditioned on that assembled object rather than on the corpus itself.

None of these stages is optional in a well-built system, and none of them is “the retrieval step” on its own — the stage most people mean when they say that word is usually just the third of six. The rest of this article walks the pipeline in the order a request actually travels through it, citing the paper or the shipped system that defines current practice at each stage.

What the user typed is not what gets searched

The first operation a well-built pipeline performs on a query is to stop treating it as the search string. Raw user text is frequently a poor match for the vocabulary and structure a retriever needs: it is short, it is conversational, it references things anaphorically (“what about the second one”), and it often states a need rather than the terms that would satisfy it.

Two distinct fixes have become standard, and they solve different problems. The first is rewriting: replacing the user’s text with a reformulation better suited to search, optionally trained end-to-end against the reader that will eventually consume the result. Ma and colleagues formalised this as Rewrite-Retrieve-Read, inserting a rewriting step before retrieval and training a small rewriter language model with reinforcement learning against feedback from a frozen downstream reader, explicitly to close the gap between the input text and the knowledge actually needed for retrieval ^[7]. The rewriter is not answering the question. It is producing a better question to search with, and it is scored on whether that improves what comes back.

The second fix is expansion into a different representation entirely. Gao and colleagues’ HyDE has the generator write a hypothetical answer to the query — a plausible-sounding passage that may contain factual errors — and then embeds that hypothetical document rather than the query itself, using the encoder’s dense bottleneck to filter the specifics back out while keeping the pattern of relevance; the method is reported to outperform the prior state-of-the-art unsupervised dense retriever, Contriever, without using any relevance labels ^[6]. The intuition is unintuitive on first read: a query and a good answer are lexically closer to each other than a query is to a real supporting passage, so searching with a fabricated answer’s embedding can retrieve real passages more precisely than searching with the question’s embedding. Nothing about the hypothetical document needs to be true. It exists only to be embedded.

Both techniques share a structural consequence worth stating plainly: by the time anything is searched, the string that reaches the index is not the string the user typed. Debugging a retrieval miss by re-reading the user’s original question is therefore frequently pointless — the failure may already have happened one step earlier, in what the query became.

Two searches, run at the same time, from two different premises

With a search query settled, most production systems do not run one retriever. They run two, in parallel, over separately built indexes of the same corpus, because the two approaches fail in close to independent ways.

Lexical retrieval scores a document by how well its terms overlap the query’s terms, weighted by how rare each term is and normalised for document length. The canonical scoring function, still the default first-stage ranker across a large share of production search two decades after its formulation, is BM25:

Robertson and Zaragoza’s account of the probabilistic relevance framework behind this formula is worth reading past the equation for one design choice it exposes: the term-frequency component saturates rather than growing linearly, so a document repeating a query term fifty times is scored only marginally higher than one repeating it five times, and the length-normalisation term $b$ trades off penalising long documents against rewarding genuinely comprehensive ones ^[1]. BM25 has no notion of meaning. A query for “cannot authenticate” and a document about “login failure” share no terms and score zero, regardless of how obviously related a person would find them.

Dense retrieval fixes exactly that blind spot by encoding query and passage independently into fixed-length vectors with a learned encoder and ranking by inner product. Karpukhin and colleagues’ Dense Passage Retrieval established the now-standard dual-encoder recipe — trained on a comparatively modest number of question-passage pairs — and reported a 9 to 19 percentage point absolute improvement in top-20 passage retrieval accuracy over a Lucene BM25 baseline across several open-domain question-answering benchmarks ^[2]. A dense retriever finds “login failure” for “cannot authenticate” without either string appearing in the other, because it is scoring semantic proximity in a learned space rather than lexical overlap.

That same learned space is also where dense retrieval loses information lexical retrieval never had to worry about. A fixed-length vector has a bounded capacity to preserve fine distinctions, and rare, high-information tokens — a part number, an error code, a version string — carry little weight in a space trained to capture topical similarity, even though they carry enormous discriminative weight in a bag-of-words index. Two further lines of retrieval research sit between these poles rather than choosing one side. SPLADE learns sparse representations directly, using an explicit sparsity regulariser and a log-saturating term-weighting function so that the resulting document and query representations remain compatible with an ordinary inverted index while being trained end-to-end like a dense model ^[3]. ColBERT keeps independent encoding for efficiency but replaces the single-vector bottleneck with a late-interaction step: query and document tokens are each encoded separately, and relevance is computed afterward by a cheap token-level matching operation, reported to execute two orders of magnitude faster and at four orders of magnitude fewer FLOPs per query than a full cross-attention reranker of comparable quality ^[4].

None of these approaches is strictly better across corpora and query types than the others; each is a different bet on where the compute and the capacity should go. That is precisely why production stacks tend not to bet on just one — which is what makes the next stage necessary.

Merging two rankings that were never on the same scale

A BM25 score and a cosine similarity are not the same kind of number. One is an unbounded sum shaped by corpus term-frequency statistics; the other is bounded between negative one and one and shaped by the geometry of an embedding model’s training run. Averaging them directly is not meaningless so much as it is arbitrary — there is no principled exchange rate between the two currencies without a normalisation step chosen and validated against real queries.

Reciprocal rank fusion avoids the currency problem by discarding scores altogether and combining rank positions instead. For a document $d$ and a set of ranked result lists $R$, with $r(d)$ the rank of $d$ in a given list and a smoothing constant $k$:

Cormack, Clarke, and Buettcher introduced the method and reported that it consistently outperformed both the individual ranked lists and a Condorcet-style fusion baseline, with the specific virtue that it requires no score normalisation across systems with incompatible scoring functions ^[9]. That virtue is also its limitation: two documents ranked first in their respective lists count identically toward the fused score regardless of how much more confidently one system ranked its top result than the other, because rank position is all the method sees.

This is not an academic footnote. It is exactly how at least one major production search engine implements the fusion stage today: Elasticsearch’s own reference documentation describes an rrf retriever that combines the ranked output of two or more child retrievers — commonly a lexical BM25 query and a k-nearest-neighbour vector search — using the formula above with a configurable rank_constant (default 60) and an explicit design rationale that this removes “the need to figure out what the appropriate weighting is using linear combination” across incompatible relevance signals ^[12]. That is a vendor’s documented account of its own system’s rationale, not an independent evaluation, and it should be read as one — but it corroborates the peer-reviewed finding on the point that matters here: fusing by rank rather than by raw score is the practical, engineered answer to the units problem, adopted in shipped infrastructure rather than only in papers.

The fused list produced at this stage is not the final answer set. It is a candidate set — typically hundreds of documents wide — assembled specifically to be handed to something that cannot afford to look at the whole corpus but can afford to look closely at a few hundred items.

The expensive pass over a list that has already been made short

Fusion produces a wide, cheaply computed candidate list. Reranking narrows it with an expensive, carefully computed score, and the entire justification for running two separate stages rather than one is a compute budget that a single stage could not meet.

Nogueira and Cho’s passage reranking work made the case in its simplest form: pass the query and a candidate passage together through a full cross-attention transformer — BERT, at the time — so that every query token can attend to every passage token before a relevance score is produced, rather than comparing two independently computed vectors. Their reported result was substantial: state-of-the-art performance on TREC-CAR and the top entry on the MS MARCO passage-ranking leaderboard, a 27% relative improvement in MRR@10 over the prior best method ^[5]. The cost of that joint attention is exactly why it cannot run as the first stage: a cross-encoder scores one query-passage pair at a time and cannot be precomputed or indexed the way a bi-encoder’s vectors can, so running it over an entire corpus, or even over the raw output of a single retriever, is computationally out of reach at production latency.

Vespa’s own ranking documentation describes the production answer to this constraint directly, structuring the ranking computation itself into ordered phases: an inexpensive match phase and first-phase function evaluated over everything that was retrieved, followed by a substantially more expensive second-phase function restricted to only the top candidates surfaced by the first, with the documented rationale that “a good quality ranking expression will for most applications consume too much CPU to be runnable on all retrieved or matched documents within the latency budget,” and that the practical engineering task is to “find a first-phase function which correlates sufficiently well with the second-phase function” so that the cheap pass does not discard anything the expensive pass would have kept ^[11]. That documentation describes Vespa’s own architecture and should be read as a vendor’s account of its own system, but the phase structure it describes — cheap pass over everything, expensive pass over a short list bought by the cheap pass — is the same structure the reranking literature converges on independently, which is why it is worth citing as a real, shipped instance of the pattern rather than only its research description.

The number of candidates carried into this stage, not the number surfaced by any single first-stage retriever, is usually the parameter that most directly trades cost against recall in a production system: too narrow and the reranker never sees the right passage; too wide and its per-pair cost dominates the request’s latency budget.

What actually goes into the prompt

Reranking produces an ordered list of scored passages. What reaches the generator is not that list — it is a curated, budgeted, deduplicated subset of it, assembled by a stage that gets far less attention than either retrieval or generation despite making decisions that directly determine what the model can possibly say.

Three separate decisions happen here, and conflating them is a common source of quiet failure. Ordering decides the sequence in which surviving passages appear in the prompt, and it is not neutral: Liu and colleagues found that performance on tasks requiring use of a specific piece of information within a long input was highest when that information appeared at the very beginning or the very end of the context, and degraded significantly when it had to be used from the middle — a pattern the authors observed even in models built specifically for long contexts ^[13]. A context-assembly stage that simply places passages in descending rerank-score order puts the second- and third-best evidence exactly where a generator is least likely to use it; a stage that instead interleaves top-ranked passages toward both ends of the block is a deliberate, cheap correction that a naive pipeline does not make on its own.

Deduplication removes near-redundant passages before they consume budget that could go to genuinely new evidence. Goldstein and Carbonell’s maximal marginal relevance criterion, developed originally for reordering retrieved documents and selecting summary content, formalises the trade-off directly: rank each remaining candidate not purely by its relevance to the query but by relevance discounted by its similarity to items already selected, so that a passage nearly identical to one already in the assembled set contributes little marginal value regardless of how highly it individually scored ^[10]. Two passages can both be highly relevant and highly redundant with each other; a context-assembly stage that packs both has spent budget for the effect of one.

The token budget is the hard constraint the first two decisions operate under. Every model serves a request against a finite context window, and every token spent on a marginal ninth passage is a token not spent on the instructions, the conversation history, or the eventual answer. A budgeted assembly stage packs passages greedily against that ceiling in whatever order the first decision established, stopping — or substituting a shorter passage for a long one — the moment the budget would be exceeded, which means the passage that “would have been” ninth-most-relevant may never reach the model at all, not because it was judged unhelpful but because the tray was already full when its turn came.

None of these three decisions is visible in a rerank score. A system that reports only its retrieval and rerank metrics has not measured what the generator actually received.

Generation is conditioned on a small assembled object, not a corpus

The final stage is the one most descriptions of RAG treat as an afterthought — “then the model answers using the retrieved context” — but it has its own architectural history and its own design choices.

The cleanest illustration of what it means to condition generation on multiple passages by construction, rather than by convention, is Izacard and Grave’s Fusion-in-Decoder. Each retrieved passage, concatenated with the question, is encoded independently and in parallel by the encoder half of a sequence-to-sequence model; only at the decoder does the architecture attend jointly across all of the resulting representations at once. The paper’s central empirical observation is that performance continues to improve as the number of retrieved passages grows, because the encoding cost scales linearly with passage count while the more expensive joint attention happens only once, in the decoder, rather than being repeated per passage — and the approach reached state-of-the-art results on the Natural Questions and TriviaQA open-domain benchmarks on that basis ^[8].

That architecture is a deliberate answer to a scaling problem: ordinary self-attention over many long passages concatenated together grows quadratically with total input length, while fusing after independent encoding avoids paying that cost per passage. It is worth naming explicitly because it is not how most deployed large-language-model RAG stacks work today. The common production pattern — a single decoder-only model with retrieved passages concatenated as plain text inside one prompt — pays the quadratic cost Fusion-in-Decoder was built to avoid, in exchange for architectural simplicity and compatibility with an unmodified, general-purpose model API. That is a description of prevailing engineering practice, not a claim from any cited source: it follows from the fact that general-purpose chat and completion APIs expose a single text context rather than a mechanism for passing separately encoded passages, so a system built on top of one has little choice but to concatenate. Whichever architecture is in use, the underlying commitment is the same one this whole pipeline has been building toward: the generator’s output is conditioned on the assembled context object and nothing else. It cannot fall back on the corpus, and it cannot fall back on what was cut for space.

What production systems actually ship

It is worth stepping back from the individual stages to notice what the two pieces of vendor documentation cited above have in common, because it is not an accident of which companies happened to publish accessible docs. Vespa’s ranking documentation and Elasticsearch’s fusion documentation each describe, independently, a pipeline broken into named, configurable phases that closely track the stages walked through here: cheap retrieval across the whole matched set, an explicit combination step where more than one ranking signal is reconciled, and a restricted, more expensive pass applied only to what survives ^{[11] [12]}. Neither system was designed by copying the other, and neither is describing a research prototype; both are documenting infrastructure their customers run today. That convergence is evidence, in the weak but real sense that independent engineering teams solving the same latency-and-recall problem arrived at structurally similar answers, that the six-stage decomposition in this article is not an academic simplification imposed after the fact — it is closer to how the systems are actually built, exposed as configuration surfaces a practitioner can name, inspect, and tune independently.

This matters for anyone operating rather than merely designing such a system. Each stage above has its own recall, its own latency budget, and its own failure signature, and a single end-to-end “did the answer look right” evaluation cannot distinguish a query-rewriting failure from a fusion-weighting failure from a token-budget eviction. A rewritten query that drifted from user intent, a fused list that a bad rank_constant skewed toward one retriever, a reranker cutoff set too narrow, a passage evicted by the token budget before ordering ever helped it — these are four different bugs with four different fixes, and they produce the identical visible symptom: a wrong or unsupported answer.

Predictions, with the observations that would falsify them

These are forecasts, separated from the sourced analysis above. Horizon: 11 August 2028.

One. Query rewriting will move from an optional add-on to a default first stage in mainstream RAG frameworks, exposed as a named, configurable step rather than left to bespoke prompt engineering. Disconfirmed if the major open-source RAG frameworks in 2028 still treat the user’s raw text as the default search string with rewriting as an unlabelled optional extra.

Two. Rank-based fusion (reciprocal rank fusion or a close variant) will remain more common in production than learned or score-normalised fusion, because it requires no per-corpus calibration and shipped systems have already standardised on it. Disconfirmed if a learned fusion method becomes the documented default in at least two of the major production search or vector-database systems.

Three. Context-assembly logic — explicit ordering, deduplication, and budget-aware packing, as distinct configurable steps — will become a named component of mainstream RAG frameworks rather than implicit behaviour buried in a prompt template. Disconfirmed if leading RAG frameworks in 2028 still expose no ordering or deduplication controls separate from the retriever’s raw rank.

Four. The gap between decoder-only prompt concatenation and purpose-built multi-passage architectures such as Fusion-in-Decoder will persist rather than close, because general-purpose model APIs will continue to expose a single flat context rather than a structured multi-passage interface. Disconfirmed if a mainstream commercial model API ships a first-class multi-passage conditioning interface distinct from prompt concatenation.

None of these requires a capability discontinuity. They follow from the structure already visible: six stages with different cost profiles, two of them already exposed as named configuration in shipped systems, and a generation interface that has not yet caught up to the architectures built specifically to condition on many passages at once.

What to take away

“The system retrieves the answer” describes none of what actually happens. A query is rewritten or expanded before anything is searched. Two retrievers built on different premises search in parallel and disagree. Their rankings are merged onto a common scale by discarding the very scores that made them incomparable. A short list bought by that merge is rescored by a pass too expensive to run any wider. What survives is ordered, deduplicated, and packed against a hard token ceiling by a stage most evaluations never look at. And only then does a generator condition its output on the resulting object — an object that is not the corpus, not the query, and not even the full rescored list, but whatever was left after every one of the preceding decisions took its cut.

Six stages, six literatures, six separate places for a system to fail quietly while still returning a fluent, confident, wrong answer. Ask which one broke before asking whether “retrieval” is the problem — the word names the whole pipeline, and by now it should be clear that is not a single thing to blame.