Ten Failure Modes That Define Production RAG

Ten ways a fluent answer can be a wrong one

A production retrieval-augmented generation system rarely fails by throwing an exception. It answers in well under a second, the response is grammatical, it often cites something, and it reads as competent. The failure is somewhere else entirely: in an index that stopped tracking its source three weeks ago; in a chunk boundary that cut the one clause that reversed a claim’s meaning; in a passage that was retrieved correctly and then never used, because of where it happened to land in the prompt; in an instruction smuggled into a web page the system was told to trust; in a reranker whose top score means something different on Tuesday than it meant on Monday. None of these raise an error. All of them produce a plausible paragraph.

Lewis and colleagues framed retrieval augmentation as a way to combine a model’s parametric knowledge with a non-parametric store that can be inspected, updated and cited independently of the weights ^[1]. That framing is also a warning, read the other way: everything gained by separating the store from the model is a new place for the system to be wrong without the model itself doing anything wrong at all. The ten mechanisms below are chosen because each has independent documentation, in a peer-reviewed paper, a public benchmark, or production-grade vendor evidence, rather than assembled from anecdote or folk wisdom about “RAG being flaky.” Several interact with each other in a running system; a few are usually invisible until someone builds the specific instrument that catches them. That instrument, in every case, sits outside the answer the system produced.

1. Index staleness against a moving corpus

A retrieval index is built from a snapshot of the corpus taken at a particular moment. The corpus keeps changing after that moment: documents are edited, superseded, withdrawn. Nothing in a standard ranking function conditions on time, so a passage that described a policy correctly six months ago and has since been revised scores exactly as high as the passage that replaced it, and often higher, since the older text may be more thoroughly cross-referenced by everything else in the corpus.

Vu and colleagues built FreshQA specifically to probe this failure in language models’ own parametric knowledge, and found that every model they tested, regardless of size, struggled on questions involving fast-changing facts, which motivated search augmentation as a mitigation in the first place ^[2]. Their diagnosis transfers directly to the retrieval layer itself: a store consulted at query time is a snapshot of a particular assembly moment, and its staleness is a function of how long ago that moment was, not of how good the retriever scoring the snapshot happens to be. Under a periodic full rebuild with interval $\Delta$, and changes to the corpus arriving at roughly uniform times within that interval, the expected age of an index entry just before the next rebuild and the worst case obey

which is a statement about the rebuild schedule, not about retrieval quality — a system can have an excellent retriever sitting on top of an index that is, on average, two weeks wrong.

2. Chunk-boundary information loss

Every index commits, at build time, to a unit of retrieval: a fixed-length passage, a paragraph, a sliding window. That commitment is made once, before any query exists, by whoever configured the ingestion pipeline. Chen and colleagues showed the choice is not neutral: indexing a corpus at the level of atomic propositions rather than fixed passages measurably improved both retrieval accuracy and downstream question answering, because a conventional passage boundary routinely separates a fact from the qualifying clause, the exception, or the date that gives it meaning ^[3]. The failure is retrieval-side as well as representation-side — if the true answer to a query spans two adjacent chunks, no top-$k$ selection over those chunks can hand the generator both halves as one coherent unit, and overlap windows only reduce the frequency of the problem, not its existence, because the correct amount of overlap is a property of the query that has not been asked yet.

3. Retrieved-but-ignored evidence: the position effect

Suppose retrieval succeeds outright — the correct passage is in the top-$k$, correctly chunked, sitting in the prompt. The system can still fail for a reason that has nothing to do with what was retrieved. Liu and colleagues varied the position of the one relevant document inside a long input and found performance highest when it sat at the very beginning or the very end, and markedly worse when it had to be used from the middle, a pattern that held even in models built specifically for long contexts ^[4]. This is a usage failure layered on top of a successful retrieval: the evidence is present in the context window, and the generator simply does not draw on it as reliably from the middle of a long list as from its edges. A retrieval pipeline that ranks strictly by descending relevance places its second- and third-best passages exactly where they are least likely to be used, which is the opposite of what the ranking was meant to accomplish.

4. Indirect prompt injection through retrieved content

Retrieval erases provenance. A span pulled from an internal specification and a span pulled from a web page an anonymous author edited an hour ago arrive in the same block, in the same format, with the same implicit authority, because the prompt is a flat sequence of tokens and the trust boundary between them was never encoded in it. Greshake and colleagues named this indirect prompt injection and demonstrated it against deployed systems: content placed anywhere a retrieval-augmented application is likely to fetch it can carry instructions that the application then follows, enabling data exfiltration, unintended tool calls, and self-propagating manipulation, all without the attacker ever touching the prompt directly ^[5]. OWASP’s Gen AI Security Project now classifies this as the top risk in its taxonomy, defining the indirect variant explicitly as external content that alters model behaviour when interpreted, and recommending that untrusted content be separated and clearly denoted before it reaches the model ^[6]. A retrieval-augmented system that treats every retrieved span as equally authoritative has, without deciding to, granted write access to its own instructions to anyone who can get a document indexed.

5. Over-retrieval and context dilution

The intuitive fix for weak recall is to retrieve more: raise $k$, widen the net, let the generator sort it out. Cuconasu and colleagues ran the first systematic study of exactly this move and found the relationship between added context and accuracy is not monotonic in the direction most engineers assume. Passages that are topically related to the query but do not actually contain the answer measurably hurt accuracy when added to the prompt — worse than passages that are entirely unrelated, some of which left accuracy unchanged or, in their reported experiments, improved it by as much as 35% ^[7]. That is a specific, counterintuitive, vendor-independent finding from a controlled study, not a general claim that noise helps; the mechanism they identify is that a near-miss distractor competes for the generator’s attention in a way a genuinely irrelevant passage does not. The practical consequence is that $k$ has an interior optimum that has to be measured against a corpus’s own distribution of near-misses, and that raising $k$ to improve recall can silently move a system further from that optimum rather than closer to it.

6. Hallucination despite grounding

Retrieval is supposed to be the cure for fabrication: give the model something real to argue from, and it will argue from it. Niu and colleagues built RAGTruth specifically to check that assumption, assembling nearly eighteen thousand responses generated by a range of large language models operating inside standard retrieval-augmented pipelines and annotating them for hallucination at the level of individual words across multiple task types ^[8]. The corpus exists because the assumption is false often enough to need a dedicated benchmark: a model given correctly retrieved, relevant source material will still, on a measurable fraction of responses, assert something the source does not say, contradict the source outright, or attach a citation to a real retrieved document that does not actually support the claim beside it. That last pattern is the most dangerous of the three, because the citation is not fabricated in the sense of pointing nowhere — the document is real and was really retrieved — it simply does not say what the generated sentence claims it says, which is exactly the failure a reader skimming for the presence of a citation marker will not catch.

7. Multi-hop reasoning failure

Some queries have no single supporting span, by construction. The answer requires combining a fact from one document with a fact from another, sometimes conditionally, and no amount of top-$k$ tuning against a single-hop retriever fixes an architecture that was never asked to do the second half of that job. Tang and Yang built MultiHop-RAG specifically to test this, pairing a knowledge base with multi-hop queries whose answers require retrieving and reasoning across multiple supporting documents rather than one, and found that existing retrieval-augmented pipelines perform unsatisfactorily on it — both the embedding models used for retrieval and the language models used for reasoning over what was retrieved struggled with the multi-hop case specifically, across every system-of-the-day tested including GPT-4-class models ^[9]. The failure has two independent causes stacked on top of each other: single-hop retrieval optimises for finding one relevant document per query, so it is not even trying to assemble the right set for a compositional question, and even when the full set is somehow present in context, chaining facts across documents that were written independently, in different styles, at different times, is a harder reasoning task than the single-document case the rest of this article implicitly assumes.

8. Reranker miscalibration

A reranking stage is usually treated as an unqualified improvement: cheap first-stage retrieval followed by an expensive, more accurate scoring pass. The score that stage produces is used two ways — to sort, and, in almost every production system, to threshold, dropping anything below some cutoff before it reaches the generator. The second use requires the score to mean something as a probability, not merely to order correctly, and Penha and Hauff showed that neural ranking models routinely fail exactly that requirement: their empirical study of BERT-based rankers found scores that were not robustly calibrated, meaning the model’s confidence did not reliably track the true likelihood that a given passage was actually relevant, even though the same scores produced perfectly sensible orderings ^[10]. A miscalibrated cutoff is invisible from a ranking-quality metric like nDCG, because ranking quality only asks whether relevant items come before irrelevant ones, never whether the number attached to either one means what the threshold logic assumes it means. The practical failure this produces is a cutoff that drops genuinely relevant passages on some queries and admits irrelevant ones on others, with no way to tell which is happening from the score alone.

9. Embedding-model and index version drift

An embedding model defines a geometry: which passages sit near which queries is a property of that specific model, not of the text. Shen and colleagues formalised the consequence of upgrading one, in the general case of any learned representation used for similarity search: a newly trained embedding model ordinarily produces a space that is not directly comparable to the old one, so deploying it means recomputing embeddings for every item already in the index — a process they call backfilling — unless the new model was specifically trained to remain compatible with the old one’s geometry ^[11]. Their result was developed for visual search and face recognition, and it generalises without alteration to a text-embedding index: nothing about the argument depends on the modality, only on the fact that a nearest-neighbour index assumes every vector in it was produced by a single consistent function. A production RAG system that upgrades its embedding model incrementally — encoding new documents with the new model while leaving the existing index untouched to avoid the cost of a full rebuild — ends up with an index containing two incomparable geometries in one similarity search, and every query silently favours whichever half of the index happens to sit closer to it in whichever space the query itself was embedded in.

10. Duplicate and near-duplicate redundant retrieval

Real corpora are full of near-duplicates: a press release syndicated across a dozen outlets, a policy document copied into three different wikis, boilerplate repeated at the top of every ticket in a queue. A ranking function that scores by similarity to the query has no reason to penalise duplication, so a query that matches one instance of a widely republished passage tends to match all of them, and a naive top-$k$ selection can fill a majority of its slots with restatements of a single fact rather than covering the breadth of what is actually in the corpus. If $d$ of the $k$ retrieved slots are occupied by near-duplicates of an already-retrieved passage, the effective evidence budget is not $k$ but

and $d$ grows with corpus redundancy in a way that is invisible to any retrieval metric computed against a deduplicated relevance-judgement set, which is how most benchmarks are built. Schelpe’s recent empirical analysis, released as a preprint and not yet peer reviewed, measured exact-duplication rates across three regimes and found them to vary enormously by domain — under one percent in academic retrieval corpora, roughly a quarter of retrieved content in an enterprise document setting, and above four-fifths of retrieved content in a conversational-agent setting — while finding, across evaluation with several large language model providers, that removing exact duplicates before generation produced no measurable quality regression ^[12]. Read as a preliminary but suggestive result rather than a settled finding, it argues that deduplication is close to a free win precisely because the redundant slots were never doing any work.

Ordering the ten by expected loss

A catalogue is not a triage plan. The ten differ enormously in how much damage accrues before anyone notices, and instrumentation effort should follow that rather than follow which failure is most interesting to write about. For a mode $i$ with arrival rate $\lambda_i$, mean time to detection $\tau_i$, and cost per undetected instance $\ell_i$, the loss accrued before intervention is approximately

and the marginal value of building a detector is whatever it removes from $\tau_i$. Written this way, the ranking departs sharply from intuition, because $\tau_i$ spans several orders of magnitude across the ten while $\lambda_i$ and $\ell_i$ vary far less.

Indirect prompt injection sits at the top by this measure despite firing comparatively rarely, because its damage ceiling is set by whatever credentials and tools the surrounding agent holds rather than by anything about retrieval quality, and detection at the model layer is close to impossible — the input is one undifferentiated sequence, and separating instruction from data is a trained disposition, not an enforced boundary ^[5]. Close behind sits embedding-model version drift, which fires rarely, at the moment of a model upgrade, but then persists silently for as long as nobody re-measures retrieval quality against a fixed evaluation set, degrading every downstream decision made on the strength of similarity scores from a corrupted index. Index staleness ranks similarly high on persistence: it does not announce itself, it accumulates continuously between rebuilds, and its cost compounds specifically in exactly the domains — policy, pricing, clinical guidance — where a stale answer is most expensive.

In the middle sit retrieved-but-ignored evidence, chunk-boundary loss, and multi-hop failure: frequent, individually moderate in cost, and — this is the useful part — cheap to detect once someone decides to, because each has a retrieval-only test that does not require running the generator at all. Hallucination despite grounding and reranker miscalibration sit just below, expensive per instance but usually bounded by the fact that a downstream reader or reviewer catches at least some fraction of them, which is not true of the failures that operate purely inside infrastructure nobody is reading.

At the bottom in expected-loss terms, though not in ease of prevention, sit over-retrieval dilution and duplicate retrieval. Both are frequent and both are individually cheap to detect and to fix — the first by measuring accuracy as a function of $k$ against a corpus’s own near-miss rate, the second by exact or near-duplicate filtering before the context is assembled — which is exactly why they belong at the bottom of a list ordered by unresolved risk rather than by frequency.

Where the detection actually lives

None of the ten is detected by reading the answer. Each requires an instrument built specifically for it, sitting outside the path the text takes on its way to a user.

For staleness, log the build timestamp of the index shard that served each request and alert when the gap between that timestamp and the present exceeds the rebuild interval by more than a small multiple — a shard silently missing several rebuild cycles is a stronger signal than any accuracy metric will produce.

For chunk-boundary and multi-hop failure, build a retrieval-only evaluation set: queries paired with the identifiers of the spans that together would answer them, scored with recall at $k$ against ground truth, with no generator involved. It is the only artefact that survives a change to the generator, the reranker, or the embedding model without needing to be rebuilt.

For the position effect, inject a canary fact at a known middle position in a sampled fraction of production requests and assert mechanically that it was used in the response, rather than inferring usage from the response reading as though it might have been.

For injection, detection at the model layer is not the answer; containment is. Scope credentials narrowly, gate any consequential action behind deterministic policy rather than model judgement, and treat every retrieved span as adversarial input by default, exactly as OWASP’s guidance recommends ^[6].

For over-retrieval and duplication, sweep $k$ against a held-out accuracy curve per corpus rather than adopting a value from a tutorial, and run exact or embedding-distance deduplication on the retrieved set before it reaches the prompt, logging how many slots were removed so that redundancy becomes a visible, trending number rather than an invisible tax.

For hallucination despite grounding, require that every claim’s cited span, read in isolation, entail the claim rather than merely mention the same entities — not “is consistent with,” but entails — and treat claims that fail the test as unsupported rather than emitting them with a citation marker attached regardless.

For reranker miscalibration and version drift, hold a private, task-representative evaluation set outside every prompt and index the reranker or embedding model touches, and re-run it on a schedule rather than on the vendor’s announcement schedule; a score distribution that has shifted without a corresponding change in your own corpus is the signal, and it is invisible to any metric that only checks ordering.

Predictions, with the observations that would falsify them

These are forecasts, separated from the sourced analysis above. Horizon: 11 August 2028.

One. Retrieval-only evaluation sets, scored independently of any generator, will become a standard deliverable of production RAG projects rather than a research artefact reserved for benchmark papers. Disconfirmed if mainstream RAG tooling in 2028 still ships end-to-end answer accuracy as its only first-class evaluation output.

Two. Index build timestamps and embedding-model identifiers will become standard fields logged on every retrieval request, the way resolved model identifiers became standard for generation requests. Disconfirmed if leading RAG frameworks in 2028 still treat the index as an opaque dependency with no per-request provenance.

Three. Deduplication before context assembly will move from a bespoke pipeline step into framework defaults, because the finding that it costs measurable context budget for no measurable benefit is easy to reproduce and cheap to act on once measured. Disconfirmed if major RAG frameworks in 2028 still assemble top-$k$ context with no redundancy check.

Four. Provenance-aware trust labelling of retrieved spans, carried into the prompt and used to gate what actions a span’s content can trigger, will spread from security-conscious deployments into general practice, driven by incidents rather than by research consensus. Disconfirmed if the major agent frameworks in 2028 still concatenate retrieved content into prompts with no per-span trust channel.

None of these requires a capability breakthrough. Each follows from structure already visible: a store that must be kept synchronised with a source that keeps changing, a geometry that breaks when the function that built it changes, a channel that treats all input as equally authoritative unless told otherwise, and a scoring function whose number is trusted for more than it was ever shown to mean.

What to take away

Retrieval-augmented generation does not fail as a single mechanism, and treating “RAG accuracy” as one number to move obscures which of ten mostly independent things actually broke. A system can have a superb generator and an unsynchronised index. It can have perfect retrieval and a position effect burying the right passage in the middle of the context window. It can have a well-tuned reranker whose scores have quietly stopped meaning what the threshold logic assumes.

The operational discipline that follows is the same across all ten: build the instrument before the incident, put it outside the model’s own report of its success, and measure the specific mechanism rather than the aggregate symptom. A team that can name which of these ten broke, from a log rather than from a guess, has an evidence pipeline under observation. A team that can only say the answer was wrong has a ranking function nobody has measured, feeding a generator nobody has constrained, from an index nobody is watching roll forward.