Building Production RAG: An Advanced Technical Guide

Six controls, not one architecture

Ask a team why their retrieval-augmented system underperforms and the answer usually names the wrong layer. It is rarely “the embedding model was wrong.” It is almost always one of six narrower, more mundane decisions: how the corpus was cut into pieces before anything was indexed; how much weight a hybrid ranker gave to keyword match versus semantic similarity; whether a reranking stage sat in the request path at all, and whether its latency budget was ever measured; whether the filter that should have hidden a document from an unauthorized caller actually ran before the candidate list was fixed; whether last night’s edit to that document reached the index; and whether anyone would have noticed if it had not.

None of these are the problem Lewis and colleagues solved when they proposed combining a parametric language model with a non-parametric index queried at inference time, treating the retrieved passage as a latent variable marginalized into the generation step ^[1]. That paper settled the architecture. It says nothing about chunk size, fader weights, latency budgets, tenant isolation, index freshness, or what a production team should watch at three in the morning. Those are the decisions that determine whether the architecture, once built, actually works in front of real users on a real corpus — and they are the subject of this guide.

Treat what follows as six adjustable controls on one bench, each with a documented tradeoff, each traceable to a peer-reviewed paper, a vendor’s own published guidance, or a documented production evaluation rather than to folklore repeated in a conference talk. Where a number comes from a vendor’s own benchmark rather than an independent study, this guide says so explicitly — a company measuring the improvement its own product produces on its own evaluation set is real evidence, but it is a different kind of evidence than a peer-reviewed comparison across systems, and the difference matters when a recommendation is going into someone else’s production system.

Chunking: the lossy compression nobody schedules a review for

Every index-time pipeline commits to a unit of retrieval before a single query has been written against it. That unit — the chunk — determines what can ever be returned, how much surrounding context survives inside it, and what a downstream reranker or generator is even allowed to see. Get it wrong and no amount of tuning further down the pipeline recovers the loss, because the missing information was never written to the index at all.

The received wisdom in most tutorials is a fixed token count with a fixed overlap, commonly around 800 tokens with 400 tokens of overlap. Chroma’s research team built an evaluation harness specifically to test this kind of default, scoring chunking strategies against token-level relevance rather than the whole-document relevance judgments that standard information-retrieval benchmarks use, on the reasoning that whole-document benchmarks cannot see whether a chunk boundary sliced through the exact sentence a query needed. Their headline finding is that chunking strategy choice moved recall by as much as nine percentage points across the configurations they tested, and that the popular 800-token, 400-token-overlap default was, in their words, among the configurations with “particularly poor recall-efficiency tradeoffs” of any they measured — while the same recursive character splitter, reparametrized down to roughly 200 tokens with no overlap, reached 88.1% recall on their benchmark ^[3]. The gap between those two numbers is not a difference in method. It is the same method, tuned differently.

That result argues against a specific, common assumption: that “semantic” or structure-aware chunking is intrinsically superior to naive fixed-size splitting. Chroma’s own evaluation found the opposite in several cases — their semantic chunker, which segments text at points of embedding discontinuity, underperformed a well-parametrized fixed-size splitter at default settings and only became competitive once it was tuned toward a comparable chunk size. Their own conclusion was that “heuristic chunking strategies such as the popular RecursiveCharacterTextSplitter often perform well in practice when parametrized appropriately” ^[3]. Document-structure-aware splitting — chunking at heading boundaries, table edges, or section markers rather than at a raw character count — has a real and different advantage: it keeps a chunk from being torn out of the structural unit that gave it meaning, which token-count splitting cannot promise regardless of how well the token count is tuned. But it is not a shortcut around measurement. A structure-aware splitter still produces chunks of widely varying size, and an untested assumption that “aligned with document structure” implies “good for retrieval” is exactly the kind of unmeasured default this section is arguing against.

The engineering conclusion is specific rather than general: chunking is not a parameter set once from a blog post and left alone. It is a design decision with a measurable effect on recall, the measurement is comparatively cheap to run — Chroma’s own methodology requires only a labeled set of queries and the spans that answer them, not full end-to-end answer grading — and a team that has not run that measurement on its own corpus does not know whether its chunker is costing it nine points of recall or none.

Hybrid search: weighting lexical against dense, and tuning the weight

A dense retriever trained as a dual encoder — separately encoding query and passage into fixed-length vectors and ranking by similarity — was the advance that made single-stage neural retrieval practical at all. Karpukhin and colleagues showed that this architecture, trained on a modest number of question-passage pairs, could be implemented with dense representations alone and substantially outperformed a Lucene BM25 baseline on open-domain question answering ^[2]. That result is now close to a decade of production practice old, and it is still correct as far as it goes. What it does not claim, and what production teams routinely discover the hard way, is that a dense retriever scores by similarity in a learned vector space, and a vector space trained on natural-language semantics has little reason to preserve the discriminative weight of an exact token that carries no semantic content of its own — a stock-keeping unit, a ticket number, an error code, a part identifier. This is an engineering observation about how the representation works, not a finding reported in the DPR paper itself, and it is the reason production stacks almost never ship dense retrieval alone.

The fix is hybrid retrieval: combine a lexical score, which is exact-match strong and semantically blind, with a dense score, which is semantically strong and exact-match weak. The complication is that the two scores are not on comparable scales. Pinecone’s documentation states the problem directly: dense vectors scored by inner product against unit-normalized embeddings fall roughly in the range $[-1, 1]$, while BM25-style sparse scores are unbounded positive values that grow with term frequency, document length, and vocabulary rarity, so that “without explicit weighting, the sparse component dominates the combined score” ^[4]. The documented fix is a convex combination of normalized scores, governed by a single weighting parameter:

where the hats denote scores each rescaled onto a comparable range before combination. At $\alpha = 1$ the ranking is purely semantic; at $\alpha = 0$ it collapses to lexical search; $\alpha = 0.5$ weights the two signals equally.

The number that matters more than the formula is where a team should start, and Pinecone’s own documented guidance is specific rather than a shrug toward “it depends”: $\alpha = 0.75$, dense-leaning, as “a good default for natural-language queries on conversational or document-style content”; $\alpha = 0.5$ as balanced, for workloads where keyword and semantic signals contribute roughly equally; and $\alpha = 0.25$, sparse-leaning, for “queries with high keyword specificity” — the SKU, the ticket ID, the named entity — where lexical exactness should dominate ^[4]. Read those as starting points, not answers: this is a vendor’s own documented default, offered by the vendor as a default and not as a benchmarked optimum, and the same documentation says as much, recommending that teams evaluate multiple $\alpha$ values against a labeled relevance set drawn from their own workload rather than adopt the default unmeasured. A support corpus dominated by product-code lookups and a knowledge-base corpus dominated by conceptual questions have no reason to share an optimal $\alpha$, and a single global weight applied to both query types inside one product is a common, quietly expensive mistake.

Reranking: model selection is really a latency-budget decision

A reranker is a second-stage precision filter: it takes the top candidates a first-stage retriever already narrowed the corpus down to, and re-scores them with a model that can look at the query and each candidate jointly rather than comparing precomputed, independent vectors. That joint scoring is why cross-encoder rerankers are consistently more precise than the retrieval stage feeding them, and why they cannot replace that stage — a model that must look at every candidate jointly cannot be run against an entire corpus per query without the cost becoming prohibitive, so it only ever sees what the first stage already selected.

The clearest published evidence that stacking a reranker onto hybrid retrieval compounds rather than merely adds comes from Anthropic’s own evaluation of contextual retrieval, and it should be read as exactly that: the vendor’s own numbers, on its own evaluation set, not an independently replicated study. Anthropic reports that adding model-generated context to each chunk before embedding reduced their top-20-chunk retrieval failure rate by 35%, from 5.7% to 3.7%; that adding contextual BM25 alongside it took the reduction to 49%, to 2.9%; and that adding a reranking stage on top of both reached a 67% reduction, down to 1.9% ^[5]. The direction of that result — each layer adding a further, smaller reduction on top of the last — is consistent with the wider hybrid-and-rerank literature even though the specific numbers are Anthropic’s own and were not independently reproduced here.

What “reranking” costs in practice is where model selection actually happens, and the constraints are concrete rather than aesthetic. Cohere’s own documentation specifies a hard ceiling on request volume — documents multiplied by the maximum chunks scored per document cannot exceed 10,000, or the request is rejected outright — and specifies that documents longer than the model’s context window are chunked automatically and scored by the maximum relevance across their chunks, with the current generation of models trained at a 32,768-token context and the query itself truncated beyond 16,384 tokens, while the prior generation is limited to a 4,096-token context with queries truncated at 2,048 tokens ^[6]. The same documentation recommends against trusting an absolute relevance-score cutoff and instead calibrating a threshold empirically against 30 to 50 representative queries from the actual workload ^[6] — the same discipline as the chunking and hybrid-weighting sections above: measure on your own traffic rather than import someone else’s number.

The latency consequence follows from where the reranker sits in the request path. A simplified but useful model of end-to-end latency is

where $k$ is the number of candidates handed to the reranker and $\ell_{\mathrm{rerank}}$ is the marginal latency per candidate scored. Recall generally improves as $k$ grows, because a larger candidate set is more likely to contain the passage that actually answers the query — but $k$ multiplies the one latency term a team most directly controls, so it is not a value to leave at a copied default. It is a dial to be set from a measured recall-versus-$k$ curve against an explicit latency budget, the same way the hybrid weight $\alpha$ is set from a measured recall curve rather than assumed.

Metadata filtering and access control at retrieval time

Retrieved evidence has to be both relevant and permitted, and those are different checks running at different points in the pipeline. Metadata filtering restricts candidates by a structured predicate — date range, source system, document type. Access control restricts candidates by who is asking — this caller may see documents belonging to their organization, their department, or their own account, and no others. It is possible to build a system that gets the first right and the second wrong, and that failure mode is common enough that it has its own entry in the current OWASP guidance for generative AI applications: LLM02:2025 names sensitive information disclosure as a distinct risk category, and its mitigation guidance recommends limiting access to sensitive data on a least-privilege basis and limiting what data sources a model or retrieval layer can reach in the first place, rather than relying on the model to decline to repeat what it was already given ^[8].

Vector databases increasingly build tenant isolation into the storage layer itself rather than leaving it to application logic alone. Weaviate’s documentation describes multi-tenancy as structural: each tenant’s data is stored on a physically separate shard, “data stored in one tenant is not visible to another tenant,” and every read, write, and query operation is required to carry an explicit tenant identifier rather than defaulting to an unscoped query across the whole collection ^[7]. That is a genuine, useful guarantee, and it is worth being precise about what it does and does not cover. The isolation is enforced structurally, at the level of which shard a query can physically touch once a tenant identifier is supplied — the documentation does not describe this as an authentication or authorization system in itself, and it depends on the calling application supplying the correct tenant identifier for the caller actually making the request. A retrieval layer that trusts a client-supplied tenant parameter without independently verifying the caller’s identity against it has implemented storage isolation without implementing access control, and the two are easy to conflate because they sit right next to each other in the same query call.

The second, less discussed failure mode is architectural rather than a missing check: whether filtering happens before or after the candidate set is fixed. A system that retrieves its top-$k$ candidates first and then discards the ones the caller is not permitted to see can silently return fewer than $k$ usable results — or, in the worst case, zero — without any error being raised, because nothing in that architecture treats “permitted candidate count fell below what the caller expected” as a condition worth surfacing. Filtering that runs as part of the retrieval call itself, constraining which vectors are even compared rather than pruning the result afterward, avoids the silent shortfall but requires that the index and the access-control model be designed together rather than bolted on afterward. Which architecture a given vector database supports is a real, load-bearing question to ask before committing to it, not a detail to discover during an incident.

Incremental index update patterns

A production corpus is never static. Documents are edited, superseded, and deleted continuously, and re-embedding and rebuilding an index from scratch on every change does not scale past a small corpus. The index structures that made large-scale approximate nearest-neighbor search practical were designed with this in mind from the start rather than as an afterthought: Malkov and Yashunin’s description of Hierarchical Navigable Small World graphs states that the structure “incrementally builds a multi-layer structure consisting from hierarchical set of proximity graphs… for nested subsets of the stored elements,” meaning new points can be inserted into an existing graph rather than requiring the whole structure to be rebuilt from an offline batch process ^[9]. HNSW and its descendants are the reason online insertion into a large vector index is possible at all; they are also the reason many teams assume freshness is solved once the underlying index format supports incremental insertion, which understates how much operational machinery sits between “the data structure supports insertion” and “a write made two minutes ago is reliably searchable.”

Qdrant’s documentation of its own optimizer is a concrete look at that machinery. Newly written points are first placed into an unoptimized segment, where they are “accepted and durably stored” — the write itself is safe immediately — while a background optimizer later merges, vacuums, and indexes segments into the graph structure that makes search fast at scale ^[10]. Whether those newly written, not-yet-indexed points are searchable in the interim is itself a configurable tradeoff: the documentation describes a prevent_unoptimized setting, introduced specifically because allowing search over a large, growing pool of unindexed data degrades query latency under sustained write load, so the vendor’s own answer is to let an operator choose between momentary invisibility of the most recent writes and consistent search latency, rather than pretending both can be had for free ^[10]. Different vector databases will make this tradeoff differently, but the tradeoff itself — freshness against latency, decided somewhere in the write path whether or not anyone chose it deliberately — is close to universal for index structures with this shape.

Deletion is a related and separately under-tested failure mode. Graph-based approximate nearest-neighbor structures do not, in general, support cheap point deletion the way a row-store table does; a naive deletion strategy that immediately excises a node from a proximity graph risks damaging the graph’s connectivity for its neighbors. Production systems typically compensate with a tombstone or soft-delete pattern, filtering deleted identifiers out of results after the graph traversal rather than removing them from the graph in place, with the traversal itself periodically rebuilt to reclaim the space. The consequence worth stating plainly for anyone building an evaluation or an incident runbook: a document a user deleted an hour ago reappearing in a retrieved answer is not evidence of a data-integrity bug in the usual sense. It is very often evidence that the deletion path was never load-tested the way the insertion path was, because insertion is the operation every quick-start tutorial exercises and deletion is the operation almost none of them do.

Monitoring and observability for retrieval quality

An end-to-end answer-quality score — did the final response satisfy the user — is close to the least useful signal a production RAG system can report, not because it is wrong but because it collapses six independently tunable stages into one number and gives no way to tell which of them moved. A quality regression could be a chunking regression, a hybrid-weight regression, a reranker latency timeout silently dropping the reranking stage, an access-control filter over-pruning results, a stale index, or a genuine change in what users are asking. Distinguishing those requires instrumentation at each stage, not a single dashboard measuring the output.

Arize’s Phoenix documentation describes the tracing approach directly: “a trace captures model calls, retrieval, tool use, and custom logic so you can debug behavior and understand where time is spent,” with each stage of a request — the embedding call, the vector database retrieval, context assembly, the final generation — recorded as its own span carrying its own latency and payload data, inspectable in one view rather than reconstructed from separate logs after the fact ^[11]. That kind of per-stage trace is what turns “the answer was wrong” into “the reranker returned nothing above threshold for this query,” which is an actionable finding instead of a vague complaint.

Tracing shows what happened on one request; it does not by itself score whether retrieval was any good, and building that second layer without hand-labeled ground truth for every query is the specific problem Es and colleagues built RAGAS to address. Their framework proposes reference-free metrics that separately score retrieval effectiveness, the faithfulness with which a generated answer actually uses what was retrieved, and the quality of the generation itself, explicitly avoiding a requirement for ground-truth answers that most production corpora will never have fully labeled ^[12]. Combined with tracing, this gives a production system three things worth watching continuously rather than only at launch: retrieval recall against a small labeled query set that is re-run on a fixed schedule rather than measured once before shipping; the rate at which retrieval, filtering, or reranking returns fewer usable candidates than requested, which is the observable symptom of the silent access-control shortfall described earlier; and periodic faithfulness sampling, checking whether generated claims are actually entailed by what was retrieved rather than merely adjacent to it.

None of this is a one-time evaluation a team runs before a launch review and then stops thinking about. A corpus that changes continuously and a query distribution that drifts as users discover new ways to ask questions mean that whatever recall number a team measured at launch degrades on its own, silently, without any code changing at all.

Predictions, with the observations that would falsify them

These are forecasts, separated from the sourced analysis above. Horizon: 11 August 2028.

One. Hybrid weight and reranker candidate count will increasingly be treated as workload-specific values recorded and versioned alongside the index build, rather than global defaults set once at launch and left alone. Disconfirmed if mainstream RAG frameworks in 2028 still expose a single global $\alpha$ and a single global $k$ with no per-workload override as their primary configuration surface.

Two. Index freshness will move from an implementation detail into an explicitly stated service-level objective — a documented maximum delay between a write and that write becoming searchable — the way uptime already is. Disconfirmed if major vector database vendors in 2028 still decline to publish any staleness guarantee alongside their durability and uptime guarantees.

Three. Per-stage retrieval tracing, in roughly the shape Phoenix already ships, will become a default instrumentation layer built into major RAG frameworks rather than an opt-in third-party integration teams have to wire up themselves. Disconfirmed if the dominant RAG frameworks in 2028 still ship no first-class per-span tracing and treat end-to-end answer scoring as their only built-in evaluation surface.

None of these requires a capability breakthrough in any underlying model. They follow from the structure already visible in this guide: six controls with measurable, corpus-specific optima, currently shipped with global defaults that happen to be reasonable starting points and are routinely mistaken for finished answers.

What to take away

A production retrieval-augmented system is not one decision about architecture. It is six decisions, made separately, each with a documented tradeoff and a way to measure whether the tradeoff was struck well on this corpus rather than on whoever’s benchmark the default was copied from: what unit the chunker committed to before any query existed; how much weight the hybrid ranker gives lexical signal against dense signal, and whether that weight was measured or assumed; whether a reranker’s latency fits the budget it was never actually checked against; whether the permission check ran early enough to avoid silently starving results; whether an edit made an hour ago is searchable yet, and how long a team is willing to let that take; and whether anyone would notice a regression in any of the above before a user did.

A system that can answer those six questions with a measured number, not an assumption, has an engineered retrieval pipeline. A system that answers “we used the framework defaults” has six untuned dials on a bench nobody has walked over to check.