Five meters, one answer
Ask what a retrieval-augmented answer costs, and most engineering teams will quote the price of the model call: so many input tokens, so many output tokens, one line on one invoice. That line is real, but it is not the bill. Underneath a working RAG system sit at least four more meters that a plain language-model call never turns on at all: the compute spent embedding documents into vectors, the storage that holds those vectors and grows every time the corpus does, the cost of the approximate-nearest-neighbour search that finds them again, and the cost of a second model — the reranker — that re-scores what the search already returned. A fifth meter, freshness, decides how expensive the first four stay over time.
None of these five are exotic. Every one of them is priced on a public rate card, by a vendor who wants a customer to be able to do the arithmetic. The purpose of this article is to actually do it: to take the published prices for embedding, vector storage, approximate-nearest-neighbour query, and reranking, verified directly from the pages that state them, and build the standing infrastructure bill a RAG pipeline adds on top of the model call it was built to improve. It closes by pricing the one alternative that is genuinely comparable — skipping retrieval and simply pasting more into the context window — using the same vendors’ own long-context pricing tiers, because that comparison is the one procurement question this whole apparatus exists to answer.
Two ground rules before the numbers. First, every figure below is either quoted directly from a vendor’s pricing documentation, a formula a vendor documents for its own billing, or a peer-reviewed or preprint research finding, cited as such — vendor prices are facts about what a named vendor currently charges, not claims about underlying cost, and are reported that way throughout. Second, several of the worked examples that follow are illustrative arithmetic on a hypothetical workload built from those real rates, not measurements of any deployed system, and are labelled as such at the point they appear.
Paying to turn documents into vectors
Before an index exists, every document in it has to pass through an embedding model once. This is a full transformer forward pass over the text, priced per token like a language-model call, and billed by a growing list of vendors who host it as its own product.
As verified on 11 August 2026, OpenAI prices text-embedding-3-small at 0.02 dollars per million tokens, text-embedding-3-large at 0.13 dollars per million tokens, and the older ada-002 model at 0.10 dollars per million tokens [1]. Cohere’s Embed 4 is priced at 0.12 dollars per million text tokens and 0.47 dollars per million image tokens, and produces a 1,536-dimension vector by default over an input window of up to 128,000 tokens [2, 3]. Pinecone, a vector database rather than a model vendor, also sells hosted embedding directly inside its own billing: llama-text-embed-v2 at 0.16 dollars per million tokens and multilingual-e5-large at 0.08 dollars per million tokens on its Standard plan [5]. Three different kinds of company, three overlapping rate cards, for what is architecturally the same operation.
At these rates, ingestion looks cheap, and at small scale it is. A modest internal knowledge base of a hundred million tokens — on the order of seventy-five thousand pages of ordinary prose — costs about twelve dollars to embed once with Cohere’s Embed 4, or thirteen dollars with OpenAI’s text-embedding-3-large. That is the number most teams see first, and it is the number that makes RAG look like a rounding error next to the cost of the model itself.
It stops looking that way at the scale the rest of this article is built around: a corpus of one billion chunks, roughly the scale the approximate-nearest-neighbour literature itself uses as its standard billion-point benchmark [8]. At five hundred tokens per chunk, that corpus is five hundred billion tokens, and embedding it once with Embed 4 costs about sixty thousand dollars — before a single query has been served, and before the index has been built at all. Ingestion is the one meter in this article that is not a recurring monthly charge; it behaves more like a capital cost, paid in full or in large increments each time a large new source is brought in, and it reads as small only because most organisations experience it one modest corpus at a time.
The mechanism underneath ingestion cost is the same one that governs any transformer forward pass on current accelerators: decoding, and embedding is architecturally the same category of operation as one step of decoding, is bound by how many bytes of weights and activations must move off the accelerator, not by how much arithmetic is available, and that memory-bandwidth ceiling has grown far more slowly than raw compute over the last two decades [12]. What follows from that is not a headline number but a design implication: batching many embedding calls together amortises the fixed cost of reading the model’s weights across far more output, exactly as it does for language-model prefill, so a pipeline that embeds documents in large batches spends less energy per token than one that embeds them one at a time as they arrive. No vendor publishes joules per embedded token, so this is a mechanism rather than a measurement — but it is the same mechanism the freshness section returns to below, and it is worth holding onto now.
The index is a bill that renews itself
Ingestion is paid once per document. Storage is paid every month, for as long as the vector stays in the index, whether or not anyone ever queries it again.
Pinecone’s documentation gives the exact formula it bills against: index size equals the number of records multiplied by the sum of the identifier size, the metadata size, and the embedding dimension multiplied by four bytes per float, and its own worked example — five hundred thousand records at 768 dimensions with 500-byte metadata — comes to about 1.79 gigabytes [5]. Storage on the Standard serverless plan is priced at 0.33 dollars per gigabyte per month, with write units at four to 4.50 dollars per million and read units at sixteen to eighteen dollars per million, varying by cloud and region [4].
Apply that formula, and Pinecone’s own worked ratio for identifier and metadata overhead, to the billion-chunk corpus above, using Cohere’s default 1,536-dimension vector: each record comes to roughly 6,650 bytes, so the full index runs to about 6,650 gigabytes, and the monthly storage line alone is approximately 2,195 dollars — every month, indefinitely, regardless of traffic. This is illustrative arithmetic built on real per-unit prices and a stated set of assumptions, not a measurement of a deployed system, and it does not include the replication most production deployments add for availability, which would raise the figure further.
The comparison worth making is against the cost of storing the same corpus as plain text instead of as vectors. Using a common approximation of roughly four bytes of source text per token, the billion-chunk corpus is about two thousand gigabytes of raw text, which costs about forty-six dollars a month in Amazon S3 Standard storage at 0.023 dollars per gigabyte for the first fifty terabytes [14]. The vector index built from that same text costs roughly forty-eight times more to store than the text it was derived from. That multiplier is the honest price of making a corpus searchable by similarity rather than only retrievable by name, and it recurs every month for as long as the index exists — which, unlike the one-time ingestion charge above, makes storage the first genuinely standing cost in the pipeline.
What a nearest-neighbour query actually bills for
The algorithmic literature on approximate nearest-neighbour search is optimistic about query cost, and it has earned the right to be. Hierarchical Navigable Small World graphs, the graph-based structure underneath most production vector indices today, was designed specifically to give logarithmic-complexity search over a metric space without the coarse quantisation earlier methods relied on, and its authors reported that it strongly outperformed the prior open-source vector-only approaches available at the time [6]. ANN-Benchmarks, the standard tool the field uses to compare implementations, found across a wide range of methods and datasets that very different approaches to nearest-neighbour search land on comparable quality-versus-performance trade-off curves, which is exactly what a field with a shared, well-understood algorithmic problem looks like from outside [7]. DiskANN went further still, showing that a billion-point index could be searched from a single workstation with 64 gigabytes of memory and an ordinary solid-state drive, sustaining over five thousand queries a second at under three milliseconds of mean latency and better than 95 percent recall — without holding the whole index in expensive memory at all [8].
None of that algorithmic efficiency is what shows up on an invoice. Pinecone’s own documentation states its query billing rule plainly: read units scale linearly with the size of the namespace being searched, at one read unit for every gigabyte of namespace, with a minimum of a quarter of a read unit per query [5]. This is a billing rule, not a disclosed claim about the underlying compute path — Pinecone does not publish how read-unit consumption maps onto actual memory or disk access inside its serverless engine, and nothing here should be read as a claim that the engine performs work proportional to index size just because the invoice does. What the formula does establish reliably is the shape of the bill.
Apply it to the same billion-chunk, 6,650-gigabyte index used above: a single query against that namespace costs on the order of 6,650 read units, or roughly eleven cents at the midpoint of Pinecone’s sixteen-to-eighteen-dollars-per-million rate. At a hundred thousand queries a month — a moderate production workload, not a large one — read units alone come to approximately 11,300 dollars a month, more than five times the monthly storage bill computed above, and dramatically larger than the sixty-thousand-dollar one-time ingestion cost spread over any reasonable amortisation period. This, again, is illustrative arithmetic on stated assumptions using one vendor’s published formula, not a measurement of a real deployment, and other vector databases price the same underlying work differently; a self-hosted alternative built on pgvector or an open-source HNSW implementation does not eliminate this cost, it moves it into rented compute and disk instead of a per-query meter. The structural point survives the choice of vendor: query cost in a large index is a function of how much has been indexed, not only of how often it is asked, in a way that a plain model call never is.
The reranker keeps its own meter
Most production RAG systems do not stop at approximate-nearest-neighbour retrieval. They over-retrieve — pulling back fifty or a hundred candidates instead of the five or ten that will actually be used — and then run a second, more expensive model over that shortlist to reorder it by relevance before anything reaches the generator. That second model is a reranker, and it is architecturally different from the retriever in a way that explains why it is billed separately rather than folded into the query cost above.
An embedding retriever encodes the query and every candidate independently, once, and compares fixed vectors by inner product — the entire point of building an index in the first place. A cross-encoder reranker does the opposite: it runs the query and each candidate passage through a single transformer together, so that the two texts can attend to each other directly, which the original BERT-based passage-reranking work showed produced a substantial jump in ranking quality over representations computed separately, at the cost of one full model forward pass per query-candidate pair rather than one lookup against a precomputed index [13]. That is real, separate inference work, run on its own hardware, and it appears on the bill as a third meter rather than a refinement of the first.
Cohere prices Rerank 3.5 at two dollars per thousand searches and its newer Rerank 4 Pro at 2.50 dollars per thousand searches, where one search unit covers a query against up to a hundred documents, with any document over five hundred tokens split into additional counted chunks [2]. Pinecone’s own hosted reranking is priced at two dollars per thousand requests, with its smallest model, bge-reranker-v2-m3, free for the first thousand requests each month [5]. At the hundred-thousand-query-a-month workload used above, reranking adds roughly two hundred dollars a month at Cohere’s Rerank 3.5 rate — a small line next to the eleven-thousand-dollar query bill, because unlike approximate-nearest-neighbour query cost, rerank cost tracks query volume rather than index size. That relationship inverts at very high query volumes against a small index, where reranking can become the larger of the two: which meter dominates depends on the shape of the workload, not on a fixed rule, and treating either one as always secondary is the mistake a cost model exists to prevent.
Freshness has two different prices
A RAG index that never changes is easy to cost, because every number above is a snapshot. Almost no production corpus behaves that way; documents are added, edited, and deleted continuously, and the index has to reflect that or the system answers from evidence that is quietly out of date. Freshness is not one cost. It is two, and they are frequently priced as if they were the same thing.
The first is re-embedding: any document that changes has to pass back through the embedding model, at the same per-token rates given above. This cost is proportional to how much of the corpus changed, not to how large the corpus has grown to be, and it is the cheaper of the two freshness costs by construction — a ten-percent daily churn rate on the billion-chunk corpus used throughout this article is still a meaningful embedding bill, but it is bounded by the churn, not by the total.
The second is re-indexing: once a vector changes, the graph or tree structure that makes it findable has to be updated too, and the naive way to do that is to periodically throw the whole structure away and rebuild it from scratch. FreshDiskANN, extending the DiskANN approach specifically to handle a corpus under continuous change, was built because its authors judged periodic full rebuilds “prohibitively expensive” at scale, and reported that its incremental graph-maintenance approach — inserting, deleting, and searching concurrently rather than batching everything into a scheduled rebuild — achieved what the paper describes as a five-to-tenfold reduction in the cost of maintaining freshness compared to full periodic rebuilds, while sustaining thousands of concurrent inserts, deletes, and searches per second [9]. That figure is the paper’s own reported benchmark result rather than an independently audited comparison, and it should be read as a claim from the team that built the system being measured. But the direction it points in does not depend on the exact multiplier: incremental-maintenance cost tracks how much of the index changed, while full-rebuild cost tracks how large the whole index has grown to be, and those two quantities only coincide when the corpus has not grown at all.
The two freshness costs compound with the memory-bandwidth mechanism raised in the ingestion section above. A pipeline that reacts to every single document change in real time, embedding and indexing it immediately, forgoes the batching that amortises the fixed cost of a weight read across many tokens at once; a pipeline that accumulates changes and processes them together on a schedule captures that amortisation but runs on stale evidence between batches. No vendor publishes the joules difference between the two regimes, so this remains an argument from mechanism rather than a cited measurement — but it is the same mechanism, applied to the same kind of transformer forward pass, that the ingestion section already established [12].
Paying the model instead of the pipeline
None of the five meters above are mandatory. The alternative to building a retrieval pipeline is to skip it and put more of the corpus directly into the model’s context window on every call, and both major model vendors now price that choice explicitly rather than leaving it implicit in a flat per-token rate.
As verified on 11 August 2026, OpenAI’s GPT-5.6 family charges materially more for long-context requests than for short ones on every tier: gpt-5.6-sol moves from five dollars input and thirty dollars output per million tokens in its short-context tier to ten dollars input and forty-five dollars output in its long-context tier; gpt-5.6-terra moves from two and twelve dollars to four and eighteen; gpt-5.6-luna moves from 0.20 and 1.20 dollars to 0.40 and 1.80 dollars — an input price that roughly doubles at every tier the moment a request crosses into the long-context band [1]. Google’s pricing shows the same shape on a different threshold: Gemini 3.1 Pro Preview charges two dollars per million input tokens for prompts up to 200,000 tokens and four dollars above that; Gemini 2.5 Pro charges 1.25 dollars below the same threshold and 2.50 dollars above it [11]. Two vendors, two unrelated rate cards, and the same roughly two-times step at the point where context gets long. That convergence should be read cautiously rather than as proof of a shared underlying cost — it may reflect a genuine shared judgment about what long prompts cost to serve, or it may simply be that doubling is an easy round number for a rate card. Nothing published lets an outside observer distinguish the two, and this article does not claim to.
What the convergence does support is a real, worked comparison. Write the retrieval pipeline’s monthly cost, using the terms established above, as a fixed storage component plus a per-query component that itself has two parts:
where
where
Filling in the billion-chunk illustration used throughout: at a hundred thousand queries a month,
The peer-reviewed literature on this exact question agrees with the shape of that result without endorsing either side unconditionally. Li and colleagues compared retrieval against long-context processing directly and found that, when resourced sufficiently, long-context processing consistently outperformed retrieval on average answer quality, while retrieval remained significantly cheaper — and proposed routing each query to whichever approach the model itself judges sufficient, rather than committing to one path for an entire workload [10]. Read together with the arithmetic above, the honest conclusion is narrower than either “always retrieve” or “always inline”: retrieval’s standing infrastructure bill buys a cost advantage that grows with query volume against a large corpus, and that advantage is what the five meters in this article are the price of.
Predictions, with the observations that would falsify them
These are forecasts, separated from the sourced analysis above. Horizon: 11 August 2028.
One. As production vector indices cross into the billion-vector range more routinely, at least one more major vector-database vendor will publish an explicit formula tying query cost to index size, following the pattern already documented here, because flat per-query pricing becomes commercially unworkable once index size varies by orders of magnitude across customers on the same plan. Disconfirmed if the major vector-database vendors converge instead on flat per-query pricing independent of index size.
Two. The gap between short-context and long-context per-token pricing observed at two vendors here will persist or widen rather than close, because serving very long prompts costs disproportionately in the same memory-bandwidth terms that govern serving generally. Disconfirmed if the major model providers move to a single flat per-token input rate regardless of context length by the horizon date.
Three. Incremental, streaming index maintenance in the style FreshDiskANN describes will become the default architecture in mainstream production vector databases, displacing scheduled full reindexing as the common pattern, driven by the cost gap described in this article rather than by a change in query quality. Disconfirmed if the majority of production RAG deployments at the horizon date still rely on periodic full reindexing as their primary freshness mechanism.
Four. For high-volume RAG deployments run against corpora at or beyond the billion-chunk scale, the standing retrieval-infrastructure bill — storage, query, and rerank combined — will exceed the generation bill for the same traffic in a majority of published cost breakdowns, reversing the assumption that retrieval is the cheap half of the system. Disconfirmed if published case studies at that scale continue to show retrieval infrastructure as a minority of total serving spend.
None of these requires a capability discontinuity. They follow from what is already visible: a query cost formula tied to data volume rather than only to traffic, a freshness cost with two structurally different components, and a long-context alternative whose own vendors price it to rise with exactly the quantity retrieval exists to bound.
What to take away
A RAG answer is not priced by one meter. It is priced by five: what it cost to turn the corpus into vectors, what it costs every month to keep those vectors stored, what a single query costs against an index of that size, what a separate reranking pass costs on top of it, and what it costs to keep all four of those current rather than stale. Every one of them is real, every one of them is on a public rate card somewhere, and none of them appears in the number most teams quote when they say what their system costs.
None of this is an argument against building the pipeline. The worked comparison above shows retrieval’s standing bill buying a real and substantial cost advantage over simply inlining more context, at realistic query volumes against a large corpus — which is exactly the trade the architecture was built to make. It is an argument for pricing the pipeline the way its vendors already price it to you: five line items, not one, each growing on its own schedule, and each worth checking against the size of the corpus actually being searched rather than against the size of the corpus the system was designed for on day one.
