The Unit Economics of a Token

Three objects, and the gaps between them

A water utility that charges for consumption owns three different objects, and confuses them at its peril. There is the meter, a mechanical device that counts something correlated with the thing supplied. There is the tariff, an administrative schedule of rates, blocks and standing charges that no physical law obliges to resemble anything. And there is the thing consumed, which is water, and which the utility acquires at a cost bearing only a loose and time-varying relationship to either of the other two. Every billing dispute in the history of municipal supply has been an argument about the gap between one of these objects and another.

Token pricing has exactly this structure, and reading it any other way produces the confusion that dominates most procurement conversations about language models. The token is a meter unit: a vendor-defined division of text produced by a tokenizer the buyer does not control and cannot audit. The tariff is a published rate card with categories — input, cached input, output — that do not map one-to-one onto anything the buyer asked for. The thing consumed is occupancy of a shared accelerator: arithmetic, memory bandwidth, and resident memory held for the duration of a request.

The first gap is unusually well documented. Anthropic’s own pricing page notes that models from Claude 4.7 onward use a newer tokenizer that “produces approximately 30% more tokens for the same text,” with the exact increase depending on content and workload shape ^[4]. The same document, the same work, roughly a third more billable units. That is a vendor disclosure, not an accusation: a better tokenizer can improve model quality and still enlarge the meter. But it settles a methodological question permanently. Rate cards are not comparable across vendors, or even across generations of one vendor, until they are normalised by the meter — which means measuring your own corpus through each tokenizer before comparing anything.

What follows takes the tariff apart line by line, asks what each line implies about the machine underneath, and ends with a method for forecasting spend from a workload’s own token distribution rather than from a headline rate.

Input and output are not the same good

The single most consequential fact about serving a transformer is that a request has two phases with different physics.

Prefill processes the entire prompt in one pass. Every position is available simultaneously, so the work is dense matrix multiplication with high arithmetic intensity, and the accelerator’s compute units can be kept genuinely busy. Decode produces one token at a time, and each step must read the model weights and the accumulated key–value cache out of memory to produce a single position of output. The arithmetic per byte moved is low, so the phase is bound by memory bandwidth rather than by arithmetic.

Pope and colleagues quantified both ends of this in a study of large-model inference partitioning, reporting 76% model FLOPS utilisation during large-batch processing of input tokens and 29 milliseconds per token during low-batch-size generation with int8 weight quantisation ^[11]. Those two numbers describe the same hardware doing two different kinds of work, and they are the physical reason the invoice has two columns.

The price ratios follow. As verified on 8 August 2026, OpenAI lists gpt-5.6-sol at 5 dollars per million input tokens against 30 dollars per million output tokens, a ratio of six to one ^[1]. Anthropic lists Claude Opus 5 at 5 dollars input against 25 dollars output, five to one ^[4]. These are published rates, and a rate is not a cost. But there is one public artefact that goes further than a rate card. Microsoft’s capacity-planning documentation for provisioned deployments states that “output tokens require more processing capacity than input tokens,” that each model has a ratio expressing how many input tokens one output token is equivalent to for capacity purposes, and that for GPT-4.1 and later Azure OpenAI models “this ratio matches the model’s global standard pricing ratio between output and input tokens” ^[9].

That is worth stating precisely, because it is easy to over-read. It does not demonstrate that the price ratio equals the marginal cost ratio. It demonstrates something narrower and still useful: a provider’s own internal capacity accounting — the arithmetic it uses to tell customers how much hardware to reserve — uses the same number as its rate card. The tariff and the engineering model are at least mutually consistent.

The serving literature has spent several years on the fact that these two phases interfere when they share a machine. Sarathi-Serve splits long prefills into near-equal chunks so that new requests can be admitted without pausing ongoing decodes, reporting stall-free schedules and substantially higher serving capacity than the systems it compares against ^[13]. DistServe goes further and places the phases on different accelerators entirely, co-optimising the resource allocation and parallelism strategy for each, and reports serving 7.4 times more requests or meeting 12.6 times tighter latency objectives than the colocated baselines ^[14]. For a buyer, the implication is concrete rather than academic: the two categories on the invoice may correspond to work that does not even run on the same physical machines, which is why their prices can move independently.

What a cache discount discloses

A cached-prefix discount is the most information-dense line on any rate card, because unlike a headline price it has internal structure, and structure constrains the machine that produced it.

Start with the numbers, all verified on 8 August 2026. OpenAI prices cached input for gpt-5.6-sol at 0.50 dollars against 5 dollars uncached, one tenth ^[1]. Anthropic states a cache-read multiplier of 0.1 times the base input price ^[5]. Google prices context caching for Gemini 3.1 Pro Preview at 0.20 dollars per million tokens against a standard input rate of 2 dollars for prompts up to 200,000 tokens — again one tenth ^[7]. Three providers, three architectures, one ratio. That convergence is itself evidence: whatever a cache hit avoids, it avoids about ninety per cent of the input-side cost and not more.

What differs is who declares the reuse. Google documents implicit caching as enabled by default for Gemini 2.5 and newer models, with the provider automatically passing on cost savings when a request hits cache, alongside explicit cache objects that a caller creates and manages ^[8]. Anthropic offers both an automatic mode, where a single top-level marker lets the system manage breakpoints as a conversation grows, and explicit breakpoints placed on individual content blocks ^[4]. That distinction matters more than it looks: under implicit caching the discount arrives without the buyer asking for it, which means the buyer also cannot reliably predict it, and a cost model that assumes a hit rate it did not request is a forecast built on someone else’s scheduling decisions.

Now read the rest of the structure the way an engineer reads a tariff.

A read is cheap but not free. Reuse eliminates recomputation of the attention keys and values for the shared prefix. It does not eliminate the need to hold that state and stream it into the accelerator on every step of decoding. A discount that stops at ninety per cent is what you would expect if recomputation were the dominant avoided cost and memory traffic were the irreducible remainder.

A write costs more than an uncached token, and the premium scales with duration. Anthropic charges 1.25 times the base input price for a five-minute cache write and 2 times for a one-hour write ^[5]. Google prices the same economics differently, charging an explicit storage rate — 4.50 dollars per million tokens per hour for Gemini 3.1 Pro Preview — on top of a discounted cached-token rate ^[7]. These are the same fact under two accounting conventions: holding state in fast memory is a rented resource, and the longer you hold it the more of someone else’s concurrency you have taken.

There is a minimum, and it is block-shaped. Anthropic’s minimum cacheable prompt lengths are 512, 1,024, 2,048 or 4,096 tokens depending on the model, and requests below the threshold are simply processed without caching and without an error ^[5]. OpenAI documents caching for prefixes of at least 1,024 tokens on GPT-5.6 and later, with earlier models ranging from 1,024 to 2,048 ^[2]. Minimums of that shape are what block-granular allocation looks like from the outside: below some quantum, the bookkeeping costs more than the saving.

There is a time-to-live, and it behaves like an eviction policy. OpenAI documents that a cached prefix on GPT-5.6 models remains eligible for reuse for at least thirty minutes, that in-memory retention on earlier models generally lasts five to ten minutes of inactivity up to a maximum of one hour, and that extended retention reaches twenty-four hours ^[2]. “At least, generally, up to” is the vocabulary of a shared pool under contention, not of a guaranteed allocation.

It is a prefix cache, so order is load-bearing. Anything that changes early in the prompt invalidates everything after it. Anthropic documents a case that catches people repeatedly: the resolved effort value is rendered into the prompt, so changing the effort level between requests invalidates cache breakpoints, and the published example shows a cache read of 3,546 tokens on the second request replaced by a fresh 3,546-token cache write on the third after nothing changed but the effort setting ^[6].

The systems literature makes the same shape visible from the other side. PagedAttention removes the fragmentation and over-reservation inherent in contiguous key–value allocation, allowing far higher batch occupancy at equal memory, with reported throughput gains of two to four times over prior systems at comparable latency ^[12]. RadixAttention, in SGLang, organises the cache as a prefix tree so that key–value state is automatically reused across calls, with reported throughput up to 6.4 times that of the systems compared ^[15]. A tariff with block minimums, duration-priced writes, tenth-price reads and an idle-time eviction window is what a paged prefix-tree cache in a shared memory pool looks like when someone has to write it down as prices.

The buyer’s rule that falls out of this is arithmetic, not strategy. Anthropic’s own documentation states the break-even: a cache hit at one tenth the input price means caching pays off after one read for the five-minute duration and after two reads for the one-hour duration ^[4]. If your traffic is bursty and your idle gaps exceed the time-to-live, you will pay the write premium repeatedly and collect the discount never.

The third category: billed and never returned

Reasoning tokens are the line item that breaks cost models built by inspecting outputs, and both major providers document the mechanism plainly.

OpenAI states that “while reasoning tokens are not visible via the API, they still occupy space in the model’s context window and are billed as output tokens,” and exposes the count in the response under output_tokens_details.reasoning_tokens; its worked example shows 1,024 reasoning tokens inside 1,186 total output tokens ^[3]. Anthropic is more explicit still. Its pricing table for thinking shows that billed output tokens are “the full thinking tokens Claude generated internally” while visible output tokens are only the summarised thinking text, and carries a warning in the documentation that “the billed output token count does not match the visible token count in the response. You are billed for the full thinking process, not the thinking content visible in the response” ^[6].

This breaks naive forecasting in three separate ways, and they compound.

It is unobservable from the artefact. You cannot estimate spend by measuring the length of the answers you received, because the largest component of many answers was never in them. It is measurable — both providers expose a usage field for exactly this, with Anthropic’s at usage.output_tokens_details.thinking_tokens ^{[6, 3]} — but only from telemetry, and only if you recorded it. An organisation that logged responses and not usage objects has no way to reconstruct its own cost history.

It is billed in the most expensive category. Reasoning is charged at the output rate, which on the models above is five to six times the input rate ^{[1, 4]}. A workload whose reasoning share drifts upward pays that drift at the highest rate on the card.

It is input-dependent and heavy-tailed. OpenAI notes that reasoning token consumption ranges from hundreds to tens of thousands depending on task difficulty ^[3]. Anthropic’s adaptive thinking makes the decision per request: at lower effort levels the model may skip thinking entirely on simple inputs, while at high effort it almost always thinks ^[6]. The per-request cost distribution is therefore a mixture — a mass near zero plus a long right tail — and its mean is stable only as long as the mix of incoming work is stable. A change in your users’ behaviour, with no change to your prompt, your model or your rate card, moves your bill.

Two further consequences deserve naming. First, the cost control and the quality control are the same knob: max_tokens is a hard cap over thinking and answer combined, so a cap tuned for cost produces truncated responses that are billed in full and usually retried ^[6]. Second, on models that preserve thinking blocks across turns, prior reasoning that remained in context is billed as input on subsequent turns ^[6]. Text you never saw can be paid for twice: once at the output rate when it was generated, and again at the input rate on every turn that carries it forward.

Concurrency, the variable nobody quotes

Short-context serving is cheap for one reason: batching. Because decode is bandwidth-bound and the weights are streamed once per step regardless of how many sequences are in flight, adding sequences to a batch is very nearly free until something stops you. The thing that stops you is memory. Every concurrent sequence holds key–value state proportional to its context length for as long as it is alive, and that state competes for the same device memory as the weights.

This is precisely what PagedAttention addressed, and the reported result carries the economics inside it: the gains from eliminating fragmentation and over-reservation are described as more significant with longer sequences, larger models and more complex decoding ^[12]. Longer sequences are where the memory pressure lives, which is another way of saying that long contexts are expensive because of what they prevent, not because of what they cost directly. A request carrying a 200,000-token context occupies a share of the batching budget two orders of magnitude larger than one carrying 2,000 tokens, and every other tenant of that accelerator pays for the difference in reduced batch size. The long prefill also arrives as a burst of compute that stalls other users’ decode steps unless the scheduler is built to prevent it — the interference that chunked prefills and phase disaggregation were designed to remove ^{[13, 14]}.

Here the published tariffs disagree with each other, and the disagreement is worth characterising rather than adjudicating. Google prices long prompts higher: Gemini 3.1 Pro Preview at 2 dollars per million input tokens for prompts up to 200,000 tokens and 4 dollars above that, with Gemini 2.5 Pro showing the same doubling from 1.25 to 2.50 dollars ^[7]. Anthropic states the opposite policy for Claude 4.6 and later models, offering the full one-million-token context window at standard pricing, with the documentation making the point explicitly: a 900,000-token request is billed at the same per-token rate as a 9,000-token request ^[4].

Both cannot be tracking marginal cost. My reading — analysis, not fact — is that at least one of three things must hold: the two serving architectures have genuinely different long-context cost curves; one provider has chosen to cross-subsidise long contexts to win agentic and document workloads; or the fleets differ enough in load and hardware mix that the marginal cost of a long request is simply not the same number at the two companies. Nothing published lets an outsider choose between these. What matters operationally is that the choice is invisible in a headline rate: a price comparison run at 4,000 tokens does not predict the ranking at 400,000, and a workload’s context distribution can invert a vendor ranking without either rate card changing.

One more disclosure confirms that the scarce resource being priced is memory occupancy rather than text. In Microsoft’s capacity sizing model, cached tokens do not consume provisioned capacity at all, so a higher cache rate directly reduces the number of provisioned units a workload requires ^[9]. The cache is not merely a discount; it is capacity relief, which is exactly what it would be if the constraint were resident state.

Two kinds of price cut

Prices fall, and the reasons are announced in the same channel with the same vocabulary. They should not be treated the same way.

The first kind is a serving improvement: the same computation, produced more efficiently. Better batching and memory management, prefix reuse, chunked prefills, phase disaggregation — each lowers the cost of serving an identical request against identical weights ^{[12, 15, 13, 14]}. A price cut of this kind is safe to bank, because the object you qualified is still the object you are buying.

The second kind is substitution: a different, cheaper thing is served, and it happens to clear the bar you cared about. Epoch AI’s analysis of inference price trends, published on 12 March 2025, found that the price to reach GPT-4’s performance on a set of PhD-level science questions fell by a factor of forty per year, with the rate of decline across the benchmarks studied ranging from ninefold to nine-hundredfold per year; the authors add their own caveat that the fastest declines occurred in the most recent year and may not persist ^[10]. Read carefully, that is price at a fixed capability threshold, not price for a fixed object. Most of a decline of that magnitude is substitution — a smaller or newer model reaching a bar that previously required a frontier model — and substitution requires re-qualification, because the system answering your traffic is not the system you tested.

A third category exists and is neither: scheduled tariff changes with no technical event attached. Anthropic’s pricing page documents introductory pricing for Claude Sonnet 5 of 2 and 10 dollars per million input and output tokens through 31 August 2026, reverting to 3 and 15 dollars on 1 September 2026 ^[4]. That is a fifty per cent increase arriving on a calendar date. And beneath all three sits the meter itself: a per-token price cut can be entirely undone by a tokenizer change that produces roughly thirty per cent more tokens for the same text ^[4]. Any spend forecast expressed in tokens per dollar rather than documents per dollar is exposed to a unit change it cannot see.

Margin, commitment and reservation

The remaining lines on the rate card are commercial instruments. They price properties of the transaction, not quantities of computation, and reading them as engineering facts is a category error.

Batch processing sells latency back to the provider. Anthropic, OpenAI and Google all offer a fifty per cent discount for asynchronous processing ^{[4, 1, 7]}. Identical tokens, identical model, half price, in exchange for the right to schedule your work into troughs. It is the clearest case on any rate card of a price attached to a scheduling property rather than to a quantity.

Latency premiums sell the same property in the other direction. OpenAI’s fast mode, renamed from priority processing on 30 July 2026, carries a two-times multiplier ^[1]. Anthropic’s fast mode for Claude Opus 5 is priced at 10 and 50 dollars against standard rates of 5 and 25 ^[4]. The factor of four between a batch token and a fast token on the same model is entirely a queueing product.

Routing constraints are priced too. Anthropic applies a 1.1 times multiplier for US-only inference across all token categories, and notes a ten per cent premium for regional and multi-region endpoints on partner clouds ^[4]. Nothing about the computation changes; the constraint on where it may happen does.

Capacity reservation converts a variable cost into a fixed one. Microsoft’s provisioned deployments hold a fixed amount of processing capacity exclusively for one customer whether or not requests are being made, billed per provisioned throughput unit per hour regardless of tokens consumed, with the meter starting at deployment creation and stopping at deletion; Azure reservations then apply a discount to that meter in exchange for a one-month or one-year commitment ^[9]. The buyer accepts utilisation risk and receives a lower effective rate plus a latency target.

And one caveat in that documentation reveals what is actually scarce. Holding quota does not guarantee that capacity is available; reservations explicitly do not guarantee capacity; and capacity availability changes throughout the day with demand across regions and models ^[9]. A commitment discount in this market is not primarily a volume rebate. It is a price for demand predictability, offered by a seller whose binding constraint is accelerators.

The procurement rule is simply never to compare across tiers. A batch rate and a fast rate are prices for different products that happen to share a model name.

Forecasting from your own distribution

A headline rate cannot forecast spend, because the rate is one number and a workload is a joint distribution over five counts. The method below is deliberately mechanical.

Instrument each request to record five quantities: new input tokens, cache-write tokens, cache-read tokens, visible output tokens, and reasoning tokens. The cost of a single request is then

where $n$, $w$, $r$, $y$ and $z$ are the five counts in that order and each $p$ is the corresponding published rate. The important feature is the last term: visible output and reasoning share one price, and the reasoning count is the only one of the five you cannot recover by inspecting what came back.

The nonlinearity that surprises people does not live in this equation, which is linear. It lives in agent loops. Suppose a loop runs for $k$ steps, each step appends roughly $a$ tokens of tool output and assistant text, and the context started at $b$ tokens. The total input billed across the whole loop is then

which is quadratic in the number of steps. Doubling the length of an agent trajectory roughly quadruples its input billing. This is why cache hit rate dominates agentic economics: prefix caching converts most of that quadratic term into reads at one tenth the price, and a workload that loses its cache to a time-to-live expiry or an effort change pays the full quadratic ^{[5, 6]}.

From there the procedure is:

One. Measure usage objects, not string lengths. Reasoning tokens and cache reads are invisible in response text and visible in usage fields ^{[3, 6]}.

Two. Segment before averaging. Fit the five counts per request class — simple lookup, document analysis, agent trajectory — and report medians with ninetieth and ninety-ninth percentiles. A single blended average across classes describes no request you actually send.

Three. Price failure explicitly. Truncation at the output cap is billed in full and normally retried, so the number that matters is cost per accepted answer: cost per attempt divided by acceptance rate.

Four. Check your realised cache hit rate against break-even, which is one read at the five-minute write premium and two reads at the one-hour premium ^[4].

Five. Re-price under every tier — standard, batch, latency-premium, provisioned — and for reserved capacity compute the utilisation at which the per-unit-hour charge undercuts what you would have paid per token ^[9].

A worked illustration, using published rates for Claude Opus 5 at 5 dollars input, 25 dollars output and 0.50 dollars for cache reads ^[4]. This is arithmetic on a hypothetical workload, not a measurement of a real one. Take a support-triage request with a 20,000-token cached policy prefix, 800 new input tokens, 300 visible output tokens, and 1,500 reasoning tokens on thirty per cent of requests and none on the rest. The cache read costs one cent; the new input, four tenths of a cent; the visible output, three quarters of a cent; and reasoning, in expectation, about 1.1 cents. The total is roughly 3.3 cents, of which about a third is reasoning that never appears in the response. Serve the same prefix uncached and the prefix alone costs ten cents, taking the request to roughly 12.3 cents — a factor of nearly four, driven entirely by a cache hit rate that no rate card mentions.

Predictions, with disconfirmation conditions

These are forecasts, separated from the sourced analysis above. Horizon: 8 August 2028. Assumptions: accelerator supply remains the binding constraint on serving capacity, and the token remains the billing unit.

One. At least two major providers will bill reasoning tokens as a distinct line rather than folding them into an undifferentiated output rate, because the category is now large enough that buyers demand it be separable. Disconfirmed if reasoning remains billed at the plain output rate across all major providers.

Two. Long-context pricing will diverge further rather than converge: at least one major provider will retain a context-length-dependent input tier while at least one retains a flat rate across the full window. Disconfirmed if the major providers converge on a single flat per-token input rate irrespective of context length.

Three. Cache read discounts will remain close to one tenth of base input while cache write and storage terms proliferate into more durations and more explicit meters. Disconfirmed if read discounts move materially outside the band around one tenth across providers.

Four. Published declines in price per token will overstate declines in price per completed task, because tokens consumed per task rise with reasoning and agent trajectories. Disconfirmed if median tokens per completed task in comparable agentic workloads falls year over year at the same time as per-token prices.

What to take away

The meter, the tariff and the thing consumed are three objects. A token is the meter’s unit and the vendor defines it. The rate card is an administrative schedule with categories that price properties of a transaction — latency tolerance, geography, commitment, memory residency — alongside quantities of computation. The thing consumed is occupancy of a shared machine, and it is the one object nobody publishes.

Four numbers describe a workload’s economics better than any headline rate: the realised cache hit rate, the reasoning share of output tokens, the distribution of context lengths, and the acceptance rate of first attempts. None appears on a rate card, all are recoverable from usage telemetry, and a forecast built on them survives the events that invalidate a forecast built on a price — a tokenizer change, an effort-level change, a longer agent loop, a substitution of what is served. Ask for the rate last. It is the least informative number in the transaction.