The Physical Economics of Inference: Bandwidth, Energy, and the Limits of a Datacenter

Three constraints, in order

Discussion of model economics usually starts with floating-point operations, because that is the number vendors publish and the number scaling laws are written in. For serving, it is close to the least relevant quantity. Three physical constraints bind before arithmetic does, and they bind in a fixed order.

Bytes moved decides how many tokens a device can emit per second. Joules per byte decides what those tokens cost. Delivered megawatts and rejected heat decide how much of the whole apparatus can exist in one building. Each is a harder constraint than the one before it, and each is less responsive to engineering effort.

Why bytes, not operations

Generating one token from a dense model requires reading every parameter it uses plus the request’s accumulated key–value cache, and performing roughly two operations per parameter read. The arithmetic intensity — operations per byte — is therefore near unity, one to two orders of magnitude below the ratio at which current accelerators saturate their arithmetic units. Decoding sits far to the memory-bound side of the roofline and stays there.

Gholami and colleagues quantified how this gap arrived and why it is widening: over roughly two decades, peak server hardware FLOPS scaled at about 3.0× every two years while DRAM bandwidth scaled at about 1.6× and interconnect bandwidth at about 1.4× over the same interval. Their analysis of encoder and decoder transformers shows memory bandwidth becoming the dominant bottleneck specifically for decoder inference, and they argue this warrants rethinking model architecture and deployment rather than waiting for hardware ^[1].

The practical shape of the constraint is a ceiling on tokens per second per device that barely responds to a faster arithmetic unit. Pope and colleagues formalised the partitioning problem this creates, showing how latency, throughput, and cost trade against each other under different sharding strategies once the bottleneck is data movement rather than computation ^[2].

Prefill is the exception that proves the rule. Processing a long prompt handles many positions simultaneously, raising arithmetic intensity into the compute-bound regime. This is why input and output tokens have genuinely different cost structures and why a workload’s shape — long prompt, short answer, or the reverse — changes its economics more than the choice of model often does.

The four levers that move bytes

Every serving optimisation of consequence attacks the same denominator.

Batching amortises the weight read across concurrent requests. Weights are read once per step regardless of batch size, so throughput rises nearly linearly until memory for the per-request caches runs out.

Cache layout attacks that limit directly. Grouped-query attention shrinks the cache by sharing key and value heads across groups of query heads, with quality recoverable by uptraining from existing multi-head checkpoints ^[5]. PagedAttention removes the fragmentation and worst-case over-reservation of contiguous cache allocation, allowing substantially higher batch occupancy in the same memory ^[3].

Precision halves the bytes per parameter. Dettmers and colleagues showed this is not simple rounding: beyond roughly 6.7 billion parameters, transformers develop systematic large-magnitude outlier features across all layers that destroy naive quantisation, and their method isolates those in higher precision while quantising the remainder, halving inference memory while retaining full-precision performance on a 175-billion-parameter checkpoint ^[6].

Speculation raises tokens produced per weight read. A small draft model proposes several tokens which the target verifies in one parallel pass, and the construction provably leaves the target’s output distribution unchanged ^[4].

These are recovery techniques, not escapes. Each recovers a fixed fraction of what the bandwidth constraint takes, and they compose sub-additively — halving the cache and then halving precision does not quarter cost, because other terms become binding. The floor set by having to read the weights at all is not removed by any of them.

Joules, and what the published accounting establishes

Energy per token follows bytes, because moving data off-chip dominates the energy budget of the operation by a wide margin. This gives the useful heuristic that anything reducing bytes moved reduces joules roughly proportionally, and anything reducing arithmetic alone does not.

The published accounting is much stronger on training than on serving, and the distinction matters. Patterson and colleagues computed energy use and carbon footprint for several large models — T5, Meena, GShard, Switch Transformer, GPT-3 — and refined earlier estimates for a neural architecture search. Their headline finding for architecture choice is that large but sparsely activated networks can consume under one tenth the energy of large dense networks without sacrificing accuracy ^[7]. That is a training-side result about a training-side choice, and sparsity’s serving-side profile is different: routing reduces activated parameters but every expert must remain resident, so the memory constraint is unrelieved ^[9].

Serving energy has no comparable public accounting, for a straightforward reason: it depends on a request mix, a batching regime, a hardware generation, and a utilisation factor that providers do not publish and that change continuously. Any figure quoted for the energy of a single query should be read as an estimate under undisclosed assumptions, not a measurement.

Two structural facts about serving energy are nonetheless firm. It is proportional to tokens generated, and reasoning-trained models generate far more tokens per task by design, since extended deliberation is precisely what verifiable-reward training installs ^[10]. And it is paid per request forever, whereas training energy is paid once — so for any widely used model the lifetime serving term exceeds the training term, often by a large multiple.

Megawatts, and the constraint nobody engineers around

At the level of the building, the constraint stops being about efficiency and becomes about supply. A datacenter is limited by the electrical capacity it can be delivered and by the heat it can reject, and both are set by infrastructure with lead times measured in years.

The broader trend here is frequently misstated, and there is a careful measurement worth holding onto. Masanet and colleagues, integrating data across sources, challenged the widely repeated claim that datacenter energy use had doubled over the preceding decade and would triple or quadruple in the next, finding instead much more modest growth in global datacenter energy use because efficiency gains had substantially offset rising demand ^[8]. That analysis predates the current inference build-out and should not be read as a forecast of it; its enduring contribution is methodological, showing that demand growth and energy growth are decoupled by efficiency and must be measured rather than extrapolated.

What is specific to inference is the shape of the load. Serving is not a batch job that can be scheduled into cheap hours or migrated to where power is available. It is latency-sensitive, geographically constrained by where users are, and close to constant. A training run can chase cheap electricity across the calendar and the map; a serving fleet cannot. This makes inference the harder half of the energy problem operationally even where it is the smaller half in aggregate joules.

Which optimisations move the floor

Sorting the available techniques by whether they change the physics or merely move cost between parties is the most useful thing this framing offers.

Moves the floor. Reducing bytes per token — cache layout, precision, speculation — genuinely reduces the energy and time required for the same output. So does reducing tokens per task, which is why the reasoning-effort control on current models is an energy lever as much as a quality lever.

Moves cost between parties. Batching improves the provider’s utilisation at some cost to individual latency. Caching prompts shifts cost from compute to storage. Routing a request to a cheaper model tier trades the provider’s margin against the user’s quality. All are legitimate and none reduces the joules required to produce a given token from a given model.

Does not help. Faster arithmetic units, absent proportional bandwidth improvement, do not raise decoding throughput. This is the clearest practical consequence of the scaling mismatch, and it is why accelerator generations are increasingly marketed on memory bandwidth and capacity rather than on peak FLOPS.

Price movement in the market reflects this mix rather than any single cause. OpenAI’s own change log records substantial mid-generation reductions — Terra at 20% lower and Luna at 80% lower than at introduction, in late July 2026 ^[11]. From outside, that is consistent with serving improvements, with model substitution, with margin compression, or with all three, and the disclosure does not distinguish them.

Working the arithmetic

The framing above becomes concrete once the quantities are written down, and the exercise is worth doing because it shows which term dominates and by how much.

Take a dense model with $N$ parameters served at $p$ bytes per parameter. The weight traffic per decoding step is $Np$, incurred once per step regardless of how many requests share it. The per-request cache traffic is

read in full every step for every active request. Total bytes moved per step across a batch of $b$ requests at sequence length $s$ is therefore approximately

and the achievable step rate is $B$ divided into the device’s usable bandwidth. Two things follow immediately.

First, the weight term is fixed and the cache term is not, so there is a batch size below which you are paying full weight traffic for very little output — the reason low-traffic deployments of large models are so unfavourable per token. Second, because the cache term scales with both $b$ and $s$, long contexts and high concurrency compete directly for the same budget. You may have one or the other and not both, which is a physical statement rather than a pricing policy.

The energy version of the same expression is more useful still. If $e$ is the average energy per byte moved off-chip, energy per token is roughly $e \cdot B/b$ — the weight traffic divided across the batch, plus each request’s own cache traffic. Batching therefore improves energy per token steeply at first and then asymptotically, converging on the irreducible per-request cache cost. That asymptote is the floor referred to throughout this article, and only cache-reduction techniques lower it.

Why the constraint is unusually stable

Most engineering constraints soften over time. This one has hardened, and it is worth being explicit about why, because it determines how much of the current cost structure is likely to persist.

Arithmetic throughput improves through parallelism, which is comparatively easy to add: more units, more dies, more packages. Memory bandwidth improves through interface width, signalling rate, and physical proximity, all of which run into packaging, power, and thermal limits that do not yield to the same tactics. The result is the divergence Gholami and colleagues document — roughly 3.0× per two years against 1.6× and 1.4× — and there is no indication in the trend of the two lines reconverging ^[1].

Two consequences follow for anyone reasoning about the next several years. Architectural changes that reduce data movement per useful token will keep outperforming architectural changes that reduce arithmetic, which is why attention-layout work has been unusually productive relative to its apparent modesty. And model families will keep differentiating on memory profile rather than on parameter count, because memory profile is what the constraint actually prices.

There is a corollary that is easy to miss. Because the constraint is on movement rather than computation, techniques that spend more arithmetic to move fewer bytes are favourable, which is exactly the trade speculative decoding makes: a draft model’s tokens cost extra computation and save a weight read ^[4]. Expect more of that shape — deliberately wasteful arithmetic in service of avoided memory traffic — and expect it to look inefficient by any measure that counts operations.

What this means for anyone sizing a system

Estimate in bytes, not operations. Your cost is set by the product of tokens generated and bytes read per token, and both are things you can measure directly rather than infer.

Measure your context length distribution before anything else, because it governs achievable concurrency and therefore unit cost more strongly than model choice. A workload with a heavy long-context tail is expensive in a way no vendor comparison will reveal.

Treat token budget as the first-class economic control it now is. Reasoning effort is the one lever that changes joules, latency, and quality simultaneously, and it is under your control rather than the provider’s.

And separate the two energy questions when they come up, because they are routinely conflated. Training energy is a one-time capital cost that has been carefully accounted for in the literature. Serving energy is an operating cost that has not been, that grows with adoption, and that is paid every time anyone presses a key.