How AI Inference Serving Actually Works

Two machines wearing one name

A companion piece in this series took apart what a hosted language model charges for a token, treating the token as a billing unit and asking what a buyer actually pays for. This piece goes a level further down, into the box that produces the number on the invoice. Its claim is narrower and mechanical: continuous batching, key-value caching, speculative decoding, quantized precision and mixture-of-experts routing all exist to manage one structural fact about how a transformer answers a request.

That fact is that a request has two phases with different physics, and the split is not incidental to the architecture — it follows directly from what autoregressive generation requires. Prefill consumes the entire prompt in one pass. Every position in the prompt is known in advance, so the accelerator computes attention and feed-forward outputs for all of them at once, as one large, dense matrix multiplication. Decode then produces the answer one token at a time, and each step depends on the one before it: the model cannot compute position $t+1$ until it has sampled position $t$, so there is no way to parallelize across output positions the way prefill parallelizes across input positions. The later positions simply do not exist yet.

This single asymmetry is why a GPU inference deployment does not look like one uniform pipeline. It looks like two loosely coupled systems sharing a device: a burst of dense computation that wants a big batch and finishes quickly, followed by a long, thin, serial process that reads an enormous amount of state out of memory to produce one number, over and over. Reiner Pope and colleagues, in one of the more careful empirical accounts of what this looks like on real hardware, reported 76 percent model FLOPS utilization during large-batch processing of input tokens on a 540-billion-parameter model, against a per-token latency of 29 milliseconds during low-batch generation with int8 weight quantization ^[1]. Those two numbers are not two settings of one dial; they are two different regimes, measured on the same weights.

Everything below is best read as five engineering answers to the question that asymmetry raises: given that one phase is compute-bound and wants to be big, and the other is memory-bound and wants to be many, how do you run both, for many users at once, on hardware that was not built to tell them apart? Continuous batching answers it at the scheduler level. Paged key-value caching answers it at the memory-allocator level. Speculative decoding answers it by borrowing prefill’s parallelism to speed up decode’s serial process. Quantization answers it by shrinking what has to move through memory in the first place. Mixture-of-experts routing answers it by changing which parameters that memory traffic even touches.

Why the two phases obey different rooflines

The standard way to make this precise is a piece of hardware-performance reasoning called the roofline model, and stating it once explains why every mitigation below targets one phase and not the other. Every accelerator has two ceilings: a peak arithmetic rate, in floating-point operations per second, and a peak memory-bandwidth rate, in bytes per second. Which one binds a given piece of work depends on that work’s arithmetic intensity — the ratio of arithmetic performed to bytes moved:

A computation is compute-bound when $I$ exceeds the hardware’s own ratio of FLOPs to bandwidth, and memory-bound when it falls below it. Prefill, processing every position of a prompt against the same weight matrices in one pass, reuses each loaded weight across many tokens at once, so its arithmetic intensity is high and it sits on the compute-bound side of the roofline — which is what the 76 percent utilization figure above looks like in practice ^[1]. Decode loads the same weights, plus the accumulated key-value cache, to produce a single new token; each byte fetched from memory is used for one position’s worth of arithmetic and then it is gone. Its arithmetic intensity is low by construction, not by implementation quality, so decode sits on the memory-bound side of the roofline no matter how efficient the kernel is. That is the structural reason the 29-millisecond decode figure cannot simply be optimized away the way a compute-bound kernel can: no amount of faster arithmetic helps a step that is waiting on bytes, not FLOPs ^[1].

The one lever available on the memory-bound side is the batch. If $b$ independent sequences decode together, the weights are read from memory once per step and reused across all $b$ sequences’ worth of arithmetic, so the batched step’s effective arithmetic intensity rises roughly in proportion to $b$ — up to the point where something else runs out first. That single observation is the entire mechanical argument for batching decode aggressively, and the reason the batch cannot simply be made arbitrarily large, which is where the next two sections begin.

Continuous batching: filling the serial phase without waiting for it to end

The naive way to batch generation is at request granularity: collect a group of prompts, decode all of them together until every sequence has finished, then start the next group. This wastes exactly the resource the previous section identified as scarce. Sequences finish at different lengths, so as soon as the shortest one in a group is done, its slot sits idle — fully allocated, contributing nothing — until the longest sequence finally finishes and the whole batch can be refilled.

Gyeong-In Yu and colleagues, building the Orca serving system, replaced request-level batching with iteration-level scheduling: the scheduler operates at the granularity of a single forward step, evicting finished sequences and admitting new ones after every iteration instead of waiting for a whole batch to clear ^[2]. Because a transformer batches cleanly only when every sequence in it has compatible shapes, and generation naturally produces sequences of changing length, they paired this with selective batching: batch the operations that tolerate mixed-length inputs, such as the dense feed-forward layers, while handling attention separately per sequence, since each sequence’s attention depends on its own key-value history ^[2]. Measured on a GPT-3-scale 175-billion-parameter model, Orca reported a 36.9-times throughput improvement over NVIDIA’s FasterTransformer at matched latency ^[2] — a figure that large is itself evidence of how much of decode static, request-level batching leaves idle.

Continuous batching creates a new problem, though, the moment a long prompt’s prefill has to be admitted into an already-running batch of decode steps. Prefill is a compute-heavy, bursty operation; folding one into an iteration otherwise made of small, memory-bound decode steps means that iteration now does a large dense matmul instead of the quick pass every in-flight sequence was expecting, and every one of those sequences’ token-to-token latency stalls while it waits. Two different fixes have emerged, and they are worth distinguishing because they solve the same interference problem in opposite ways.

Chunked prefill keeps prefill and decode sharing the same hardware but stops them from sharing an uninterrupted iteration. Amey Agrawal and colleagues, in Sarathi-Serve, split a long prefill into near-equal chunks and interleave those chunks with ongoing decode steps rather than absorbing the whole prefill in one shot — “stall-free scheduling” that admits new requests without pausing decodes already in flight ^[3]. Across model sizes and hardware configurations they reported between 2.6 times higher serving capacity, on a single-GPU Mistral-7B deployment, and 5.6 times higher end-to-end serving capacity on a pipeline-parallel Falcon-180B deployment, compared with an unchunked baseline ^[3].

Disaggregation takes the opposite approach: stop sharing hardware for the two phases at all. Yinmin Zhong and colleagues, in DistServe, place prefill and decode on physically separate GPU pools, each independently parallelized for its own bottleneck, and transfer the resulting key-value cache across the interconnect once prefill finishes so decode can pick up where it left off ^[4]. They frame the tension plainly: a colocated system compromises between optimizing time-to-first-token, which prefill controls, and time-per-output-token, which decode controls, and the compromise degrades both. Disaggregating removed it well enough to serve 7.4 times more requests, or meet service-level objectives 12.6 times tighter, than the colocated systems compared against, while keeping over 90 percent of requests within their latency target ^[4].

Neither fix is free. Chunking adds scheduling overhead and can stretch a single prefill’s own completion time across more iterations. Disaggregation adds a cache transfer over the network and needs enough concurrent traffic to keep both pools busy independently. What the two approaches share is the diagnosis: prefill and decode differ enough, physically, that treating every iteration as uniform work is the thing that has to change — whether that change happens in time, on shared hardware, or in space, across separate hardware.

The key-value cache: the memory decode cannot avoid holding

Attention needs, at every generation step, the key and value vectors of every previous token, for every layer and every head. Recomputing those from scratch each step would make generation cost grow quadratically with sequence length, so servers instead cache them once and append one new pair per step. This cache is exactly why decode’s arithmetic intensity is so low: most of a decode step is streaming this accumulated state out of memory, not computing on it.

Its footprint follows directly from the shape of the cache. For a model with $L$ layers, $h_{kv}$ key-value attention heads, head dimension $d_h$, holding a sequence of length $s$ across $b$ concurrently served sequences, stored at $p$ bytes per element, the total resident size is

with the factor of two accounting for storing both keys and values. Two things follow immediately from this equation, and both matter more than the equation’s arithmetic itself. It scales linearly with context length, so a conversation twice as long holds twice the cache. And it scales linearly with the batch $b$ — the exact quantity the previous section identified as the only lever available to make a memory-bound decode step efficient. The key-value cache therefore competes directly, in the same pool of device memory, with the batch size continuous batching is trying to grow. It is not a side cost of serving; it is the resource that sets the ceiling on the technique the previous section depends on.

Early serving systems allocated this memory the simple way: reserve a contiguous buffer sized for a sequence’s maximum possible length as soon as it starts. Woosuk Kwon and colleagues showed how wasteful that is in practice — most sequences finish well short of any maximum, so much of every reservation sits allocated and empty, and buffers of different sizes fragment the remaining space so that even nominally free memory often cannot be assigned to a new sequence ^[5]. Their system, vLLM, borrows the idea behind operating-system virtual memory paging: store the cache in fixed-size, non-contiguous blocks, track which blocks belong to which sequence with a per-sequence block table, and allocate new blocks on demand rather than upfront. The reported result is near-zero waste in key-value cache memory and a 2-to-4-times throughput improvement over FasterTransformer and Orca at the same latency, with the gain growing larger for longer sequences, bigger models and more complex decoding algorithms ^[5].

That last detail matters: paged allocation is not a fixed win but a growing one, because the gap between reserved and actually used memory grows with everything that makes serving harder. It directly converts wasted memory into usable batch capacity, which makes it arguably the single highest-leverage change in the modern serving stack — every other technique here ultimately competes for the memory paging frees up. It has one more consequence worth naming without re-deriving it: because blocks are tracked rather than owned outright by one sequence, identical blocks can be shared between sequences with a common prefix instead of duplicated. That sharing is the memory-side mechanism behind the cached-prompt discounts a pricing analysis would measure from the billing side.

Speculative decoding: borrowing prefill’s parallelism to shorten decode’s chain

Because decode is memory-bound and strictly serial — one full pass over the weights per output token — the way to speed it up without more memory bandwidth is to get more than one verified token out of each pass over those weights. Speculative decoding does exactly that. A small, cheap “draft” model proposes several candidate tokens by running its own inexpensive autoregressive decode ahead of the target model. The large “target” model then checks all of the proposed tokens in a single forward pass, because verifying whether a fixed sequence of already-chosen tokens is consistent with the target’s own distribution is a prefill-shaped operation — dense, and parallel across positions — rather than a decode-shaped one ^[6].

Yaniv Leviathan, Matan Kalman and Yossi Matias, who introduced the method, proved a property that makes it more than a heuristic approximation: the accepted tokens are sampled from exactly the target model’s own distribution, not from some blend of the draft and target distributions. Accepted proposals are kept; the first rejected token is resampled from a corrected distribution that accounts for what the draft got wrong, and everything the draft proposed after that point is discarded and regenerated. The output is therefore statistically identical to standard autoregressive sampling from the target model alone ^[6]. On a T5-XXL model, they reported a 2-to-3-times acceleration over a standard T5X implementation ^[6].

The size of the win has a clean shape. Model the draft’s acceptance probability as $\alpha$ per token, roughly constant and independent across the $\gamma$ tokens proposed in a round — an idealization real traffic does not fully satisfy, but a useful one for seeing the ceiling. The expected number of tokens accepted per verification round is then

This rises with both $\alpha$ and $\gamma$, but with steeply diminishing returns in $\gamma$ for any $\alpha$ below one: drafting fifty tokens ahead does not buy anywhere near fifty accepted tokens, because the marginal proposal deep into a long draft is unlikely to be exactly what the target would have generated. The bottleneck decode started with reappears one level down — the draft model is itself a small, serial, memory-bound process, so speculative decoding is not a free lunch; it trades some of decode’s own bandwidth cost for a second model running continuously, plus wasted target-model computation on every rejected proposal. Its win is largest where the roofline argument above says spare compute is most available: at low batch sizes, where the target model is memory-bound and its arithmetic units sit mostly idle during an ordinary decode step anyway. That idle arithmetic verifies the extra proposed tokens at close to no additional cost.

Quantization: shrinking what has to move

Since decode is bound by bytes moved rather than arithmetic performed, the most direct lever available is to make each byte carry more of the model — fewer bits per weight, per activation, and per cached key-value pair — so the same memory bandwidth streams more of the network through the accelerator each step, and the same device memory holds a larger key-value cache and a larger batch.

Naive uniform 8-bit integer quantization works adequately on smaller models and degrades badly at scale. Tim Dettmers and colleagues traced the failure to systematic outlier features: at sufficient model scale, a small number of feature dimensions consistently take on far larger magnitudes than the rest and dominate attention and prediction quality. Uniform int8 quantization compresses that entire dynamic range into eight bits, which crushes the outliers’ precision precisely in the large models where the memory savings matter most. Their fix, LLM.int8(), separates the two populations: outlier dimensions, well under one-tenth of one percent of values, are computed in 16-bit precision, while more than 99.9 percent of values are multiplied in 8-bit, and the two results are recombined. The reported result is memory needed for inference cut roughly in half with no measured performance degradation up to 175-billion-parameter scale, enough to run models such as OPT-175B or BLOOM on a single server built from consumer-grade GPUs rather than a specialized cluster ^[7].

Floating-point 8-bit formats sidestep the outlier problem differently, by building a wider dynamic range into the format itself rather than patching around a fixed-range integer scheme. Paulius Micikevicius and colleagues, working across NVIDIA, Arm and Intel, proposed two 8-bit floating-point encodings: E4M3, with four exponent bits and three mantissa bits, trading away representable infinities for extra dynamic range and aimed at weights and activations; and E5M2, with five exponent bits and two mantissa bits following ordinary IEEE-754 conventions, better suited to the larger range gradients need in training. They reported FP8 training reaching accuracy parity with 16-bit training across convolutional, recurrent and transformer architectures up to 175 billion parameters, and FP8 post-training quantization succeeding on language models that had resisted fixed-point int8 quantization outright ^[8].

None of this pays off without matching hardware. NVIDIA describes the Hopper generation’s Transformer Engine as pairing fourth-generation Tensor Cores, rated at four times the FP8 throughput of the prior generation’s 16-bit rate, with dynamic per-tensor scaling computed from each tensor’s own statistics so a fixed 8-bit format can track values that would otherwise overflow or underflow it ^[9]. That is a vendor’s own account and should be read as one: NVIDIA states the Transformer Engine delivers “up to 30x faster AI inference” relative to the prior generation, a headline that bundles several hardware and software generations rather than isolating the precision change alone ^[9]. What can be stated more narrowly, because it follows from the format rather than a vendor benchmark, is the mechanism: FP8 halves the bytes needed per parameter relative to 16-bit precision, which roughly doubles the arithmetic intensity of a memory-bound decode step, and roughly doubles the batch a fixed amount of device memory can hold once weights and cache both sit in the lower precision.

The throughput implication is therefore directional, not uniform, across the two phases. Prefill, already compute-bound, gains from the higher raw FLOPs rate lower-precision circuitry provides. Decode, memory-bound, gains chiefly from the smaller footprint moving through the bandwidth ceiling, since it was rarely limited by arithmetic to begin with. A stack that quantizes only the weights captures the first gain and part of the second; quantizing the key-value cache too is what extends the win to the resource decode actually depends on.

Mixture-of-experts routing: paying compute for a fraction of the parameters

A dense transformer touches every parameter for every token, so compute per token scales directly with total parameter count. A sparsely activated mixture-of-experts model instead replaces some dense feed-forward blocks with a bank of “expert” feed-forward networks and a small router that selects, per token, only a handful to actually run, so parameter count and per-token compute stop being the same number. William Fedus, Barret Zoph and Noam Shazeer formalized the simplest version in the Switch Transformer, routing each token to a single top-1 expert and describing the result as a model with an “outrageous” number of parameters but constant computational cost per token; against dense baselines at comparable compute, they reported up to seven times faster pretraining at smaller scale and a four-times speedup over a dense T5-XXL baseline at trillion-parameter scale ^[10].

For serving, this decoupling cuts two ways at once. On the arithmetic side it is good news: per-token compute stays small regardless of total parameter count, keeping prefill’s compute-bound cost in check even as capability scales. On the memory side it is closer to a wash, or worse: every expert’s weights still have to sit somewhere in the fleet’s device memory, whether or not a given token routes to it, because which experts a token needs is not known until the router decides. The parameter count that sets memory footprint does not shrink the way the count that sets arithmetic does.

That footprint has to be distributed across accelerators before serving begins, and the distribution problem — expert parallelism — folds back into the batching argument from earlier. Continuous batching earns its efficiency by routing many sequences’ work through the same loaded weights at once; an MoE layer only gets that benefit for a given expert if enough tokens in the batch route to it. A batch too small, or too imbalanced across experts, leaves some experts starved and others queued — a batching problem wearing a routing costume. DeepSeek-V3’s technical report discloses a concrete answer at scale: each token routes to 8 of 256 routed experts plus one always-active shared expert, activating 37 billion of the model’s 671 billion total parameters, and the deployment splits prefill and decode onto separate GPU pools running different degrees of expert parallelism — prefill uses 32-way expert parallelism across a minimum of 4 nodes (32 GPUs), sized so each expert sees a large enough batch of routed tokens to run efficiently, while decode uses 320-way expert parallelism across 40 nodes (320 GPUs), with each GPU hosting essentially one primary expert, since decode’s smaller per-step batches worsen the tokens-per-expert problem and need far more parallelism just to keep every expert fed ^[11]. The same report describes duplicating experts that online traffic statistics show are disproportionately popular, deploying 32 redundant experts during prefill as extra copies of the busiest experts on additional GPUs — a disclosed patch for the fact that routing decisions are made by the trained model rather than the operator, and traffic across 256 experts never arrives evenly ^[11].

The net effect on the serving cost profile, relative to a dense model of comparable capability, is not simply cheaper or more expensive — it is differently shaped. Compute cost per token can fall sharply, since only a fraction of parameters do arithmetic on any given request. Memory cost does not fall in proportion, since the full parameter set has to be resident somewhere. The resulting cluster compounds the disaggregation problem from earlier: a separate parallelism strategy per phase, and a separate, load-dependent placement decision for which accelerator holds which expert.

Where the mechanism is heading

These are forecasts, separated from the sourced description above. Horizon: 16 August 2028. Assumption: accelerator memory bandwidth, not arithmetic throughput, remains the binding constraint on decode-side serving cost.

One. Disaggregated prefill/decode deployment, not colocated continuous batching alone, becomes the default for large-scale hosted serving, because the interference cost that chunking manages is structural and disaggregation removes it rather than scheduling around it. Disconfirmed if the dominant serving stacks in 2028 still colocate the two phases by default.

Two. Key-value cache precision falls below weight precision, rather than tracking it, as caches keep growing with context length and concurrency while a quantized weight footprint stays fixed. Disconfirmed if published serving configurations in 2028 still store the cache at the same or higher precision than the weights as standard practice.

Three. Expert-parallelism degree keeps diverging further between prefill and decode deployments as MoE models grow, extending the logic behind DeepSeek-V3’s 32-way versus 320-way split, because the tokens-per-expert problem worsens as expert count grows while decode batches stay comparatively small. Disconfirmed if future large MoE deployments converge on one parallelism degree for both phases.

Four. Speculative decoding is tuned primarily by measured acceptance rate under production traffic rather than by draft length, because the geometric ceiling in expected accepted tokens per round makes acceptance rate the operative lever. Disconfirmed if providers continue to report gains chiefly as a function of draft length rather than measured acceptance rate.

What to take away

Decode’s memory-bandwidth bound is the single fact that explains almost every serving technique in this article. Continuous batching fills it with more concurrent work, at the granularity of one iteration rather than one request ^[2]. Chunked prefill and disaggregation keep the two phases from interfering while they share, or split, the hardware ^{[3, 4]}. Paged key-value cache allocation reclaims the wasted memory that determines how large that batch can get ^[5]. Speculative decoding borrows the idle arithmetic the bound leaves lying around at small batch sizes ^[6]. Quantization shrinks what has to move through the bound in the first place, provided the cache is quantized along with the weights ^{[7, 8, 9]}. Mixture-of-experts routing changes which parameters that bound even applies to, at the cost of a harder placement problem ^{[10, 11]}.

None of this is a pricing decision. Each is engineering with a measurable, cited effect on throughput or latency — but each shows up, one remove downstream, as a line on an invoice measured elsewhere in this series. Understanding the mechanism will not tell you what a provider charges. It tells you what a price change can, and cannot, be evidence of.