≥

The passage around this formula

Bandwidth-limited throughput is the regime where achieved FLOPS is irrelevant because $\beta$ $\cdot$ I is the binding term. Autoregressive decoding in a served language model is the clearest case: generating a single token requires streaming the model weights and the accumulated key-value cache out of memory, and performs only a small number of operations per byte read. The time per decoding step obeys $t_{\mathrm{step}} \ge \frac{B_{\mathrm{weights}} + B_{\mathrm{kv}}}{\beta}$. a floor set entirely by memory traffic, in which peak arithmetic does not appear. This is why batching improves throughput so dramatically — the same weight bytes are amortised across many sequences, raising I — and why it does not improve single-stream latency at all.

Read this part in the article →