What does this equation mean?

The standard way to make this precise is a piece of hardware-performance reasoning called the roofline model, and stating it once explains why every mitigation below targets one phase and not the other. Every accelerator has two ceilings: a peak arithmetic rate, in floating-point operations per second, and a peak memory-bandwidth rate, in bytes per second. Which one binds a given piece of work depends on that work’s arithmetic intensity — the ratio of arithmetic performed to bytes moved: $I = \frac{\text{FLOPs performed}}{\text{bytes read and written}}$. A computation is compute-bound when I exceeds the hardware’s own ratio of FLOPs to bandwidth, and memory-bound when it falls below it. Prefill, processing every position of a prompt against the same weight matrices in one pass, reuses each loaded weight across many tokens at once, so its arithmetic intensity is high and it sits on the compute-bound side of the roofline — which is what the 76 percent utilization figure above looks like in practice [ 1 ] . Decode loads the same weights, plus the accumulated key-value cache, to produce a single new token; each byte fetched from memory is used for one position’s worth of arithmetic and then it is gone. Its arithmetic intensity is low by construction, not by implementation quality, so decode sits on the memory-bound side of the roofline no matter how efficient the kernel is. That is the structural reason the 29-millisecond decode figure cannot simply be optimized away the way a compute-bound kernel can: no amount of faster arithmetic helps a step that is waiting on bytes, not FLOPs [ 1 ] .