superscript

The passage around this formula

Write F(t) for peak arithmetic throughput and B(t) for memory bandwidth. The quantity that matters for a serving system is their ratio, because it sets the arithmetic intensity — operations per byte moved — at which a machine becomes compute-bound rather than bandwidth-bound: $I^{*}(t) = \frac{F(t)}{B(t)} = I^{*}_0 \left(\frac{g_F}{g_B}\right)^{t}$. with $g_F$ and $g_B$ the annual growth factors. When $g_F$ > $g_B$ , $I^{*}$ grows without bound, and the batch size required to keep the arithmetic units busy grows with it. Autoregressive decoding sits on the wrong side of this: generating one token requires streaming the weights and the accumulated key–value cache, so decode time is bounded below by

Read this part in the article →