$$t_{\mathrm{token}} \gtrsim \frac{W + K}{B}$$

Equation 8 · Foundation Models

Model Systems in 2035: Four Scenarios, Their Signals, and What Would Falsify Them

This mathematical expression combines the displayed quantities; its precise role follows from the surrounding article text.

with $g_F$ and $g_B$ the annual growth factors. When $g_F$ > $g_B$ , $I^{*}$ grows without bound, and the batch size required to keep the arithmetic units busy grows with it. Autoregressive decoding sits on the wrong side of this: generating one token requires streaming the weights and the accumulated key–value cache, so decode time is bounded below by $t_{\mathrm{token}} \gtrsim \frac{W + K}{B}$. where W is the weight bytes touched and K the cache bytes. Pope and colleagues formalised the partitioning analysis behind this and showed how latency, throughput and cost trade against one another under different sharding layouts [ 12 ] . The mitigations are well documented and all attack the numerator or the batch: grouped-query…

Meanings in this article

• $W$: the weight bytes touched and K the cache bytes.
• $K$: the cache bytes.

Equation guide → · Article →