Symbol t_token

The passage around this formula

with $g_F$ and $g_B$ the annual growth factors. When $g_F$ > $g_B$ , $I^{*}$ grows without bound, and the batch size required to keep the arithmetic units busy grows with it. Autoregressive decoding sits on the wrong side of this: generating one token requires streaming the weights and the accumulated key–value cache, so decode time is bounded below by $t_{\mathrm{token}} \gtrsim \frac{W + K}{B}$. where W is the weight bytes touched and K the cache bytes. Pope and colleagues formalised the partitioning analysis behind this and showed how latency, throughput and cost trade against one another under different sharding layouts [ 12 ] . The mitigations are well documented and all attack the numerator or the batch: grouped-query…

Read this part in the article →