Symbol t_step
tep is a part of this expression. Its role is fixed by the surrounding article and by the operations shown in the formula.
Read this term in its guide →Published equation contexts
Why this formula appears here
Bandwidth-limited throughput is the regime where achieved FLOPS is irrelevant because I is the binding term. Autoregressive decoding in a served language model is the clearest case: generating a single token requires streaming the model weights and the accumulated key-value cache out of memory, and performs only a small number of operations per byte read. The time per decoding step obeys . a floor set entirely by memory traffic, in which peak arithmetic does not appear. This is why batching improves throughput so dramatically — the same weight bytes are amortised across many sequences, raising I — and why it does not improve single-stream latency at all.
Read the representative guide
Symbol B_weights
eights occurs above the fraction bar. The numerator is divided by the entire denominator below it.
Read this term in its guide →Symbol B_kv
v occurs above the fraction bar. The numerator is divided by the entire denominator below it.
Read this term in its guide →Symbol β
β occurs below the fraction bar. The quantity above the bar is divided by this expression; zero is excluded as a denominator.
Read this term in its guide →Numerator: B_weights + B_kv
The complete quantity above the fraction bar.
Read this term in its guide →How to interpret it
With a fixed numerator, increasing a nonzero denominator reduces the fraction.
Research cited beside this formula
Published contexts (1)
A symbol can carry a different meaning in another article. Each occurrence keeps its own guide and term definitions.
Equation 24 · AI Hardware & Semiconductors
What an AI Accelerator Actually Is: Silicon, Packaging, and the Memory It Can Reach
This equation states a bound: one expression must stay on the indicated side of the other under the article’s assumptions.
Bandwidth-limited throughput is the regime where achieved FLOPS is irrelevant because I is the binding term. Autoregressive decoding in a served language model is the clearest case: generating a single token requires streaming the model weights and the accumulated key-value cache out of memory, and performs only a small number of operations per byte read. The time per decoding step obeys . a floor set entirely by memory traffic, in which peak arithmetic does not appear. This is why batching improves throughput so dramatically — the same weight bytes are amortised across many sequences, raising I — and why it does not improve single-stream latency at all.
Equation guide → · Article →