Equation 7 · Part 1 · How AI Inference Serving Actually Works
Symbol b
What this part means
the number of independent sequences.
Its job in the formula
b is a part of this expression. Its role is fixed by the surrounding article and by the operations shown in the formula.
Full expression→Symbol b→Article meaning
Where the article explains it
If b independent sequences decode together, the weights are read from memory once per step and reused across all b sequences’ worth of arithmetic, so the batched step’s effective arithmetic intensity rises roughly in proportion to b — up to the point where something else runs out first.
The passage around this formula
The one lever available on the memory-bound side is the batch. If b independent sequences decode together, the weights are read from memory once per step and reused across all b sequences’ worth of arithmetic, so the batched step’s effective arithmetic intensity rises roughly in proportion to b — up to the point where something else runs out first. That single observation is the entire mechanical argument for batching decode aggressively, and the reason the batch cannot simply be made arbitrarily large, which is where the next two sections begin.
Learn the underlying idea
A variable is a named place for a value. Its letter is a local label: x can mean position in one formula and a data point in another.
Open the illustrated variables: a letter stands for a value guide →
See this notation across published equations →
Sources cited in the article section
These citations provide research context; check each source for the exact claim it supports.