Equation 5 · How AI Inference Serving Actually Works
What does this equation mean?
Read the formula alongside the article passage below. Each part has a deeper page with its role in the equation, the supporting passage and nearby citations.
the number of independent sequences. Read the equation part by part below; each part has a contextual explanation and a link to its mathematical background.
Read it piece by piece
How to interpret it
Read this expression with the definitions, units, and assumptions supplied by the article.
What the article says around this equation
The one lever available on the memory-bound side is the batch. If b independent sequences decode together, the weights are read from memory once per step and reused across all b sequences’ worth of arithmetic, so the batched step’s effective arithmetic intensity rises roughly in proportion to b — up to the point where something else runs out first. That single observation is the entire mechanical argument for batching decode aggressively, and the reason the batch cannot simply be made arbitrarily large, which is where the next two sections begin.
Sources cited in the article section
These citations give research context. Read each source to check which claims it supports.