Equation 14 · Part 1 · How AI Inference Serving Actually Works
Symbol M_kv
What this part means
the total resident size.
Its job in the formula
v is part of the quantity the equation computes from the expression on the right.
Full expression→Symbol M_kv→Article meaning
Where the article explains it
For a model with L layers, key-value attention heads, head dimension , holding a sequence of length s across b concurrently served sequences, stored at p bytes per element, the total resident size is
The passage around this formula
Its footprint follows directly from the shape of the cache. For a model with L layers, key-value attention heads, head dimension , holding a sequence of length s across b concurrently served sequences, stored at p bytes per element, the total resident size is . with the factor of two accounting for storing both keys and values. Two things follow immediately from this equation, and both matter more than the equation’s arithmetic itself. It scales linearly with context length, so a conversation twice as long holds twice the cache. And it scales linearly with the batch b — the exact quantity the previous section identified as the only lever available to make a…
Learn the underlying idea
A subscript is a label attached below a symbol. It often selects a time step, component, category, or member of a sequence.
Open the illustrated subscripts: which member of a family? guide →
See this notation across published equations →
Sources cited in the article section
These citations provide research context; check each source for the exact claim it supports.