$$M_{\mathrm{kv}} = 2\,L\,h_{kv}\,d_h\,s\,b\,p$$

Equation 14 · Inference Economics

How AI Inference Serving Actually Works

This equation states an equality: the expressions on both sides have the same value under the article’s assumptions.

Its footprint follows directly from the shape of the cache. For a model with L layers, $h_{kv}$ key-value attention heads, head dimension $d_h$ , holding a sequence of length s across b concurrently served sequences, stored at p bytes per element, the total resident size is $M_{\mathrm{kv}} = 2\,L\,h_{kv}\,d_h\,s\,b\,p$. with the factor of two accounting for storing both keys and values. Two things follow immediately from this equation, and both matter more than the equation’s arithmetic itself. It scales linearly with context length, so a conversation twice as long holds twice the cache. And it scales linearly with the batch b — the exact quantity the previous section identified as the only lever available to make a…

Meanings in this article

• $M_{\mathrm{kv}}$: the total resident size.
• $L$: the number of layers.
• $h_{kv}$: the number of key-value attention heads.
• $d_h$: the head dimension.
• $s$: the number of positions in each sequence.
• $b$: the number of concurrently served sequences.
• $p$: the bytes stored per element.

Equation guide → · Article →