$$M_{\mathrm{KV}} = 2 \, L \, H_{\mathrm{kv}} \, d_h \, S \, B \, p$$

Equation 13 · Semiconductors

AI Memory Systems and the Bandwidth Wall in 2035: Scenarios, Signals, and Falsifiable Predictions

This equation states an equality: the expressions on both sides have the same value under the article’s assumptions.

Fact. The memory a transformer must hold for one in-flight generation — its key-value cache — grows linearly in exactly the variables that make serving expensive: sequence length and batch size. For a model with L layers, $H_{\mathrm{kv}}$ key-value heads, head dimension $d_h$ , sequence length S , batch size B , and p bytes stored per element, the cache occupies approximately $M_{\mathrm{KV}} = 2 \, L \, H_{\mathrm{kv}} \, d_h \, S \, B \, p$. bytes, the leading factor of two accounting for keys and values together. That equation is worth writing out because it exposes the one real lever every mitigation below actually pulls: none of them escapes linear growth in S and B . Each instead shrinks one of the other factors — most consequentially…

Meanings in this article

• $L$: the number of layers.
• $d_h$: the head dimension.
• $S$: the sequence length.
• $B$: the batch size.
• $p$: the number of bytes.

Equation guide → · Article →