What does this equation mean?

Fact. The memory a transformer must hold for one in-flight generation — its key-value cache — grows linearly in exactly the variables that make serving expensive: sequence length and batch size. For a model with L layers, $H_{\mathrm{kv}}$ key-value heads, head dimension $d_h$ , sequence length S , batch size B , and p bytes stored per element, the cache occupies approximately $M_{\mathrm{KV}} = 2 \, L \, H_{\mathrm{kv}} \, d_h \, S \, B \, p$. bytes, the leading factor of two accounting for keys and values together. That equation is worth writing out because it exposes the one real lever every mitigation below actually pulls: none of them escapes linear growth in S and B . Each instead shrinks one of the other factors — most consequentially $H_{\mathrm{kv}}$ , the number of key-value heads actually stored.