=

The passage around this formula

Hooper and colleagues, working on KV cache compression for very long contexts, state the resulting footprint precisely: for a model with n layers and h attention heads of dimension d , stored using e bytes per element, the KV cache size for batch size b and sequence length l is $M_{\mathrm{KV}} \;=\; 2 \, n \, h \, d \, e \, b \, l$. which grows linearly in both batch size and sequence length, with the leading factor of two accounting for storing both keys and values [ 9 ] . That single equation is the whole mechanism: nothing about it is a design choice an inference engineer can simply decline. Extend the conversation, and l grows; serve more requests at once, and b grows; either way $M_{\mathrm{KV}}$ grows with it, and Hooper…

Read this part in the article →