Symbol l

The passage around this formula

…for very long contexts, state the resulting footprint precisely: for a model with n layers and h attention heads of dimension d , stored using e bytes per element, the KV cache size for batch size b and sequence length l is $M_{\mathrm{KV}} \;=\; 2 \, n \, h \, d \, e \, b \, l$. which grows linearly in both batch size and sequence length, with the leading factor of two accounting for storing both keys and values [ 9 ] . That single equation is the whole mechanism: nothing about it is a design choice an inference engineer can simply decline. Extend…

Read this part in the article →