=

The passage around this formula

A transformer decoder generates autoregressively: each new token attends over all previous positions [ 1 ] . To avoid recomputing the whole prefix at every step, implementations retain the per-layer key and value tensors for every past position. That is the key–value cache, and its size grows linearly with sequence length: $M_{\mathrm{kv}} = 2\, L \, n_{\mathrm{kv}} \, d_{\mathrm{head}} \, s \, b \, p$. for L layers, $n_{\mathrm{kv}}$ key–value heads, head dimension $d_{\mathrm{head}}$ , sequence length s , batch size b , and p bytes per element. The factor of two counts keys and values.

Read this part in the article →