$$M_{\mathrm{kv}} = 2\, L \, n_{\mathrm{kv}} \, d_{\mathrm{head}} \, s \, b \, p$$

Equation 1 · Foundation Models

Serving a Frontier Model: The KV Cache, Batching, and What a Token Actually Costs

This equation states an equality: the expressions on both sides have the same value under the article’s assumptions.

A transformer decoder generates autoregressively: each new token attends over all previous positions [ 1 ] . To avoid recomputing the whole prefix at every step, implementations retain the per-layer key and value tensors for every past position. That is the key–value cache, and its size grows linearly with sequence length: $M_{\mathrm{kv}} = 2\, L \, n_{\mathrm{kv}} \, d_{\mathrm{head}} \, s \, b \, p$. for L layers, $n_{\mathrm{kv}}$ key–value heads, head dimension $d_{\mathrm{head}}$ , sequence length s , batch size b , and p bytes per element. The factor of two counts keys and values.

Meanings in this article

• $L$: the number of layers.
• $d_{\mathrm{head}}$: the head dimension.
• $s$: the sequence length.
• $b$: the batch size.

Equation guide → · Article →