Symbol d_h

The passage around this formula

…in-flight generation — its key-value cache — grows linearly in exactly the variables that make serving expensive: sequence length and batch size. For a model with L layers, $H_{\mathrm{kv}}$ key-value heads, head dimension $d_h$ , sequence length S , batch size B , and p bytes stored per element, the cache occupies approximately $M_{\mathrm{KV}} = 2 \, L \, H_{\mathrm{kv}} \, d_h \, S \, B \, p$. bytes, the leading factor of two accounting for keys and values together. That equation is worth writing out because it exposes the one real lever every mitigation below…

Read this part in the article →