Symbol d_h

The passage around this formula

…and a real workload — serving a large language model’s key-value cache is the canonical case — hits both at once. The KV cache for a single sequence grows as $M_{\mathrm{kv}} \;\approx\; 2 \, L \, H \, d_h \, b \, T$. with L layers, H key/value heads, $d_h$ head dimension, b bytes per stored element and T tokens of context — a quantity that scales linearly with context length and batch size, and has to be both stored somewhere and streamed through the arithmetic units on every generated token. Pope and colleagues showed that partitioning…

Read this part in the article →