Symbol b

The passage around this formula

…— serving a large language model’s key-value cache is the canonical case — hits both at once. The KV cache for a single sequence grows as $M_{\mathrm{kv}} \;\approx\; 2 \, L \, H \, d_h \, b \, T$. with L layers, H key/value heads, $d_h$ head dimension, b bytes per stored element and T tokens of context — a quantity that scales linearly with context length and batch size, and has to be both stored somewhere and streamed through the arithmetic units on every generated token. Pope and colleagues showed that partitioning strategies which shrink the…

Read this part in the article →