≈

The passage around this formula

None of the four sections above describes a system anyone ships in isolation. A contemporary AI accelerator composes several of these answers on top of each other, for a reason the earlier article in this series already established: capacity and bandwidth are separate constraints with separate symptoms, and a real workload — serving a large language model’s key-value cache is the canonical case — hits both at once. The KV cache for a single sequence grows as $M_{\mathrm{kv}} \;\approx\; 2 \, L \, H \, d_h \, b \, T$. with L layers, H key/value heads, $d_h$ head dimension, b bytes per stored element and T tokens of context — a quantity that scales linearly with context length and batch size, and has to be both stored somewhere and…

Read this part in the article →