What does this equation mean?

None of the four sections above describes a system anyone ships in isolation. A contemporary AI accelerator composes several of these answers on top of each other, for a reason the earlier article in this series already established: capacity and bandwidth are separate constraints with separate symptoms, and a real workload — serving a large language model’s key-value cache is the canonical case — hits both at once. The KV cache for a single sequence grows as $M_{\mathrm{kv}} \;\approx\; 2 \, L \, H \, d_h \, b \, T$. with L layers, H key/value heads, $d_h$ head dimension, b bytes per stored element and T tokens of context — a quantity that scales linearly with context length and batch size, and has to be both stored somewhere and streamed through the arithmetic units on every generated token. Pope and colleagues showed that partitioning strategies which shrink the key/value representation, such as multi-query attention, can extend the usable context length by up to 32 times for a fixed memory budget, precisely because they attack the capacity term of that formula without touching the bandwidth term [ 11 ] . Kwon and colleagues showed the complementary move on the software side: treating the KV cache as pageable memory rather than a contiguous reservation removed fragmentation waste and improved serving throughput by 2 to 4 times at matched latency, entirely by using existing capacity more efficiently rather than adding any [ 12 ] .