Equation 8 · AI Memory Systems and the Bandwidth Wall in Practice: An Advanced Technical Guide
What does this equation mean?
Read the formula alongside the article passage below. Each part has a deeper page with its role in the equation, the supporting passage and nearby citations.
This mathematical expression combines the displayed quantities; its precise role follows from the surrounding article text. Read the equation part by part below; each part has a contextual explanation and a link to its mathematical background.
Read it piece by piece
Symbol M_KV
V is a part of this expression. Its role is fixed by the surrounding article and by the operations shown in the formula.
subscript
The lower label selects a particular version, component, or indexed member of the quantity. For example, x₀ and xₜ can be values at different positions.
How to interpret it
Read this expression with the definitions, units, and assumptions supplied by the article.
What the article says around this equation
Paging attacks the allocator, not the product. Kwon and colleagues observed that existing serving systems store each request’s KV cache in one contiguous span of memory sized for its maximum possible length, and that this produces internal fragmentation, external fragmentation, and duplicated memory across parallel sampling requests — the paper reports that existing systems waste 60 to 80 percent of KV cache memory this way. Their PagedAttention design instead manages the cache in fixed-size blocks addressed indirectly, the same idea an operating system uses for virtual memory, and reports near-zero waste in KV cache memory alongside a 2 to 4 times improvement in serving throughput at the…
Read the full surrounding passage
Paging attacks the allocator, not the product. Kwon and colleagues observed that existing serving systems store each request’s KV cache in one contiguous span of memory sized for its maximum possible length, and that this produces internal fragmentation, external fragmentation, and duplicated memory across parallel sampling requests — the paper reports that existing systems waste 60 to 80 percent of KV cache memory this way. Their PagedAttention design instead manages the cache in fixed-size blocks addressed indirectly, the same idea an operating system uses for virtual memory, and reports near-zero waste in KV cache memory alongside a 2 to 4 times improvement in serving throughput at the same level of latency compared with prior systems, with the improvement growing for longer sequences, larger models and more complex decoding algorithms [ 4 ] . Paging does not reduce per token; it reduces the waste around it, which in practice is often the larger number.
Sources cited in the surrounding passage
These citations give research context. Read each source to check which claims it supports.
Return to AI Memory Systems and the Bandwidth Wall in Practice: An Advanced Technical Guide