What does this equation mean?

Underneath all of the pricing mechanics sits a constraint pricing cannot remove: generating one token from a large language model is, for most of a request, bound by how fast bytes can move through accelerator memory rather than by how much arithmetic the accelerator can perform. Producing each token requires reading the model’s active weights and the accumulated key-value cache for that request from memory once; the arithmetic performed per byte read is low, so decoding is memory-bandwidth-bound while only the initial processing of a long prompt is compute-bound, a distinction Pope and colleagues formalized in their analysis of transformer inference efficiency [ 15 ] . Approximately, $t_{\mathrm{token}} \;\gtrsim\; \frac{2\,N_{\mathrm{active}}\,\beta + \mathrm{KV}(c)}{\mathrm{BW}}$. where N-active is the number of parameters actively read per token, beta the bytes each occupies at the serving precision, KV of c the memory footprint of the key-value cache at context length c, and BW the accelerator’s memory bandwidth. Nothing about pricing changes this floor; it only changes which side of it a given task sits on. Kwon and colleagues showed that KV of c is frequently larger than it needs to be in practice, because conventional cache allocation reserves memory contiguously and wastes much of it to fragmentation, motivating the PagedAttention scheme that manages key-value memory the way an operating system manages virtual memory, in fixed-size pages rather than one contiguous block per request [ 16 ] . That is an engineering fix for waste inside the bound, not a repeal of the bound itself: a longer agentic session, carrying more context turn over turn in exactly the way the earlier accounting model describes, grows the key-value footprint and therefore grows the minimum time every subsequent token in that session can take, independent of price.