≈

The passage around this formula

What binds is memory, and specifically the cache. Weights are shared across the batch; the key–value cache is not. Each concurrent request carries its own, and each grows with every token it generates. Achievable batch size is therefore $b_{\max} \approx \frac{M_{\mathrm{device}} - M_{\mathrm{weights}}}{M_{\mathrm{kv}}(s)}$. which falls as contexts lengthen. This is the mechanism behind an effect users notice without explanation: long-context workloads cost disproportionately more, because they crowd out the concurrency that made short-context serving cheap.

Read this part in the article →