What does this equation mean?

Autoregressive decoding sits at the far left of that graph. Generating one token requires reading essentially every weight and the accumulated key-value cache once, and performing roughly two arithmetic operations per parameter. At one byte per weight the intensity is about two operations per byte; at four bits per weight, about four. Pope and colleagues formalised this partitioning problem for large transformers and showed how latency, throughput and cost trade off under different sharding strategies, with generation and prefill behaving as different regimes [ 3 ] . The bound that matters on a device follows immediately: if W bytes of weights and cache must cross the memory bus for each token and the sustained bandwidth is B , then the time per token cannot be less than $T_{\mathrm{tok}} \ge \frac{W}{B}$. no matter how fast the arithmetic units are. As an arithmetic illustration with assumed values rather than a measurement of any product: a four-billion-parameter model at four bits per weight is roughly two gigabytes of weights, so a device sustaining fifty gigabytes per second cannot exceed about twenty-five tokens per second, and a device sustaining twenty-five gigabytes per second cannot exceed about twelve. Nothing in the model, the framework or the prompt changes that ceiling. Only fewer bytes or more bandwidth does.