What does this equation mean?

This is the reason batching and caching matter for price rather than only for latency: a serving system that keeps a GPU at 80% average utilization across a day divides its fixed hourly cost across roughly twice as many tokens as one running at 40%, and can profitably charge roughly half as much per token for the same margin. Independent benchmarking gives some visibility into what utilization current hardware and software combinations can actually achieve under realistic load. MLCommons’ MLPerf Inference benchmark suite tests submitted systems under both a latency-bounded “server” scenario and an unconstrained “offline” scenario designed to maximize throughput through batching, and recent rounds have shown substantial throughput gains attributable specifically to newer accelerator generations and to serving-software improvements like better batching and KV-cache management, rather than to raw chip count alone [ 9 ] . (Fact, attributed to the cited benchmark reporting; MLPerf figures describe controlled benchmark conditions and are not a direct measurement of any specific commercial provider’s live-traffic utilization, which providers do not publish.) $\text{price per token} \gtrsim \frac{C_{\text{fixed}} + C_{\text{energy}}}{U \cdot T_{\max}} + m$. Here $C_{\text{fixed}}$ is the amortized hardware and facility cost per unit time, $C_{\text{energy}}$ the energy cost per unit time, $T_{\max}$ the chip’s maximum achievable token throughput under ideal batching, U $\in$ (0,1] the realized utilization fraction, and m a margin term. This is not a model any provider discloses or that this article claims to have measured; it is a bookkeeping identity stated to make one point precisely: U appears in the denominator, so it multiplies with, rather than merely adds to, every other lever in this article. Halving fixed cost through better batching and halving it again through utilization gains compound rather than add. (Analysis: an accounting identity offered to expose the assumption, not a fitted or disclosed cost model.)