Tagged serving systems · show all articles
A token is a billing unit, not a physical quantity. What a buyer actually pays for is a parallel pass, a serial one, a block of resident memory, and a stretch of reasoning that is charged in full and never returned.
Decoding is not limited by arithmetic. It is limited by how fast memory can be read, and almost every serving technique in production exists to work around that single fact.
A frontier model is a budget divided three ways. Where the compute goes—into weights, into behaviour, or into each individual answer—determines almost everything else about the system.