Tagged memory bandwidth · show all articles
A datacentre gives a model whatever it asks for. A device gives it a fixed envelope — memory, bandwidth, a sustained thermal budget, a battery — and a model has to be built to live inside it, not merely shrunk until it fits.
Two axes govern the next decade of frontier model systems: where the next increment of capability is bought, and whether the physical constraints tighten. Crossing them gives four futures, none of which is the favourite.
Serving a frontier model is a thermal and logistical problem before it is a computational one. The binding constraints are bytes moved, joules dissipated, and megawatts delivered — in that order.
Decoding is not limited by arithmetic. It is limited by how fast memory can be read, and almost every serving technique in production exists to work around that single fact.