Tagged edge inference · show all articles
A model that passes every benchmark on the bench can still fail in the field — not from one bug, but from ten separate ways a device diverges from the conditions the benchmark assumed.
Distillation, pruning and quantization are not three names for the same haircut. Each removes a different kind of thing from a trained checkpoint, in a specific order, and each recovers the resulting loss by a different mechanism.
A datacentre gives a model whatever it asks for. A device gives it a fixed envelope — memory, bandwidth, a sustained thermal budget, a battery — and a model has to be built to live inside it, not merely shrunk until it fits.