Tagged knowledge distillation · show all articles
A capable small model can be reached by compressing a large one, training a small one on purpose, searching architecture space for it, or routing around density altogether. These are four different engineering problems, not four contestants in one race.
Distillation, pruning and quantization are not three names for the same haircut. Each removes a different kind of thing from a trained checkpoint, in a specific order, and each recovers the resulting loss by a different mechanism.
Before a phone could run a language model, it had to run anything at all: a narrower convolution, a dedicated chip, an always-listening detector, and a trick for compressing what a bigger model already knew. This is that decade, in order.