Tagged mixture of experts · show all articles
Meta's own papers specify exactly how Llama encodes position, shares attention across heads, gates its activations and normalizes its signals — and exactly what changed, and did not change, from Llama 2 through Llama 3 to Llama 4.
A capable small model can be reached by compressing a large one, training a small one on purpose, searching architecture space for it, or routing around density altogether. These are four different engineering problems, not four contestants in one race.
Three ways of getting more capability into a servable model, each with a different cost profile, a different failure mode, and a different answer to the question of what a parameter is for.