What does this equation mean?

What this buys, mechanically, is a denser training signal per example. A hard label carries $\log_2$ C bits at most, for C classes, and in practice far less once you account for how skewed real label distributions are. A full teacher distribution over the same classes carries a graded judgment about every one of them, on every training example, for free. That is why a student trained this way can match a network many times its size on the same data budget: it is not learning faster, it is being taught with a richer curriculum. The modern survey literature on this technique describes it as the backbone of an entire ecosystem for building small open models from large proprietary ones, with the harder problem no longer the loss function above but generating enough varied prompts and teacher completions to distill against [ 14 ] .