What does this equation mean?

Hinton, Vinyals and Dean’s 2015 paper, “Distilling the Knowledge in a Neural Network,” is the one the field actually cites, and it earned that position by generalizing the 2006 idea and giving it a mechanism simple enough to fit into any neural network’s training pipeline. Instead of training a small model against a large model’s hard output labels, distillation trains it against the large “teacher” model’s full, softened probability distribution over classes — the softmax output computed at a raised temperature T : $q_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}$. where $z_i$ are the teacher’s pre-softmax logits. “Using a higher value for T produces a softer probability distribution over classes” [ 4 ] , and it is precisely that softness — the relative probabilities the teacher assigns to the wrong answers, not only the single correct label — that carries information a hard label discards: how confidently a teacher model preferred one wrong answer over another says far more about the shape of its decision boundary than one correct label does. The paper states the mechanism directly: “Knowledge is transferred to the distilled model by training it on a transfer set and using a soft target distribution for each case in the transfer set that is produced by using the cumbersome model with a high temperature in its softmax” [ 4 ] . On a speech-recognition acoustic model, the authors reported that “more than 80% of the improvement in frame classification accuracy achieved by using an ensemble of 10 models is transferred to the distilled model” [ 4 ] — most of what ten expensive models knew, recovered in a single model built to run where the ensemble could not.