$$\mathcal{L}_{\mathrm{distill}} = (1-\lambda)\,\mathcal{L}_{\mathrm{CE}}(y,\, \sigma(z_s)) + \lambda\, T^2 \, D_{\mathrm{KL}}\big(\sigma(z_t/T) \,\|\, \sigma(z_s/T)\big)$$

Equation 6 · Data & Training

Comparing the Main Approaches to Training Data and Synthetic Data

This equation states an equality: the expressions on both sides have the same value under the article’s assumptions.

Distillation is the oldest of the five and the easiest to state precisely. A student model is trained not on hard labels but on a teacher’s softened output distribution, on the reasoning that the relative probabilities the teacher assigns to wrong answers carry information a one-hot label discards entirely [ 1 ] . Where logits are unavailable — the ordinary case when the teacher is a closed API — the same idea is applied at one remove: the teacher’s sampled text stands in for its distribution, and the student is trained on that text directly. Writing $z_t$ and $z_s$ for teacher and student logits, $\sigma$ for the softmax, y for the hard label, and T for a softening temperature, the canonical…

Meanings in this article

• $z_t$: the writing.

Equation guide → · Article →