Equation 3 · Shrink It, Train It Small, or Search for It: The Main Strategies for Small Models, Compared
What does this equation mean?
Read the formula alongside the article passage below. Each part has a deeper page with its role in the equation, the supporting passage and nearby citations.
the teacher’s and student’s output logits. Read the equation part by part below; each part has a contextual explanation and a link to its mathematical background.
Read it piece by piece
subscript
The lower label selects a particular version, component, or indexed member of the quantity. For example, x₀ and xₜ can be values at different positions.
How to interpret it
Read this expression with the definitions, units, and assumptions supplied by the article.
What the article says around this equation
where and are the teacher’s and student’s output logits, a softmax, T a temperature that softens the distribution, and y the ground-truth label. The second term is the entire point of the method: it transfers the relative probability the teacher assigns to every wrong answer, not just which answer was right, a far richer training signal per example than a raw label alone provides. That is the rationale, stated plainly by the paper that introduced it.
Sources cited in the article section
- [1] Distilling the Knowledge in a Neural Network ↗
- [3] Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning ↗
- [2] Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding ↗
These citations give research context. Read each source to check which claims it supports.