Equation 4 · How a Model Actually Gets Small Enough to Run on a Phone
What does this equation mean?
Read the formula alongside the article passage below. Each part has a deeper page with its role in the equation, the supporting passage and nearby citations.
This equation states an equality: the expressions on both sides have the same value under the article’s assumptions. Read the equation part by part below; each part has a contextual explanation and a link to its mathematical background.
Read it piece by piece
Symbol σ
σ is part of the quantity the equation computes from the expression on the right.
Symbol z
z is an argument of the function-like quantity on the left; its role is set by that function’s stated inputs.
Symbol i
i occurs above the fraction bar. The numerator is divided by the entire denominator below it.
Symbol z_i
occurs above the fraction bar. The numerator is divided by the entire denominator below it.
Symbol T
T is an input to the expression that computes the quantity on the left.
Symbol j
j occurs below the fraction bar. The quantity above the bar is divided by this expression; zero is excluded as a denominator.
Symbol z_j
occurs below the fraction bar. The quantity above the bar is divided by this expression; zero is excluded as a denominator.
=
The expressions on both sides represent the same quantity under the stated assumptions.
See an illustrated explanation →subscript
The lower label selects a particular version, component, or indexed member of the quantity. For example, x₀ and xₜ can be values at different positions.
Numerator: exp(z_i / T)
The complete quantity above the fraction bar.
See an illustrated explanation →Denominator: sum_j exp(z_j / T)
The complete quantity below the fraction bar; it must be nonzero for this division.
See an illustrated explanation →Starting index or lower bound: j
This label says where the repeated addition, multiplication, or accumulation starts. Read its value or condition together with the article’s description of the index.
How to interpret it
With a fixed numerator, increasing a nonzero denominator reduces the fraction. Read it with the definitions, units, and assumptions supplied by the article.
What the article says around this equation
Distillation recovers it by training the small “student” network against the large “teacher” network’s full output distribution rather than against the label alone. Write the teacher’s and student’s pre-softmax outputs for a given input as and . A softmax with a temperature T is . At T=1 this is the ordinary softmax used for prediction. Raising T flattens the distribution, pulling the small probabilities assigned to wrong classes up toward visibility, which is exactly the dark knowledge the method wants to expose. The training objective blends two terms: ordinary cross-entropy against the true label, and a match between the student’s and teacher’s…
Read the full surrounding passage
Distillation recovers it by training the small “student” network against the large “teacher” network’s full output distribution rather than against the label alone. Write the teacher’s and student’s pre-softmax outputs for a given input as and . A softmax with a temperature T is . At T=1 this is the ordinary softmax used for prediction. Raising T flattens the distribution, pulling the small probabilities assigned to wrong classes up toward visibility, which is exactly the dark knowledge the method wants to expose. The training objective blends two terms: ordinary cross-entropy against the true label, and a match between the student’s and teacher’s temperature-softened distributions, measured by KL divergence:
Sources cited in the article section
- [1] Distilling the Knowledge in a Neural Network ↗
- [14] A Survey on Knowledge Distillation of Large Language Models ↗
These citations give research context. Read each source to check which claims it supports.
Return to How a Model Actually Gets Small Enough to Run on a Phone