Equation 4 · Part 13 · How a Model Actually Gets Small Enough to Run on a Phone
Starting index or lower bound: j
What this part means
This label says where the repeated addition, multiplication, or accumulation starts. Read its value or condition together with the article’s description of the index.
Its job in the formula
j occurs below the fraction bar. The quantity above the bar is divided by this expression; zero is excluded as a denominator.
Full expression→Starting index or lower bound: j→Article meaning
The passage around this formula
Distillation recovers it by training the small “student” network against the large “teacher” network’s full output distribution rather than against the label alone. Write the teacher’s and student’s pre-softmax outputs for a given input as and . A softmax with a temperature T is . At T=1 this is the ordinary softmax used for prediction. Raising T flattens the distribution, pulling the small probabilities assigned to wrong classes up toward visibility, which is exactly the dark knowledge the method wants to expose. The training objective blends two terms: ordinary cross-entropy against the true label, and a match between the student’s and teacher’s…
Learn the underlying idea
Σ adds a collection of terms. Π multiplies them. The lower and upper labels tell you which terms belong to the collection.
Open the illustrated sums and products: repeat an operation over an index guide →
See this notation across published equations →
Sources cited in the article section
- [1] Distilling the Knowledge in a Neural Network ↗
- [14] A Survey on Knowledge Distillation of Large Language Models ↗
These citations provide research context; check each source for the exact claim it supports.