Symbol T

The passage around this formula

…model against a large model’s hard output labels, distillation trains it against the large “teacher” model’s full, softened probability distribution over classes — the softmax output computed at a raised temperature T : $q_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}$. where $z_i$ are the teacher’s pre-softmax logits. “Using a higher value for T produces a softer probability distribution over classes” [ 4 ] , and it is precisely that softness — the relative probabilities the teacher assigns to the wrong answers, not only the single correct…

Read this part in the article →