Symbol T

The passage around this formula

…against the large “teacher” network’s full output distribution rather than against the label alone. Write the teacher’s and student’s pre-softmax outputs for a given input as $z_t$ and $z_s$ . A softmax with a temperature T is $\sigma(z)_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}$. At T=1 this is the ordinary softmax used for prediction. Raising T flattens the distribution, pulling the small probabilities assigned to wrong classes up toward visibility, which is exactly the dark knowledge the method wants to expose. The training objective blends two…

Read this part in the article →