≈

The passage around this formula

The structural difficulty is that a next-token distribution is not an epistemic state. A model trained to maximise likelihood over text produces a confident-sounding continuation because confident-sounding continuations are what the corpus contains, not because it has assessed its own evidence. Calibration can be measured — for a predicted confidence c one can ask whether $\Pr(\text{correct} \mid \hat{c} = c) \approx c$. holds across bins — and it can be improved by post-hoc adjustment. What has not been demonstrated is a mechanism by which a model represents its own ignorance in a way that survives fine-tuning, distribution shift, and the pressure of an objective that rewards answering.

Read this part in the article →