√

The passage around this formula

Reported variance, not a bare point estimate. A score computed from n independent trials with an underlying success probability $\hat p$ carries a standard error of roughly $\mathrm{SE}(\hat p) \approx \sqrt{\frac{\hat p (1-\hat p)}{n}}$. and two point estimates whose intervals overlap should not be reported as a ranking. Evan Miller’s statistical treatment of language-model evaluation makes this argument in more general form, framing individual evaluation questions as draws from an unseen larger population and deriving the formulas needed to report genuine uncertainty rather than a single noisy number [ 1 ] . Epoch AI’s benchmarking hub applies the same discipline operationally: it runs each model sixteen times on GPQA Diamond and Mock…

Read this part in the article →