Numerator: hat p (1-hat p)

The passage around this formula

The second argument is about how much any one number can actually tell you, independent of harness effects, purely from how many tasks produced it. Treat a benchmark’s headline pass rate as an estimate $\hat p$ of a task-set success probability, drawn from n roughly independent trials. The familiar normal approximation to a binomial confidence interval, $\hat p \; \pm \; z_{\alpha/2}\sqrt{\frac{\hat p (1-\hat p)}{n}}$. gives a rough sense of how much noise sits under a given n , even granting the generous and almost certainly false assumption that every task in the set is an equally difficult coin flip. At the widest plausible spread, $\hat p$ = 0.5 , a 95% interval ( $z_{\alpha/2}$ $\approx$ 1.96 ) works out to roughly $\pm$ 13.9 points on a…

Read this part in the article →