What does this equation mean?

The second argument is about how much any one number can actually tell you, independent of harness effects, purely from how many tasks produced it. Treat a benchmark’s headline pass rate as an estimate $\hat p$ of a task-set success probability, drawn from n roughly independent trials. The familiar normal approximation to a binomial confidence interval, $\hat p \; \pm \; z_{\alpha/2}\sqrt{\frac{\hat p (1-\hat p)}{n}}$. gives a rough sense of how much noise sits under a given n , even granting the generous and almost certainly false assumption that every task in the set is an equally difficult coin flip. At the widest plausible spread, $\hat p$ = 0.5 , a 95% interval ( $z_{\alpha/2}$ $\approx$ 1.96 ) works out to roughly $\pm$ 13.9 points on a fifty-task set — close to the size of tau-bench’s original airline domain or GAIA’s seventy-five-question Level 3 tier — versus roughly $\pm$ 4.4 points on SWE-bench Verified’s five hundred tasks, or $\pm$ 5.1 points on OSWorld’s 369. A three- or four-point difference between two systems on a fifty- or seventy-five-task slice is, on this generous accounting, close to noise; the same gap on a five-hundred-task set is not. Real task-to-task difficulty variation only widens these intervals further, so treat the numbers above as a floor on the uncertainty, not the whole of it. The lesson is not to distrust small benchmarks — tau-bench’s airline domain and GAIA’s Level 3 tier are deliberately small because hard, well-verified tasks in those domains are expensive to build — but to stop reading a two-point movement on either of them as a finding.