subtraction

The passage around this formula

That instability is worth making precise rather than gesturing at. If a benchmark of n instances is treated as n independent Bernoulli trials with true resolve probability p , the standard error of the observed proportion $\hat p$ is $\mathrm{SE}(\hat p) = \sqrt{\frac{\hat p (1 - \hat p)}{n}}$. For n = 500 and $\hat p$ near one half, this is on the order of two percentage points — before accounting for the additional variance introduced by sampling temperature, agentic scaffolding, or a different number of retries per problem. A published gap of four or five points between two systems evaluated under different harnesses, different reasoning-effort settings, and different retry budgets is not obviously a capability gap at all; it may be…

Read this part in the article →