What does this equation mean?

That instability is worth making precise rather than gesturing at. If a benchmark of n instances is treated as n independent Bernoulli trials with true resolve probability p , the standard error of the observed proportion $\hat p$ is $\mathrm{SE}(\hat p) = \sqrt{\frac{\hat p (1 - \hat p)}{n}}$. For n = 500 and $\hat p$ near one half, this is on the order of two percentage points — before accounting for the additional variance introduced by sampling temperature, agentic scaffolding, or a different number of retries per problem. A published gap of four or five points between two systems evaluated under different harnesses, different reasoning-effort settings, and different retry budgets is not obviously a capability gap at all; it may be substantially a measurement-configuration gap. This is precisely why the practice of building a single cross-vendor leaderboard out of self-reported scores, each collected under undisclosed or differing scaffolds, is not a defensible ranking — it is several different experiments stapled together and relabeled as one.