What does this equation mean?

For n = 500 and $\hat p$ near one half, this is on the order of two percentage points — before accounting for the additional variance introduced by sampling temperature, agentic scaffolding, or a different number of retries per problem. A published gap of four or five points between two systems evaluated under different harnesses, different reasoning-effort settings, and different retry budgets is not obviously a capability gap at all; it may be substantially a measurement-configuration gap. This is precisely why the practice of building a single cross-vendor leaderboard out of self-reported scores, each collected under undisclosed or differing scaffolds, is not a defensible ranking — it is several different experiments stapled together and relabeled as one.