What does this equation mean?

4. Insufficient trial counts producing false confidence. A success rate reported from a small number of trials carries a much wider margin of error than the single reported number suggests, and the interval widens fast as the trial count shrinks. For a naive binomial estimate — treating each trial as an independent coin flip with unknown success probability p — the 95 percent confidence interval around an observed rate $\hat{p}$ from n trials is approximately $\hat{p} \pm 1.96\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$. At ten trials and an observed 90 percent success rate, that formula gives an interval of roughly plus or minus nineteen percentage points — an agent genuinely succeeding anywhere from about 71 percent to essentially 100 percent of the time is statistically indistinguishable from the ten runs actually observed. Evan Miller, writing at Anthropic, makes this the central practical argument of a 2024 paper on evaluation statistics: he recommends that “new evals should contain at least 1,000 questions in order to have good signaling ability,” and works a concrete power calculation showing that detecting a genuine three-percentage-point difference between two systems, at conventional statistical standards (80 percent power, 5 percent significance), requires approximately 969 questions under reasonable assumptions [ 8 ] . Most agent benchmarks reported in the field — SWE-bench Verified’s 500 instances, τ-bench’s much smaller per-domain task sets — sit well below that figure, which does not make them worthless, but does mean that a one- or two-point difference between two systems on such a suite is frequently statistical noise dressed as a finding.