≈

The passage around this formula

Because a “pass” verdict is itself often a noisy measurement, not a fact, it is worth being explicit about how much noise a single run’s outcome carries before treating a change in the pass rate as real. If a task’s true pass rate is p under a baseline configuration and a candidate change is worth detecting only once it shifts that rate by at least $\delta$ , the number of repeated trials needed per configuration to detect the shift reliably — at significance level $\alpha$ and statistical power 1-$\beta$ — is approximately $n \approx \frac{2\,p(1-p)\,(z_{\alpha/2} + z_{\beta})^2}{\delta^2}$. Plugging in a fairly ordinary case — a baseline pass rate of 70 percent, a minimum shift worth caring about of 10 percentage points, a 5 percent…

Read this part in the article →