What does this equation mean?

Because a “pass” verdict is itself often a noisy measurement, not a fact, it is worth being explicit about how much noise a single run’s outcome carries before treating a change in the pass rate as real. If a task’s true pass rate is p under a baseline configuration and a candidate change is worth detecting only once it shifts that rate by at least $\delta$ , the number of repeated trials needed per configuration to detect the shift reliably — at significance level $\alpha$ and statistical power 1-$\beta$ — is approximately $n \approx \frac{2\,p(1-p)\,(z_{\alpha/2} + z_{\beta})^2}{\delta^2}$. Plugging in a fairly ordinary case — a baseline pass rate of 70 percent, a minimum shift worth caring about of 10 percentage points, a 5 percent significance level, and 80 percent power — gives $z_{\alpha/2}$ $\approx$ 1.96 and $z_{\beta}$ $\approx$ 0.84 , and the formula returns roughly 330 trials per configuration. That number is the whole argument for treating a single failed run, or even three failed runs out of five, with real suspicion rather than as proof of a regression: at ordinary sample sizes, most of what looks like a pass-rate change between two prompt versions is sampling noise, not signal. Teams that cannot afford several hundred repeated trials on every task for every change — nearly all of them — have exactly two disciplined options: accept a larger $\delta$ before declaring a regression, watching for a trend across several consecutive merges rather than a single run, or concentrate the expensive repeated-trial budget on the smaller set of tasks where a regression would be most costly and accept thinner evidence everywhere else. Either is defensible; treating a two-point pass-rate wobble on a twenty-task suite as a finding is not.