What does this equation mean?

​ , so getting a usably precise estimate — a relative error in the neighbourhood of 20 to 25 percent, comparable to observing on the order of twenty events — requires $n \approx \frac{k}{q_\tau}$. trials in total. This is where the arithmetic becomes unforgiving in a way the mean-difference case never does. Detecting a fixed three-percentage-point difference between two average success rates takes roughly the same, bounded number of trials — on the order of a thousand — regardless of how good either system actually is, because that calculation is about the spread of a difference, not the rarity of an event. Bounding or estimating a rare catastrophic-failure rate is different in kind: the required sample size is inversely proportional to the rate itself. At $q_\tau$ = 10^{-3} , getting even a rough twenty-event estimate needs on the order of twenty thousand trials. At $q_\tau$ = 10^{-5} — the dangerous-failure probability ceiling regulated safety-critical software is held to under the SIL 4 standard, a bar Rabanser and colleagues cite alongside the FAA’s target of roughly one catastrophic error per billion flight hours as the kind of scale that consequential AI evaluation is implicitly being compared against [ 1 ] — the same twenty-event estimate needs on the order of two million trials. The safer the system claims to be, the more trials it takes to actually confirm that claim at the tail, which is exactly backwards from how confidence usually accrues, and exactly why a modest evaluation budget that comfortably detects an average-case improvement can be wildly insufficient for confirming that a catastrophic failure mode has genuinely been engineered out rather than merely not yet observed.