What does this equation mean?

The reason this ordering matters is not procedural fussiness. Zhang and Nanda’s systematic study of activation patching found that the choice of corruption distribution and evaluation metric — decisions usually made informally, late, and sometimes after a first look at the data — can by itself change which components a patching sweep identifies as important [ 3 ] . If the metric is chosen after the sweep, on the grounds that it produced the most legible result, the investigation has stopped testing a hypothesis and started constructing one to fit the data. The same failure has a clean statistical description. Sweep enough components at a nominal per-test false-positive rate $\alpha$ and report only the one that clears threshold, and the probability that at least one clears by chance alone across k independently tested components is $P_k = 1-(1-\alpha)^k$. which climbs toward certainty well before k reaches the size of a typical layer-by-layer or head-by-head sweep. A held-out set of prompts, reserved and untouched until the hypothesis is fixed, is the practical fix, and it only works if the hypothesis really was fixed first.