Numerator: 1

The passage around this formula

It is worth seeing precisely why an all-or-nothing rule across permutations is so stringent, and where the intuition needs a caveat. If a model’s answer on each rotated pass were an independent uniform guess among k options, the probability of passing all N rotations by chance would be $P_{\mathrm{chance}} = \left(\frac{1}{k}\right)^{N}$. which for a four-option item run through all four rotations is (1/4)^4 , under half a percent. That is not a real prediction of model behavior — actual models are not memoryless guessers, and a model with a fixed positional bias will do considerably worse than this floor once the correct answer rotates away from its favored slot, while a model with genuine content understanding will pass…

Read this part in the article →