addition

The passage around this formula

A simple decomposition makes clear why this is a measurement problem and not merely an inconvenience. Let $s_{\text{obs}}$ be an observed benchmark score, $s_{\text{true}}$ the underlying capability the benchmark intends to measure, and c the fraction of test instances the model has effectively memorized, whether verbatim or through a rephrased variant. If a memorized instance is answered correctly with certainty, then approximately $s_{\text{obs}} = (1 - c)\, s_{\text{true}} + c$. Every published leaderboard number implicitly assumes c $\approx$ 0 . The methods above show that assumption is not something the field can currently verify for a given open-weight release, only something it can occasionally catch violated in…

Read this part in the article →