≥

The passage around this formula

where $a_{\mathrm{true}}$ is the capability the benchmark intends to measure, $\delta_{\mathrm{contam}}$ $\geq$ 0 is a one-directional inflation term from training-set overlap, and $\varepsilon_{\mathrm{config}}$ is noise contributed by harness, prompt, effort level, and grading choices, which can push the observed score in either direction. Two systems’ $s_{\mathrm{obs}}$ values can differ substantially even when their $a_{\mathrm{true}}$ values are identical, and can appear equal even when their $a_{\mathrm{true}}$ values are not, purely through the other two terms. Nothing about this requires bad faith on anyone’s part; it follows from the measurement being taken at all.

Read this part in the article →