What does this equation mean?

A slightly more general way to see the same point: any observed score can be written as $s_{\mathrm{obs}} = a_{\mathrm{true}} + \delta_{\mathrm{contam}} + \varepsilon_{\mathrm{config}}$. where $a_{\mathrm{true}}$ is the capability the benchmark intends to measure, $\delta_{\mathrm{contam}}$ $\geq$ 0 is a one-directional inflation term from training-set overlap, and $\varepsilon_{\mathrm{config}}$ is noise contributed by harness, prompt, effort level, and grading choices, which can push the observed score in either direction. Two systems’ $s_{\mathrm{obs}}$ values can differ substantially even when their $a_{\mathrm{true}}$ values are identical, and can appear equal even when their $a_{\mathrm{true}}$ values are not, purely through the other two terms. Nothing about this requires bad faith on anyone’s part; it follows from the measurement being taken at all.