$$s_{\mathrm{obs}} = a_{\mathrm{true}} + \delta_{\mathrm{contam}} + \varepsilon_{\mathrm{config}}$$

Equation 7 · Model Evaluation

Frontier AI Model Comparisons: A First-Principles Introduction

This equation states an equality: the expressions on both sides have the same value under the article’s assumptions.

A slightly more general way to see the same point: any observed score can be written as $s_{\mathrm{obs}} = a_{\mathrm{true}} + \delta_{\mathrm{contam}} + \varepsilon_{\mathrm{config}}$. where $a_{\mathrm{true}}$ is the capability the benchmark intends to measure, $\delta_{\mathrm{contam}}$ $\geq$ 0 is a one-directional inflation term from training-set overlap, and $\varepsilon_{\mathrm{config}}$ is noise contributed by harness, prompt, effort level, and grading choices, which can push the observed score in either direction. Two systems’ $s_{\mathrm{obs}}$ values can differ substantially even when their $a_{\mathrm{true}}$ values are identical, and can appear equal even when their $a_{\mathrm{true}}$ values are not, purely through the other two terms. Nothing about this requires…

Meanings in this article

• $s_{\mathrm{obs}}$: any observed score.
• $\varepsilon_{\mathrm{config}}$: noise contributed by harness, prompt, effort level, and grading choices, which can push the observed score in either direction.

Equation guide → · Article →