$$\hat{s}_{i,b} = g_b(\theta_i) + \delta_{i,b} + \varepsilon_{i,b}$$

Equation 3 · Model Evaluation

The Hardest Unsolved Problems in Frontier AI Model Comparisons

This equation states an equality: the expressions on both sides have the same value under the article’s assumptions.

The underlying reason none of these approaches resolves the problem is structural. A published score for model i on benchmark b can be decomposed, at least conceptually, as $\hat{s}_{i,b} = g_b(\theta_i) + \delta_{i,b} + \varepsilon_{i,b}$. where $\theta_i$ is some unobservable underlying ability vector for model i , $g_b$ is the mapping from ability to expected score that is specific to benchmark b and not shared across benchmarks, $\delta_{i,b}$ is a contamination or leakage term specific to that model-benchmark pair, and $\varepsilon_{i,b}$ is sampling and decoding noise. Because $g_b$ differs across benchmarks by construction — a multiple-choice knowledge test and a pairwise human-preference vote are not measuring the same projection of…

Meanings in this article

• $g_b$: the mapping from ability to expected score that is specific to benchmark b and not shared across benchmarks.
• $\theta_i$: some unobservable underlying ability vector for model i.
• $\delta_{i,b}$: a contamination or leakage term specific to that model-benchmark pair.
• $\varepsilon_{i,b}$: sampling and decoding noise.

Equation guide → · Article →