subscript

The passage around this formula

where $\theta_i$ is some unobservable underlying ability vector for model i , $g_b$ is the mapping from ability to expected score that is specific to benchmark b and not shared across benchmarks, $\delta_{i,b}$ is a contamination or leakage term specific to that model-benchmark pair, and $\varepsilon_{i,b}$ is sampling and decoding noise. Because $g_b$ differs across benchmarks by construction — a multiple-choice knowledge test and a pairwise human-preference vote are not measuring the same projection of $\theta_i$ — there is no aggregation operator that turns a vector of $\hat{s}_{i,b}$ values into a single number without an additional, unverified assumption about how the $g_b$ functions relate to one…

Read this part in the article →