$$s = g(\theta, \mathcal{H}, T, k, \mathcal{P}, t) + \varepsilon$$

Equation 3 · Model Evaluation

How to Actually Compare Frontier AI Models Without Building a Misleading Leaderboard

This equation states an equality: the expressions on both sides have the same value under the article’s assumptions.

Formally, an observed score s is not a property of a model $\theta$ alone. It is closer to $s = g(\theta, \mathcal{H}, T, k, \mathcal{P}, t) + \varepsilon$. a function of the weights $\theta$ , the harness $\mathcal{H}$ , the sampling temperature T , the number of samples k and how they are aggregated, the prompt template $\mathcal{P}$ , the date t a snapshot was queried, plus sampling noise $\varepsilon$ . A citation that fixes $\theta$ and leaves the other five free has not specified a comparable quantity. OpenAI’s own simple-evals table makes the same point from a different angle: rows the lab ran itself under a disclosed zero-shot chain-of-thought protocol sit beside rows for competitors’ models labelled “unknown” prompt style, sourced from…

Meanings in this article

• $s$: the observed score.
• $\theta$: the function of the weights.
• $\mathcal{H}$: the harness.
• $T$: the sampling temperature.
• $\mathcal{P}$: the prompt template.
• $\varepsilon$: the plus sampling noise.

Equation guide → · Article →