=

The passage around this formula

Formally, an observed score s is not a property of a model $\theta$ alone. It is closer to $s = g(\theta, \mathcal{H}, T, k, \mathcal{P}, t) + \varepsilon$. a function of the weights $\theta$ , the harness $\mathcal{H}$ , the sampling temperature T , the number of samples k and how they are aggregated, the prompt template $\mathcal{P}$ , the date t a snapshot was queried, plus sampling noise $\varepsilon$ . A citation that fixes $\theta$ and leaves the other five free has not specified a comparable quantity. OpenAI’s own simple-evals table makes the same point from a different angle: rows the lab ran itself under a disclosed zero-shot chain-of-thought protocol sit beside rows for competitors’ models labelled “unknown” prompt style, sourced from…

Read this part in the article →