addition

The passage around this formula

One assumption threads through everything that follows, and it is worth making explicit before the catalogue starts. A published benchmark score is not a direct read of capability; it is a sum of capability plus at least two other things that vary independently of it: $s_{\text{observed}} = s_{\text{capability}} + \Delta_{\text{contamination}} + \Delta_{\text{configuration}}$. $\Delta_{\text{contamination}}$ is inflation from the model having seen the benchmark, or something close to it, during training. $\Delta_{\text{configuration}}$ is variance from everything about how the question was asked, formatted, and scored — prompt wording, decoding settings, which variant answered. Both terms are provider-specific and time-specific; neither is disclosed by the headline number. Three of…

Read this part in the article →