$$s_{\text{observed}} = s_{\text{capability}} + \Delta_{\text{contamination}} + \Delta_{\text{configuration}}$$

Equation 1 · Model Evaluation

Ten Failure Modes That Show Up Across Every Frontier Model, Not Just One

This equation states an equality: the expressions on both sides have the same value under the article’s assumptions.

One assumption threads through everything that follows, and it is worth making explicit before the catalogue starts. A published benchmark score is not a direct read of capability; it is a sum of capability plus at least two other things that vary independently of it: $s_{\text{observed}} = s_{\text{capability}} + \Delta_{\text{contamination}} + \Delta_{\text{configuration}}$. $\Delta_{\text{contamination}}$ is inflation from the model having seen the benchmark, or something close to it, during training. $\Delta_{\text{configuration}}$ is variance from everything about how the question was asked, formatted, and scored — prompt wording, decoding settings, which variant answered. Both terms are provider-specific and time-specific; neither is disclosed by the headline number. Three of…

Meanings in this article

• $\Delta_{\text{contamination}}$: inflation from the model having seen the benchmark, or something close to it, during training.
• $\Delta_{\text{configuration}}$: variance from everything about how the question was asked, formatted, and scored — prompt wording, decoding settings, which variant answered.

Equation guide → · Article →