$$s_{\text{observed}} = s_{\text{capability}} + \Delta_{\text{leak}} + \Delta_{\text{checker}} + \Delta_{\text{fingerprint}}$$

Equation 1 · Model Evaluation

How Benchmark Contamination Actually Works in Agentic Evaluation

This equation states an equality: the expressions on both sides have the same value under the article’s assumptions.

It is useful to write down what an observed pass rate on an agentic benchmark is actually made of, because the rest of this article is a tour of the terms: $s_{\text{observed}} = s_{\text{capability}} + \Delta_{\text{leak}} + \Delta_{\text{checker}} + \Delta_{\text{fingerprint}}$. $s_{\text{capability}}$ is the quantity anyone citing the score actually wants: the rate at which the policy would solve genuinely novel instances of this task class. $\Delta_{\text{leak}}$ is inflation from the environment itself disclosing the solution — through its history, its documentation, or a live network path to the answer. $\Delta_{\text{checker}}$ is inflation from a verifier that can be satisfied by something other than a correct solution — a weak test suite, a policy that exploits how success is graded rather…

Meanings in this article

• $s_{\text{capability}}$: the quantity anyone citing the score actually wants: the rate at which the policy would solve genuinely novel instances of this task class.
• $\Delta_{\text{leak}}$: inflation from the environment itself disclosing the solution — through its history, its documentation, or a live network path to the answer.
• $\Delta_{\text{checker}}$: inflation from a verifier that can be satisfied by something other than a correct solution — a weak test suite, a policy that exploits how success is graded rather than what it is graded on.
• $\Delta_{\text{fingerprint}}$: a shift, in either direction, from the policy detecting that it is inside an evaluation and adjusting its behavior because of that detection alone, independent of the task.

Equation guide → · Article →