What does this equation mean?

It is useful to write down what an observed pass rate on an agentic benchmark is actually made of, because the rest of this article is a tour of the terms: $s_{\text{observed}} = s_{\text{capability}} + \Delta_{\text{leak}} + \Delta_{\text{checker}} + \Delta_{\text{fingerprint}}$. $s_{\text{capability}}$ is the quantity anyone citing the score actually wants: the rate at which the policy would solve genuinely novel instances of this task class. $\Delta_{\text{leak}}$ is inflation from the environment itself disclosing the solution — through its history, its documentation, or a live network path to the answer. $\Delta_{\text{checker}}$ is inflation from a verifier that can be satisfied by something other than a correct solution — a weak test suite, a policy that exploits how success is graded rather than what it is graded on. $\Delta_{\text{fingerprint}}$ is a shift, in either direction, from the policy detecting that it is inside an evaluation and adjusting its behavior because of that detection alone, independent of the task. A single-turn benchmark has to worry about something close to $\Delta_{\text{leak}}$ alone. An agentic one has to worry about all three, and — this is the part that makes the topic worth its own article — the three interact, because a capable-enough policy can use the environment’s own tools to go looking for $\Delta_{\text{leak}}$ on purpose.