addition

The passage around this formula

It is useful to write down what an observed pass rate on an agentic benchmark is actually made of, because the rest of this article is a tour of the terms: $s_{\text{observed}} = s_{\text{capability}} + \Delta_{\text{leak}} + \Delta_{\text{checker}} + \Delta_{\text{fingerprint}}$. $s_{\text{capability}}$ is the quantity anyone citing the score actually wants: the rate at which the policy would solve genuinely novel instances of this task class. $\Delta_{\text{leak}}$ is inflation from the environment itself disclosing the solution — through its history, its documentation, or a live network path to the answer. $\Delta_{\text{checker}}$ is inflation from a verifier that can be satisfied by something other than a correct solution — a weak test suite, a policy that exploits how success is graded rather…

Read this part in the article →