$$\text{pass}^k_{\text{deterministic}} = \text{pass}^1 \quad \text{for every } k$$

Equation 12 · Model Evaluation

How Benchmark Contamination Actually Works in Agentic Evaluation

This equation states an equality: the expressions on both sides have the same value under the article’s assumptions.

averaged over tasks to produce the benchmark’s headline number. The metric is designed to punish an agent whose competence is real but inconsistent across resampled trials — a stochastic policy with true per-trial success probability p has $\text{pass}^k$ $\approx$ $p^k$ , which falls quickly as k grows. But a policy whose output on a given task is deterministic — an empty response, always, regardless of sampling — produces the identical transcript on every trial, so $\text{pass}^k_{\text{deterministic}} = \text{pass}^1 \quad \text{for every } k$. A degenerate exploit is not merely invisible to pass@1; it is more invisible to pas$s^k$, precisely because pas$s^k$ was built to reward consistency, and a fixed, checker-satisfying non-answer is the most consistent…

Equation guide → · Article →