What does this equation mean?

tau-bench itself — the benchmark in which that first exploit was found — was built to move past shallow grading, simulating a multi-turn conversation between a user (played by a language model) and a tool-using agent, then scoring the conversation against the resulting database state, with a pas$s^k$ metric meant to capture reliability across repeated trials rather than a single lucky success [ 7 ] . The mechanism is worth stating precisely, because it is a real assumption pas$s^k$ makes, and the empty-response exploit breaks exactly it: $\text{pass}^k(t) = \prod_{i=1}^{k} \mathbb{1}\!\left[\text{trial } i \text{ on task } t \text{ succeeds}\right]$. averaged over tasks to produce the benchmark’s headline number. The metric is designed to punish an agent whose competence is real but inconsistent across resampled trials — a stochastic policy with true per-trial success probability p has $\text{pass}^k$ $\approx$ $p^k$ , which falls quickly as k grows. But a policy whose output on a given task is deterministic — an empty response, always, regardless of sampling — produces the identical transcript on every trial, so