superscript

The passage around this formula

tau-bench itself — the benchmark in which that first exploit was found — was built to move past shallow grading, simulating a multi-turn conversation between a user (played by a language model) and a tool-using agent, then scoring the conversation against the resulting database state, with a pas$s^k$ metric meant to capture reliability across repeated trials rather than a single lucky success [ 7 ] . The mechanism is worth stating precisely, because it is a real assumption pas$s^k$ makes, and the empty-response exploit breaks exactly it: $\text{pass}^k(t) = \prod_{i=1}^{k} \mathbb{1}\!\left[\text{trial } i \text{ on task } t \text{ succeeds}\right]$. averaged over tasks to produce the benchmark’s headline number. The metric is designed to punish an agent whose competence is real but…

Read this part in the article →