← Mathematical compendium

Published equation contexts

passk(t)=∏i=1k1 ⁣[trial i on task t succeeds]\text{pass}^k(t) = \prod_{i=1}^{k} \mathbb{1}\!\left[\text{trial } i \text{ on task } t \text{ succeeds}\right]

Why this formula appears here

tau-bench itself — the benchmark in which that first exploit was found — was built to move past shallow grading, simulating a multi-turn conversation between a user (played by a language model) and a tool-using agent, then scoring the conversation against the resulting database state, with a passks^k metric meant to capture reliability across repeated trials rather than a single lucky success [ 7 ] . The mechanism is worth stating precisely, because it is a real assumption passks^k makes, and the empty-response exploit breaks exactly it: passk(t)=∏i=1k1 ⁣[trial i on task t succeeds]\text{pass}^k(t) = \prod_{i=1}^{k} \mathbb{1}\!\left[\text{trial } i \text{ on task } t \text{ succeeds}\right]. averaged over tasks to produce the benchmark’s headline number. The metric is designed to punish an agent whose competence is real but…

Read the full article-specific guide →

Read the representative guide

kk

Symbol k

k appears in the bound of this product. The bound states where the repeated operation starts, ends, or which values it includes.

Read this term in its guide →
ii

Symbol i

i appears in the bound of this product. The bound states where the repeated operation starts, ends, or which values it includes.

Read this term in its guide →
i=1i=1

Starting index or lower bound: i=1

This label says where the repeated addition, multiplication, or accumulation starts. Read its value or condition together with the article’s description of the index.

Read this term in its guide →
kk

Ending index or upper bound: k

This label says where the repeated addition, multiplication, or accumulation stops. It sets the last term or end of the range.

Read this term in its guide →

How to interpret it

Read it with the definitions, units, and assumptions supplied by the article.

Research cited beside this formula

Published contexts (1)

A symbol can carry a different meaning in another article. Each occurrence keeps its own guide and term definitions.

passk(t)=∏i=1k1 ⁣[trial i on task t succeeds],\text{pass}^k(t) = \prod_{i=1}^{k} \mathbb{1}\!\left[\text{trial } i \text{ on task } t \text{ succeeds}\right],

Equation 8 · Model Evaluation

How Benchmark Contamination Actually Works in Agentic Evaluation

This equation states an equality: the expressions on both sides have the same value under the article’s assumptions.

tau-bench itself — the benchmark in which that first exploit was found — was built to move past shallow grading, simulating a multi-turn conversation between a user (played by a language model) and a tool-using agent, then scoring the conversation against the resulting database state, with a passks^k metric meant to capture reliability across repeated trials rather than a single lucky success [ 7 ] . The mechanism is worth stating precisely, because it is a real assumption passks^k makes, and the empty-response exploit breaks exactly it: passk(t)=∏i=1k1 ⁣[trial i on task t succeeds]\text{pass}^k(t) = \prod_{i=1}^{k} \mathbb{1}\!\left[\text{trial } i \text{ on task } t \text{ succeeds}\right]. averaged over tasks to produce the benchmark’s headline number. The metric is designed to punish an agent whose competence is real but…

Equation guide → · Article →