← Mathematical compendium

Published equation contexts

pass@k=1−(1−p)k,passk=pk\text{pass@}k = 1-(1-p)^k, \qquad \text{pass}^k = p^k

Why this formula appears here

Large language model agents are not deterministic in practice, even holding the prompt and the environment fixed, and this is why the tool-use literature evaluates repeated trials rather than single runs. τ-bench formalizes the distinction with two related quantities. If a single attempt succeeds with probability p , and attempts are treated as independent, pass@k=1−(1−p)k,passk=pk\text{pass@}k = 1-(1-p)^k, \qquad \text{pass}^k = p^k. where pass@ k is the probability that at least one of k attempts succeeds, and pass ^k is the probability that all k succeed [ 3 ] . The two statistics move in opposite directions as k grows: pass@ k climbs toward certainty, which is the right question when a system can retry until something works or a human picks the…

Read the full article-specific guide →

Read the representative guide

How to interpret it

Read it with the definitions, units, and assumptions supplied by the article.

Research cited beside this formula

Learn the mathematics behind it

Published contexts (1)

A symbol can carry a different meaning in another article. Each occurrence keeps its own guide and term definitions.

pass@k=1−(1−p)k,passk=pk,\text{pass@}k = 1-(1-p)^k, \qquad \text{pass}^k = p^k,

Equation 2 · Model Evaluation

The Hardest Unsolved Problems in AI Agent Evaluation and Reliability

This equation states an equality: the expressions on both sides have the same value under the article’s assumptions.

Large language model agents are not deterministic in practice, even holding the prompt and the environment fixed, and this is why the tool-use literature evaluates repeated trials rather than single runs. τ-bench formalizes the distinction with two related quantities. If a single attempt succeeds with probability p , and attempts are treated as independent, pass@k=1−(1−p)k,passk=pk\text{pass@}k = 1-(1-p)^k, \qquad \text{pass}^k = p^k. where pass@ k is the probability that at least one of k attempts succeeds, and pass ^k is the probability that all k succeed [ 3 ] . The two statistics move in opposite directions as k grows: pass@ k climbs toward certainty, which is the right question when a system can retry until something works or a human picks the…

Meanings in this article

  • pp: the probability.
Equation guide → · Article →