← All parts of this equation

Equation 2 · Part 4 · The Hardest Unsolved Problems in AI Agent Evaluation and Reliability

=

pass@k=1−(1−p)k,passk=pk,\text{pass@}k = 1-(1-p)^k, \qquad \text{pass}^k = p^k,
=

What this part means

The expressions on both sides represent the same quantity under the stated assumptions.

Its job in the formula

The equals sign connects the complete expression on the left with the complete expression on the right. Both sides must have compatible units.

The passage around this formula

Large language model agents are not deterministic in practice, even holding the prompt and the environment fixed, and this is why the tool-use literature evaluates repeated trials rather than single runs. τ-bench formalizes the distinction with two related quantities. If a single attempt succeeds with probability p , and attempts are treated as independent, pass@k=1−(1−p)k,passk=pk\text{pass@}k = 1-(1-p)^k, \qquad \text{pass}^k = p^k. where pass@ k is the probability that at least one of k attempts succeeds, and pass ^k is the probability that all k succeed [ 3 ] . The two statistics move in opposite directions as k grows: pass@ k climbs toward certainty, which is the right question when a system can retry until something works or a human picks the…

Read this part in the article →

Learn the underlying idea

An equals sign says that the expression on its left and the expression on its right have the same value under the stated definitions and assumptions.

Open the illustrated equality: what the equals sign claims guide →

Sources cited in the surrounding passage

These citations provide research context; check each source for the exact claim it supports.