← All parts of this equation

Equation 9 · Part 2 · A History of How We Learned to Evaluate AI Agents

Symbol E_problems

pass@k  =  Eproblems ⁣[ 1−(n−ck)(nk) ].\text{pass@}k \;=\; \mathbb{E}_{\text{problems}}\!\left[\,1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}\,\right].
Eproblems\mathbb{E}_{\text{problems}}

What this part means

EpE_problems appears inside an expected value, so its contribution is averaged under the distribution or condition shown by that operator.

Its job in the formula

EpE_problems appears inside an expected value, so its contribution is averaged under the distribution or condition shown by that operator.

The passage around this formula

That last detail forced a genuine statistical problem into agent-adjacent evaluation for the first time: naively estimating the chance that at least one of k sampled attempts succeeds, by drawing exactly k samples and checking, is a high-variance estimator, especially at small k . Chen and colleagues instead drew a larger fixed pool of n samples per problem, counted the number c that passed, and computed an unbiased estimate of the pass rate at budget k directly from that pool: pass@k  =  Eproblems ⁣[ 1−(n−ck)(nk) ]\text{pass@}k \;=\; \mathbb{E}_{\text{problems}}\!\left[\,1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}\,\right]. The term inside the brackets is the probability that a random draw of k items from the n samples contains no passing solution, so one minus that quantity is the probability at least one does. The…

Read this part in the article →

Learn the underlying idea

A subscript is a label attached below a symbol. It often selects a time step, component, category, or member of a sequence.

Open the illustrated subscripts: which member of a family? guide →

See this notation across published equations →

Sources cited in the article section

These citations provide research context; check each source for the exact claim it supports.