← Mathematical compendium

Published equation contexts

pass@k  =  Eproblems ⁣[ 1−(n−ck)(nk) ]\text{pass@}k \;=\; \mathbb{E}_{\text{problems}}\!\left[\,1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}\,\right]

Why this formula appears here

That last detail forced a genuine statistical problem into agent-adjacent evaluation for the first time: naively estimating the chance that at least one of k sampled attempts succeeds, by drawing exactly k samples and checking, is a high-variance estimator, especially at small k . Chen and colleagues instead drew a larger fixed pool of n samples per problem, counted the number c that passed, and computed an unbiased estimate of the pass rate at budget k directly from that pool: pass@k  =  Eproblems ⁣[ 1−(n−ck)(nk) ]\text{pass@}k \;=\; \mathbb{E}_{\text{problems}}\!\left[\,1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}\,\right]. The term inside the brackets is the probability that a random draw of k items from the n samples contains no passing solution, so one minus that quantity is the probability at least one does. The…

Read the full article-specific guide →

Read the representative guide

Eproblems\mathbb{E}_{\text{problems}}

Symbol E_problems

EpE_problems appears inside an expected value, so its contribution is averaged under the distribution or condition shown by that operator.

Read this term in its guide →
(nk)\binom{n}{k}

Denominator: binomnk

The complete quantity below the fraction bar; it must be nonzero for this division.

Read this term in its guide →

How to interpret it

With a fixed numerator, increasing a nonzero denominator reduces the fraction. Read it with the definitions, units, and assumptions supplied by the article.

Research cited beside this formula

Learn the mathematics behind it

Published contexts (1)

A symbol can carry a different meaning in another article. Each occurrence keeps its own guide and term definitions.

pass@k  =  Eproblems ⁣[ 1−(n−ck)(nk) ].\text{pass@}k \;=\; \mathbb{E}_{\text{problems}}\!\left[\,1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}\,\right].

Equation 9 · Model Evaluation

A History of How We Learned to Evaluate AI Agents

This equation states an equality: the expressions on both sides have the same value under the article’s assumptions.

That last detail forced a genuine statistical problem into agent-adjacent evaluation for the first time: naively estimating the chance that at least one of k sampled attempts succeeds, by drawing exactly k samples and checking, is a high-variance estimator, especially at small k . Chen and colleagues instead drew a larger fixed pool of n samples per problem, counted the number c that passed, and computed an unbiased estimate of the pass rate at budget k directly from that pool: pass@k  =  Eproblems ⁣[ 1−(n−ck)(nk) ]\text{pass@}k \;=\; \mathbb{E}_{\text{problems}}\!\left[\,1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}\,\right]. The term inside the brackets is the probability that a random draw of k items from the n samples contains no passing solution, so one minus that quantity is the probability at least one does. The…

Meanings in this article

  • kk: the especially at small.
  • nn: the number of samples.
Equation guide → · Article →