fraction

The passage around this formula

That last detail forced a genuine statistical problem into agent-adjacent evaluation for the first time: naively estimating the chance that at least one of k sampled attempts succeeds, by drawing exactly k samples and checking, is a high-variance estimator, especially at small k . Chen and colleagues instead drew a larger fixed pool of n samples per problem, counted the number c that passed, and computed an unbiased estimate of the pass rate at budget k directly from that pool: $\text{pass@}k \;=\; \mathbb{E}_{\text{problems}}\!\left[\,1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}\,\right]$. The term inside the brackets is the probability that a random draw of k items from the n samples contains no passing solution, so one minus that quantity is the probability at least one does. The…

Read this part in the article →