Equation 7 · Part 9 · Measuring AI Agent Reliability: What the Evidence Actually Supports
Numerator: binomn-ck
What this part means
The complete quantity above the fraction bar.
Its job in the formula
binomn-ck occurs above the fraction bar. The numerator is divided by the entire denominator below it.
Full expression→Numerator: binomn-ck→Article meaning
The passage around this formula
Start with the metric that made repeated sampling a standard practice. When OpenAI’s Codex team evaluated a code-generating model against the HumanEval benchmark, they needed a way to score the strategy of drawing several candidate solutions from the model and keeping the best one. The obvious approach — generate exactly k samples per problem and check whether any of them pass — has an undesirable property: it is a valid estimate but a high-variance one, since it throws away information every time you happen to generate more or fewer than k samples. Their fix was to over-sample: draw n total samples per task, observe how many of them, c , actually pass, and then compute the exact probability…
Learn the underlying idea
A fraction a/b means a divided by b. The top number is the numerator; the bottom number is the denominator, and it cannot be zero.
Open the illustrated fractions: division written vertically guide →
Sources cited in the surrounding passage
These citations provide research context; check each source for the exact claim it supports.