Equation 7 · Part 6 · Measuring AI Agent Reliability: What the Evidence Actually Supports
fraction
fraction
What this part means
Divide the expression above the line by the one below it.
Its job in the formula
The expression above the fraction bar is divided by the complete expression below it. The denominator must not be zero.
Full expression→fraction→Article meaning
The passage around this formula
Start with the metric that made repeated sampling a standard practice. When OpenAI’s Codex team evaluated a code-generating model against the HumanEval benchmark, they needed a way to score the strategy of drawing several candidate solutions from the model and keeping the best one. The obvious approach — generate exactly k samples per problem and check whether any of them pass — has an undesirable property: it is a valid estimate but a high-variance one, since it throws away information every time you happen to generate more or fewer than k samples. Their fix was to over-sample: draw n total samples per task, observe how many of them, c , actually pass, and then compute the exact probability…
Learn the underlying idea
A fraction a/b means a divided by b. The top number is the numerator; the bottom number is the denominator, and it cannot be zero.
Open the illustrated fractions: division written vertically guide →
Sources cited in the surrounding passage
These citations provide research context; check each source for the exact claim it supports.