Equation 8 · Measuring AI Agent Reliability: What the Evidence Actually Supports
What does this equation mean?
Read the formula alongside the article passage below. Each part has a deeper page with its role in the equation, the supporting passage and nearby citations.
This mathematical expression combines the displayed quantities; its precise role follows from the surrounding article text. Read the equation part by part below; each part has a contextual explanation and a link to its mathematical background.
Read it piece by piece
Symbol n
n is a part of this expression. Its role is fixed by the surrounding article and by the operations shown in the formula.
How to interpret it
Read this expression with the definitions, units, and assumptions supplied by the article.
What the article says around this equation
That combinatorial form is the unbiased estimator: it uses every one of the n observed outcomes rather than discarding some of them, and it has materially lower variance than the intuitive shortcut of raising a single measured success rate to a power. The paper is explicit that the naive alternative is biased, not merely noisier, so the difference is not a technicality — using the wrong formula changes which system looks better [ 2 ] . With n=100 samples per problem, their Codex model solved 70.2% of HumanEval problems by this best-of-many-samples criterion, against 28.8% at pass@1 [ 2 ] . That gap is not a measurement error. It is the honest size of the difference between “can this model…
Read the full surrounding passage
That combinatorial form is the unbiased estimator: it uses every one of the n observed outcomes rather than discarding some of them, and it has materially lower variance than the intuitive shortcut of raising a single measured success rate to a power. The paper is explicit that the naive alternative is biased, not merely noisier, so the difference is not a technicality — using the wrong formula changes which system looks better [ 2 ] . With n=100 samples per problem, their Codex model solved 70.2% of HumanEval problems by this best-of-many-samples criterion, against 28.8% at pass@1 [ 2 ] . That gap is not a measurement error. It is the honest size of the difference between “can this model ever produce a correct solution” and “does its single best guess happen to be correct.”
Sources cited in the surrounding passage
These citations give research context. Read each source to check which claims it supports.
Return to Measuring AI Agent Reliability: What the Evidence Actually Supports