Equation 9 · Measuring AI Agent Reliability: What the Evidence Actually Supports
What does this equation mean?
Read the formula alongside the article passage below. Each part has a deeper page with its role in the equation, the supporting passage and nearby citations.
This equation states an equality: the expressions on both sides have the same value under the article’s assumptions. Read the equation part by part below; each part has a contextual explanation and a link to its mathematical background.
Read it piece by piece
Symbol n
n is part of the quantity the equation computes from the expression on the right.
=
The expressions on both sides represent the same quantity under the stated assumptions.
See an illustrated explanation →How to interpret it
Read it with the definitions, units, and assumptions supplied by the article.
What the article says around this equation
That combinatorial form is the unbiased estimator: it uses every one of the n observed outcomes rather than discarding some of them, and it has materially lower variance than the intuitive shortcut of raising a single measured success rate to a power. The paper is explicit that the naive alternative is biased, not merely noisier, so the difference is not a technicality — using the wrong formula changes which system looks better [ 2 ] . With n=100 samples per problem, their Codex model solved 70.2% of HumanEval problems by this best-of-many-samples criterion, against 28.8% at pass@1 [ 2 ] . That gap is not a measurement error. It is the honest size of the difference between “can this model…
Read the full surrounding passage
That combinatorial form is the unbiased estimator: it uses every one of the n observed outcomes rather than discarding some of them, and it has materially lower variance than the intuitive shortcut of raising a single measured success rate to a power. The paper is explicit that the naive alternative is biased, not merely noisier, so the difference is not a technicality — using the wrong formula changes which system looks better [ 2 ] . With n=100 samples per problem, their Codex model solved 70.2% of HumanEval problems by this best-of-many-samples criterion, against 28.8% at pass@1 [ 2 ] . That gap is not a measurement error. It is the honest size of the difference between “can this model ever produce a correct solution” and “does its single best guess happen to be correct.”
Sources cited in the surrounding passage
These citations give research context. Read each source to check which claims it supports.
Return to Measuring AI Agent Reliability: What the Evidence Actually Supports