What does this equation mean?

Start with the metric that made repeated sampling a standard practice. When OpenAI’s Codex team evaluated a code-generating model against the HumanEval benchmark, they needed a way to score the strategy of drawing several candidate solutions from the model and keeping the best one. The obvious approach — generate exactly k samples per problem and check whether any of them pass — has an undesirable property: it is a valid estimate but a high-variance one, since it throws away information every time you happen to generate more or fewer than k samples. Their fix was to over-sample: draw n total samples per task, observe how many of them, c , actually pass, and then compute the exact probability that a random draw of k items from those n would contain at least one success [ 2 ] . That combinatorial form is the unbiased estimator: it uses every one of the n observed outcomes rather than discarding some of them, and it has materially lower variance than the intuitive shortcut of raising a single measured success rate to a power. The paper is explicit that the naive alternative is biased, not merely noisier, so the difference is not a technicality — using the wrong formula changes which system looks better [ 2 ] . With n=100 samples per problem, their Codex model solved 70.2% of HumanEval problems by this best-of-many-samples criterion, against 28.8% at pass@1 [ 2 ] . That gap is not a measurement error. It is the honest size of the difference between “can this model ever produce a correct solution” and “does its single best guess happen to be correct.”