← All parts of this equation

Equation 3 · Part 9 · How to Evaluate Codex Beyond SWE-bench and Vendor Scores

Numerator: 1

p^=1n∑i=1nYi\hat p=\frac{1}{n}\sum_{i=1}^{n}Y_i
11

What this part means

The complete quantity above the fraction bar.

Its job in the formula

1 occurs above the fraction bar. The numerator is divided by the entire denominator below it.

The passage around this formula

Let YiY_i∈\in\{0,1\} indicate whether evaluation task i is resolved. The familiar score p^=1n∑i=1nYi\hat p=\frac{1}{n}\sum_{i=1}^{n}Y_i. is an estimate of success probability only for a distribution represented by those n instances and the evaluated system. The system is not just a model. It includes context retrieval, tools, prompts, retries, execution limits, and candidate selection. The evaluation distribution is not “software engineering.” It is a constructed sample with inclusion criteria, repository mix, languages, issue styles, and executable tests.

Read this part in the article →

Learn the underlying idea

A fraction a/b means a divided by b. The top number is the numerator; the bottom number is the denominator, and it cannot be zero.

Open the illustrated fractions: division written vertically guide →

The article lists its research sources here.