← All parts of this equation

Equation 3 · Part 6 · How to Evaluate Codex Beyond SWE-bench and Vendor Scores

fraction

p^=1n∑i=1nYi\hat p=\frac{1}{n}\sum_{i=1}^{n}Y_i
fraction

What this part means

Divide the expression above the line by the one below it.

Its job in the formula

The expression above the fraction bar is divided by the complete expression below it. The denominator must not be zero.

The passage around this formula

Let YiY_i∈\in\{0,1\} indicate whether evaluation task i is resolved. The familiar score p^=1n∑i=1nYi\hat p=\frac{1}{n}\sum_{i=1}^{n}Y_i. is an estimate of success probability only for a distribution represented by those n instances and the evaluated system. The system is not just a model. It includes context retrieval, tools, prompts, retries, execution limits, and candidate selection. The evaluation distribution is not “software engineering.” It is a constructed sample with inclusion criteria, repository mix, languages, issue styles, and executable tests.

Read this part in the article →

Learn the underlying idea

A fraction a/b means a divided by b. The top number is the numerator; the bottom number is the denominator, and it cannot be zero.

Open the illustrated fractions: division written vertically guide →

The article lists its research sources here.