← All parts of this equation

Equation 3 · Part 4 · How to Evaluate Codex Beyond SWE-bench and Vendor Scores

Symbol Y_i

p^=1n∑i=1nYi\hat p=\frac{1}{n}\sum_{i=1}^{n}Y_i
YiY_i

What this part means

an estimate of success probability only for a distribution represented by those n instances and the evaluated system.

Its job in the formula

YiY_i is an input to the expression that computes the quantity on the left.

Where the article explains it

The familiar score p^\hat p=1n\frac{1}{n}∑i=1n\sum_{i=1}^{n}YiY_i is an estimate of success probability only for a distribution represented by those n instances and the evaluated system.

The passage around this formula

Let YiY_i∈\in\{0,1\} indicate whether evaluation task i is resolved. The familiar score p^=1n∑i=1nYi\hat p=\frac{1}{n}\sum_{i=1}^{n}Y_i. is an estimate of success probability only for a distribution represented by those n instances and the evaluated system. The system is not just a model. It includes context retrieval, tools, prompts, retries, execution limits, and candidate selection. The evaluation distribution is not “software engineering.” It is a constructed sample with inclusion criteria, repository mix, languages, issue styles, and executable tests.

Read this part in the article →

Learn the underlying idea

A subscript is a label attached below a symbol. It often selects a time step, component, category, or member of a sequence.

Open the illustrated subscripts: which member of a family? guide →

See this notation across published equations →

The article lists its research sources here.