← All parts of this equation

Equation 3 · Part 1 · How to Evaluate Codex Beyond SWE-bench and Vendor Scores

Symbol hat p

p^=1n∑i=1nYi\hat p=\frac{1}{n}\sum_{i=1}^{n}Y_i
p^\hat p

What this part means

hat p is part of the quantity the equation computes from the expression on the right.

Its job in the formula

hat p is part of the quantity the equation computes from the expression on the right.

The passage around this formula

Let YiY_i∈\in\{0,1\} indicate whether evaluation task i is resolved. The familiar score p^=1n∑i=1nYi\hat p=\frac{1}{n}\sum_{i=1}^{n}Y_i. is an estimate of success probability only for a distribution represented by those n instances and the evaluated system. The system is not just a model. It includes context retrieval, tools, prompts, retries, execution limits, and candidate selection. The evaluation distribution is not “software engineering.” It is a constructed sample with inclusion criteria, repository mix, languages, issue styles, and executable tests.

Read this part in the article →

Learn the underlying idea

A variable is a named place for a value. Its letter is a local label: x can mean position in one formula and a data point in another.

Open the illustrated variables: a letter stands for a value guide →

See this notation across published equations →

The article lists its research sources here.