Equation 8 · How to Evaluate Codex Beyond SWE-bench and Vendor Scores
What does this equation mean?
Read the formula alongside the article passage below. Each part has a deeper page with its role in the equation, the supporting passage and nearby citations.
This equation states an equality: the expressions on both sides have the same value under the article’s assumptions. Read the equation part by part below; each part has a contextual explanation and a link to its mathematical background.
Read it piece by piece
Symbol B_j
is part of the quantity the equation computes from the expression on the right.
Symbol N_tokens
okens is an input to the expression that computes the quantity on the left.
Symbol N_turns
urns is an input to the expression that computes the quantity on the left.
Symbol N_tools
ools is an input to the expression that computes the quantity on the left.
Symbol t_wall
all is an input to the expression that computes the quantity on the left.
Symbol C_compute
ompute is an input to the expression that computes the quantity on the left.
Symbol k_attempts
ttempts is an input to the expression that computes the quantity on the left.
=
The expressions on both sides represent the same quantity under the stated assumptions.
See an illustrated explanation →subscript
The lower label selects a particular version, component, or indexed member of the quantity. For example, x₀ and xₜ can be values at different positions.
How to interpret it
Read it with the definitions, units, and assumptions supplied by the article.
What the article says around this equation
For system j , report a resource vector . A resolution curve p(B) is more informative than one point. It reveals marginal returns: whether twice the budget buys substantial success, tiny gains, or merely longer trajectories. It also lets organizations choose configurations on a cost-quality frontier.
For background, read the article’s source list.
Return to How to Evaluate Codex Beyond SWE-bench and Vendor Scores