← Mathematical compendium

Published equation contexts

n≈2 p(1−p) (zα/2+zβ)2δ2n \approx \frac{2\,p(1-p)\,(z_{\alpha/2} + z_{\beta})^2}{\delta^2}

Why this formula appears here

Because a “pass” verdict is itself often a noisy measurement, not a fact, it is worth being explicit about how much noise a single run’s outcome carries before treating a change in the pass rate as real. If a task’s true pass rate is p under a baseline configuration and a candidate change is worth detecting only once it shifts that rate by at least δ\delta , the number of repeated trials needed per configuration to detect the shift reliably — at significance level α\alpha and statistical power 1-β\beta — is approximately n≈2 p(1−p) (zα/2+zβ)2δ2n \approx \frac{2\,p(1-p)\,(z_{\alpha/2} + z_{\beta})^2}{\delta^2} . Plugging in a fairly ordinary case — a baseline pass rate of 70 percent, a minimum shift worth caring about of 10 percentage points, a 5 percent…

Read the full article-specific guide →

Read the representative guide

nn

Symbol n

n is a part of this expression. Its role is fixed by the surrounding article and by the operations shown in the formula.

Read this term in its guide →
zα/2z_{\alpha/2}

Symbol z_α/2

z_α/2 occurs above the fraction bar. The numerator is divided by the entire denominator below it.

Read this term in its guide →
zβz_{\beta}

Symbol z_β

z_β occurs above the fraction bar. The numerator is divided by the entire denominator below it.

Read this term in its guide →
δ2\delta^2

Symbol delta^2

delta2a^2 occurs below the fraction bar. The quantity above the bar is divided by this expression; zero is excluded as a denominator.

Read this term in its guide →
2 p(1−p) (zα/2+zβ)22\,p(1-p)\,(z_{\alpha/2} + z_{\beta})^2

Numerator: 2p(1-p)(z_α/2 + z_β)^2

The complete quantity above the fraction bar.

Read this term in its guide →

How to interpret it

With a fixed numerator, increasing a nonzero denominator reduces the fraction. Its accuracy depends on the assumptions and range of use described in the article.

Research cited beside this formula

Learn the mathematics behind it

Published contexts (1)

A symbol can carry a different meaning in another article. Each occurrence keeps its own guide and term definitions.

n≈2 p(1−p) (zα/2+zβ)2δ2.n \approx \frac{2\,p(1-p)\,(z_{\alpha/2} + z_{\beta})^2}{\delta^2} .

Equation 5 · Model Evaluation

Building a Custom Evaluation Suite for a Production Agent

This equation gives an approximation: it relates the quantities while allowing an approximation.

Because a “pass” verdict is itself often a noisy measurement, not a fact, it is worth being explicit about how much noise a single run’s outcome carries before treating a change in the pass rate as real. If a task’s true pass rate is p under a baseline configuration and a candidate change is worth detecting only once it shifts that rate by at least δ\delta , the number of repeated trials needed per configuration to detect the shift reliably — at significance level α\alpha and statistical power 1-β\beta — is approximately n≈2 p(1−p) (zα/2+zβ)2δ2n \approx \frac{2\,p(1-p)\,(z_{\alpha/2} + z_{\beta})^2}{\delta^2} . Plugging in a fairly ordinary case — a baseline pass rate of 70 percent, a minimum shift worth caring about of 10 percentage points, a 5 percent…

Equation guide → · Article →