← Mathematical compendium

Published equation contexts

s^=1n∑i=1n1 ⁣[ success on xi ],xi∼Dbench\hat{s} = \frac{1}{n}\sum_{i=1}^{n} \mathbf{1}\!\left[\,\text{success on } x_i\,\right], \qquad x_i \sim \mathcal{D}_{\mathrm{bench}}

Why this formula appears here

Reframe the object. A benchmark score s^\hat{s} is an estimator of a population quantity s — the model’s success rate over some distribution of tasks D\mathcal{D} that somebody hopes resembles the work you actually have. Every property that makes an estimator trustworthy applies: s^=1n∑i=1n1 ⁣[ success on xi ],xi∼Dbench\hat{s} = \frac{1}{n}\sum_{i=1}^{n} \mathbf{1}\!\left[\,\text{success on } x_i\,\right], \qquad x_i \sim \mathcal{D}_{\mathrm{bench}} . The number is only as good as three things: whether Dbench\mathcal{D}_{\mathrm{bench}} resembles D\mathcal{D} , whether the xix_i are genuinely held out, and whether the indicator is measured with enough repetition to characterise its spread. All three fail routinely, and they fail in different directions.

Read the full article-specific guide →

Read the representative guide

nn

Symbol n

n occurs below the fraction bar. The quantity above the bar is divided by this expression; zero is excluded as a denominator.

Read this term in its guide →
ii

Symbol i

i appears in the bound of this sum. The bound states where the repeated operation starts, ends, or which values it includes.

Read this term in its guide →
Dbench\mathcal{D}_{\mathrm{bench}}

Symbol D_bench

DbD_bench is an input to the expression that computes the quantity on the left.

Read this term in its guide →
i=1i=1

Starting index or lower bound: i=1

This label says where the repeated addition, multiplication, or accumulation starts. Read its value or condition together with the article’s description of the index.

Read this term in its guide →
nn

Ending index or upper bound: n

This label says where the repeated addition, multiplication, or accumulation stops. It sets the last term or end of the range.

Read this term in its guide →

How to interpret it

With a fixed numerator, increasing a nonzero denominator reduces the fraction. Read it with the definitions, units, and assumptions supplied by the article.

Learn the mathematics behind it

Published contexts (1)

A symbol can carry a different meaning in another article. Each occurrence keeps its own guide and term definitions.

s^=1n∑i=1n1 ⁣[ success on xi ],xi∼Dbench.\hat{s} = \frac{1}{n}\sum_{i=1}^{n} \mathbf{1}\!\left[\,\text{success on } x_i\,\right], \qquad x_i \sim \mathcal{D}_{\mathrm{bench}} .

Equation 4 · Foundation Models

Measuring Frontier Models: Contamination, Variance, and What a Score Can Support

This equation states an equality: the expressions on both sides have the same value under the article’s assumptions.

Reframe the object. A benchmark score s^\hat{s} is an estimator of a population quantity s — the model’s success rate over some distribution of tasks D\mathcal{D} that somebody hopes resembles the work you actually have. Every property that makes an estimator trustworthy applies: s^=1n∑i=1n1 ⁣[ success on xi ],xi∼Dbench\hat{s} = \frac{1}{n}\sum_{i=1}^{n} \mathbf{1}\!\left[\,\text{success on } x_i\,\right], \qquad x_i \sim \mathcal{D}_{\mathrm{bench}} . The number is only as good as three things: whether Dbench\mathcal{D}_{\mathrm{bench}} resembles D\mathcal{D} , whether the xix_i are genuinely held out, and whether the indicator is measured with enough repetition to characterise its spread. All three fail routinely, and they fail in different directions.

Equation guide → · Article →