← Mathematical compendium

Published equation contexts

sobs=(1−c) strue+cs_{\text{obs}} = (1 - c)\, s_{\text{true}} + c

Why this formula appears here

A simple decomposition makes clear why this is a measurement problem and not merely an inconvenience. Let sobss_{\text{obs}} be an observed benchmark score, strues_{\text{true}} the underlying capability the benchmark intends to measure, and c the fraction of test instances the model has effectively memorized, whether verbatim or through a rephrased variant. If a memorized instance is answered correctly with certainty, then approximately sobs=(1−c) strue+cs_{\text{obs}} = (1 - c)\, s_{\text{true}} + c. Every published leaderboard number implicitly assumes c ≈\approx 0 . The methods above show that assumption is not something the field can currently verify for a given open-weight release, only something it can occasionally catch violated in…

Read the full article-specific guide →

Read the representative guide

sobss_{\text{obs}}

Symbol s_obs

sos_obs is part of the quantity the equation computes from the expression on the right.

Read this term in its guide →

How to interpret it

Read it with the definitions, units, and assumptions supplied by the article.

Research cited beside this formula

Learn the mathematics behind it

Published contexts (1)

A symbol can carry a different meaning in another article. Each occurrence keeps its own guide and term definitions.

sobs=(1−c) strue+c.s_{\text{obs}} = (1 - c)\, s_{\text{true}} + c.

Equation 12 · Open Models

The Hardest Unsolved Problems in Open-Weight AI

This equation states an equality: the expressions on both sides have the same value under the article’s assumptions.

A simple decomposition makes clear why this is a measurement problem and not merely an inconvenience. Let sobss_{\text{obs}} be an observed benchmark score, strues_{\text{true}} the underlying capability the benchmark intends to measure, and c the fraction of test instances the model has effectively memorized, whether verbatim or through a rephrased variant. If a memorized instance is answered correctly with certainty, then approximately sobs=(1−c) strue+cs_{\text{obs}} = (1 - c)\, s_{\text{true}} + c. Every published leaderboard number implicitly assumes c ≈\approx 0 . The methods above show that assumption is not something the field can currently verify for a given open-weight release, only something it can occasionally catch violated in…

Meanings in this article

Equation guide → · Article →