Equation 10 · Part 1 · The Hardest Unsolved Problems in Open-Weight AI
Symbol s_true
What this part means
the underlying capability the benchmark intends to measure.
Its job in the formula
rue is a part of this expression. Its role is fixed by the surrounding article and by the operations shown in the formula.
Full expression→Symbol s_true→Article meaning
Where the article explains it
Let be an observed benchmark score, the underlying capability the benchmark intends to measure, and c the fraction of test instances the model has effectively memorized, whether verbatim or through a rephrased variant.
The passage around this formula
A simple decomposition makes clear why this is a measurement problem and not merely an inconvenience. Let be an observed benchmark score, the underlying capability the benchmark intends to measure, and c the fraction of test instances the model has effectively memorized, whether verbatim or through a rephrased variant. If a memorized instance is answered correctly with certainty, then approximately
Learn the underlying idea
A subscript is a label attached below a symbol. It often selects a time step, component, category, or member of a sequence.
Open the illustrated subscripts: which member of a family? guide →
See this notation across published equations →
Sources cited in the article section
- [13] Hugging Face Releases Open LLM Leaderboard 2: A Major Upgrade Featuring Tougher Benchmarks, Fairer Scoring, and Enhanced Community Collaboration ↗
- [12] It's Been a Wild Ride, Folks (End of the Open LLM Leaderboard) ↗
- [9] Detecting Pretraining Data from Large Language Models ↗
- [10] Rethinking Benchmark and Contamination for Language Models with Rephrased Samples ↗
- [11] On the Fragility of Benchmark Contamination Detection in Reasoning Models ↗
These citations provide research context; check each source for the exact claim it supports.