Tagged evaluation methodology · show all articles
A quantized or pruned model reported to lose less than a point of accuracy has been shown to average well across a benchmark. Whether it kept the one capability a deployment actually needs is a separate question the average is built to hide.
An agent can behave exactly as trained, a benchmark can be built in good faith, and the reported number can still be wrong, because the failure was never in the run. It was in how the evaluation itself was designed, graded and counted.
Grading a model once meant scoring a string against a reference answer. Grading an agent required borrowing a games lab's tools, discarding them, and rebuilding evaluation from scratch, in a documented, dated sequence of its own.
A benchmark score is a coordinate, not a verdict — it names a model snapshot, a harness, a sampling setting, a prompt template, and a date. Drop three of those five and the number stops meaning anything comparable.