Tagged reproducibility · show all articles
A score like SWE-bench 74 percent looks like a fact about a model. It is really a fact about a model, a harness, a task subset, and a date bundled together, and mixing those up is the most common way to misread a benchmark.
A 95 percent success rate that is slow and expensive can be the worse agent for a job. Scoring only accuracy hides that, and a small set of independent evaluators has started reporting cost and latency as scored dimensions instead of asides.
An agent can behave exactly as trained, a benchmark can be built in good faith, and the reported number can still be wrong, because the failure was never in the run. It was in how the evaluation itself was designed, graded and counted.
A benchmark score is a coordinate, not a verdict — it names a model snapshot, a harness, a sampling setting, a prompt template, and a date. Drop three of those five and the number stops meaning anything comparable.
Openness in model releases is not a binary and never was. It is a bundle of separately granted permissions over five separable artefacts, and the word on the announcement page rarely tells you which ones you hold.
Measuring an agent is not measuring a model with extra steps. The object under test is a route through an environment, and a single success indicator reads only its last term.