Tagged benchmarks · show all articles
An agent can behave exactly as trained, a benchmark can be built in good faith, and the reported number can still be wrong, because the failure was never in the run. It was in how the evaluation itself was designed, graded and counted.
Two questions decide whether agent evaluation becomes a certified, externally audited discipline by 2035, or stays a signal vendors mostly produce and grade themselves. Four scenarios, each with a falsifier.
SWE-bench, tau-bench, GAIA, WebArena and AgentBench each put an agent in a different room and measure a different thing. Comparing their headline numbers on one axis tells you less than picking the room that matches your job.
Grading a model once meant scoring a string against a reference answer. Grading an agent required borrowing a games lab's tools, discarding them, and rebuilding evaluation from scratch, in a documented, dated sequence of its own.
A single successful run shows a system can succeed once. Pass@k, pass^k, and repeated-trial statistics are how anyone finds out whether it succeeds reliably — and task length changes the answer.
Leaderboards rank one model against another. They were never built to show what several of them do the same way, and ten failures recur across vendors no matter who is winning this quarter's chart.
A benchmark score is a coordinate, not a verdict — it names a model snapshot, a harness, a sampling setting, a prompt template, and a date. Drop three of those five and the number stops meaning anything comparable.
Measuring an agent is not measuring a model with extra steps. The object under test is a route through an environment, and a single success indicator reads only its last term.
A benchmark result is an assay: it depends on how the sample was taken, what impurity was already in the reference, and how many times the balance was read.