Filtered to Model Evaluation · show all articles
A 95 percent success rate that is slow and expensive can be the worse agent for a job. Scoring only accuracy hides that, and a small set of independent evaluators has started reporting cost and latency as scored dimensions instead of asides.
An agent can behave exactly as trained, a benchmark can be built in good faith, and the reported number can still be wrong, because the failure was never in the run. It was in how the evaluation itself was designed, graded and counted.
A single-turn quiz question can be memorized whole. An agentic benchmark is an entire environment — a repository, a test suite, a live shell — and each of those can leak, get gamed, or get recognized as a test in ways a static question bank cannot.
Public benchmarks measure someone else's agent on someone else's tasks. A suite that actually protects your system has to be grown from your own incidents, your own pass criteria, and a compute budget you set on purpose.
Two questions decide whether agent evaluation becomes a certified, externally audited discipline by 2035, or stays a signal vendors mostly produce and grade themselves. Four scenarios, each with a falsifier.
SWE-bench, tau-bench, GAIA, WebArena and AgentBench each put an agent in a different room and measure a different thing. Comparing their headline numbers on one axis tells you less than picking the room that matches your job.
Grading a model once meant scoring a string against a reference answer. Grading an agent required borrowing a games lab's tools, discarding them, and rebuilding evaluation from scratch, in a documented, dated sequence of its own.
Two variables decide the frontier-model contest by 2035: how many labs stay near the frontier, and whether the public signal of leadership is benchmark score or deployment outcome. Four scenarios follow, each with a falsifier.
Leaderboards rank one model against another. They were never built to show what several of them do the same way, and ten failures recur across vendors no matter who is winning this quarter's chart.
AIME, GPQA and FrontierMath test different things by different rules. Reasoning effort and extended thinking are documented settings, not fixed traits, and a right answer does not certify the reasoning behind it.