Tagged agent evaluation · show all articles
A tool schema is a contract offered to a caller that cannot ask a follow-up question mid-call. This is a practitioner's guide to writing that contract so it survives retries, versioning, and a model that will eventually call it wrong.
A 95 percent success rate that is slow and expensive can be the worse agent for a job. Scoring only accuracy hides that, and a small set of independent evaluators has started reporting cost and latency as scored dimensions instead of asides.
An agent can behave exactly as trained, a benchmark can be built in good faith, and the reported number can still be wrong, because the failure was never in the run. It was in how the evaluation itself was designed, graded and counted.
Public benchmarks measure someone else's agent on someone else's tasks. A suite that actually protects your system has to be grown from your own incidents, your own pass criteria, and a compute budget you set on purpose.
Two questions decide whether agent evaluation becomes a certified, externally audited discipline by 2035, or stays a signal vendors mostly produce and grade themselves. Four scenarios, each with a falsifier.
SWE-bench, tau-bench, GAIA, WebArena and AgentBench each put an agent in a different room and measure a different thing. Comparing their headline numbers on one axis tells you less than picking the room that matches your job.
Grading a model once meant scoring a string against a reference answer. Grading an agent required borrowing a games lab's tools, discarding them, and rebuilding evaluation from scratch, in a documented, dated sequence of its own.
Measuring an agent is not measuring a model with extra steps. The object under test is a route through an environment, and a single success indicator reads only its last term.
Long tasks fail through state corruption, weak observations, compounding decisions, and false completion—not merely because the model writes a bad line of code.
A coding-agent score is conditional on its tasks, harness, budget, environment, oracle, and date; useful evaluation preserves those conditions and measures operational value.