Tagged reliability · show all articles
Two questions decide whether agent evaluation becomes a certified, externally audited discipline by 2035, or stays a signal vendors mostly produce and grade themselves. Four scenarios, each with a falsifier.
Grading a model once meant scoring a string against a reference answer. Grading an agent required borrowing a games lab's tools, discarding them, and rebuilding evaluation from scratch, in a documented, dated sequence of its own.
A single successful run shows a system can succeed once. Pass@k, pass^k, and repeated-trial statistics are how anyone finds out whether it succeeds reliably — and task length changes the answer.
An agent can do exactly what its tool interfaces let it propose, no more. The schema, the granularity, the retry semantics and the error surface are not plumbing around the intelligence — they are the boundary of it.
Once a model no longer fits on one device, the interconnect stops being plumbing and becomes the design surface. Every way of splitting a model generates its own traffic, and the fabric decides which splits are affordable.
Measuring an agent is not measuring a model with extra steps. The object under test is a route through an environment, and a single success indicator reads only its last term.
A practical guide to the decisions that determine whether an application on a frontier model stays correct and affordable as the model underneath it changes without warning.
The dangerous failures are not the ones that throw an exception. They are the ones where the system keeps returning well-formed output that is quietly wrong, and nothing upstream notices.
A coding-agent score is conditional on its tasks, harness, budget, environment, oracle, and date; useful evaluation preserves those conditions and measures operational value.