Tagged model evaluation · show all articles
A circuit that looks real on your screen has to survive a matched control, a blind check, and someone else's attempt to break it before it earns a place in an audit report. This is how the practitioners who take that seriously actually work.
A 95 percent success rate that is slow and expensive can be the worse agent for a job. Scoring only accuracy hides that, and a small set of independent evaluators has started reporting cost and latency as scored dimensions instead of asides.
Two variables decide the frontier-model contest by 2035: how many labs stay near the frontier, and whether the public signal of leadership is benchmark score or deployment outcome. Four scenarios follow, each with a falsifier.
Leaderboards rank one model against another. They were never built to show what several of them do the same way, and ten failures recur across vendors no matter who is winning this quarter's chart.
System cards show resolve rates climbing fast on SWE-bench Verified and Terminal-Bench, but scaffold, effort, and test-oracle strength move a score as much as weights do. The one reproduction holding harness fixed separates OpenAI and Claude by under a point.
A benchmark score is a coordinate, not a verdict — it names a model snapshot, a harness, a sampling setting, a prompt template, and a date. Drop three of those five and the number stops meaning anything comparable.
OpenAI, Anthropic, and Google publish input and output prices in the same unit, dollars per million tokens. Three specific ratios genuinely converge across all three rate cards. The token underneath the ratio does not.
Claude carries more defensive machinery than most deployed models — classifiers, citations, constitutional training. Its failures surface at the seams of that machinery, and Anthropic's own records are usually the first place they are documented.
Codex is best understood as a changing stack of models, tools, execution boundaries, state, and review mechanisms—not as a single code-generating model.