Filtered to Model Evaluation · show all articles
System cards show resolve rates climbing fast on SWE-bench Verified and Terminal-Bench, but scaffold, effort, and test-oracle strength move a score as much as weights do. The one reproduction holding harness fixed separates OpenAI and Claude by under a point.
A benchmark score is a coordinate, not a verdict — it names a model snapshot, a harness, a sampling setting, a prompt template, and a date. Drop three of those five and the number stops meaning anything comparable.
OpenAI, Anthropic, and Google each publish a token count for how much a model can hold. Independent long-context benchmarks measure a different quantity: how much of it the model actually uses correctly, and the two numbers are routinely far apart.
OpenAI, Anthropic, and Google publish input and output prices in the same unit, dollars per million tokens. Three specific ratios genuinely converge across all three rate cards. The token underneath the ratio does not.
A coding-agent score is conditional on its tasks, harness, budget, environment, oracle, and date; useful evaluation preserves those conditions and measures operational value.