Tagged SWE-bench · show all articles
A single-turn quiz question can be memorized whole. An agentic benchmark is an entire environment — a repository, a test suite, a live shell — and each of those can leak, get gamed, or get recognized as a test in ways a static question bank cannot.
SWE-bench, tau-bench, GAIA, WebArena and AgentBench each put an agent in a different room and measure a different thing. Comparing their headline numbers on one axis tells you less than picking the room that matches your job.
Long tasks fail through state corruption, weak observations, compounding decisions, and false completion—not merely because the model writes a bad line of code.
A coding-agent score is conditional on its tasks, harness, budget, environment, oracle, and date; useful evaluation preserves those conditions and measures operational value.