Tagged benchmark contamination · show all articles
A single-turn quiz question can be memorized whole. An agentic benchmark is an entire environment — a repository, a test suite, a live shell — and each of those can leak, get gamed, or get recognized as a test in ways a static question bank cannot.
A benchmark score is not a fact about a model; it is a fact about a harness, a threshold, and a moment in time. Here is what actually goes into an Anthropic capability evaluation, who checks it from outside, and why a single leaderboard number still misleads.
A retrieval score and an answer-quality score are counted by different instruments, on different objects, at different points in the pipeline. Either one can look excellent while the other is quietly broken, and most reported RAG numbers do not say which.