Tagged Anthropic · show all articles
A benchmark score looks like a measurement. Comparing two frontier models honestly requires a shared unit, a clean test set, and a matched operating point, and no laboratory has all three yet.
A subagent is not a persona or a mode. It is a separate context window, its own system prompt, and a narrow channel back to the conversation that launched it — a real mechanism with documented costs and documented rules for when paying them is worth it.
Long before the term existed, researchers tried to read meaning into a trained network's units. This is mechanistic interpretability's dated history: a naming paper, a landmark result, and three labs converging on one tool within months of each other.
AIME, GPQA and FrontierMath test different things by different rules. Reasoning effort and extended thinking are documented settings, not fixed traits, and a right answer does not certify the reasoning behind it.
System cards show resolve rates climbing fast on SWE-bench Verified and Terminal-Bench, but scaffold, effort, and test-oracle strength move a score as much as weights do. The one reproduction holding harness fixed separates OpenAI and Claude by under a point.
OpenAI, Anthropic, and Google each publish a token count for how much a model can hold. Independent long-context benchmarks measure a different quantity: how much of it the model actually uses correctly, and the two numbers are routinely far apart.
OpenAI, Anthropic, and Google publish input and output prices in the same unit, dollars per million tokens. Three specific ratios genuinely converge across all three rate cards. The token underneath the ratio does not.
Every frontier-model vendor sells an enterprise tier with the same badge row — SOC 2, ISO, HIPAA, sometimes FedRAMP. The badges are real. What they cover, and what they leave for the customer to build, is a separate and more specific engineering question.
A benchmark score is not a fact about a model; it is a fact about a harness, a threshold, and a moment in time. Here is what actually goes into an Anthropic capability evaluation, who checks it from outside, and why a single leaderboard number still misleads.
Every refusal passes through a trained policy, a classifier, account monitoring, and a capability-gated release process — four thresholds, each trading missed misuse against blocked legitimate use, documented in Anthropic's own safeguards and RSP filings.