OpenAI and Claude on Agentic Coding: What the Independent Evidence Actually Shows

A narrower question than “which agent wins”

A companion piece in this series compares OpenAI Codex and Claude Code as software-engineering systems: sandboxes, permission grammars, hooks, subagents, and the rest of the control plane a team actually buys. That is a real and separate question, and answering it well means holding the underlying model constant wherever possible so that product differences do not get credited to, or blamed on, the weights.

This article does the opposite. It fixes the products and asks about the models: what does independent, reproducible evidence say about how well OpenAI’s and Anthropic’s frontier models perform specifically on agentic coding, as measured by benchmarks purpose-built for that — SWE-bench Verified and Terminal-Bench chief among them? “Independent” is doing real work in that sentence. A system card is a primary source about what a vendor measured and how, but it is not a third-party check on that measurement. The distinction turns out to matter more than which vendor is ahead on any single leaderboard snapshot, because the leaderboards themselves do not agree with each other about which vendor that is.

What the two benchmarks actually test

SWE-bench Verified is a 500-instance, human-validated subset of the original SWE-bench dataset, built with OpenAI’s collaboration specifically to fix earlier problems with under-specified issues, incorrect grading, and overly narrow tests ^[9]. A system under test reads a real GitHub issue against a real repository snapshot, produces a patch, and is scored by whether that patch makes a hidden test suite pass. It measures issue-resolution on a fixed, well-characterized slice of open-source Python work — not general coding ability, not long-horizon project work, and not anything outside those twelve repositories’ idioms.

Terminal-Bench is a different instrument aimed at a different failure mode. Its current 2.1 release scores an agent across 89 curated tasks spanning software engineering, system administration, data processing, model training, and security, each requiring real interaction with a live terminal environment and verified by a programmatic checker rather than a single hidden test ^[11]. Examples on the public leaderboard include recovering a corrupted SQLite database and cross-compiling a game for an unusual target architecture ^[11]. Terminal-Bench explicitly scores the agent-plus-model pair, not an isolated model, which becomes important below.

Neither benchmark is a general intelligence test, and neither was designed to produce a single cross-vendor number. Both were designed to be hard to game by memorization and easy to grade automatically. Whether they still are either of those things, a year and several model generations later, is most of what follows.

The vendor numbers, in the order they were published

Laid end to end, the self-reported trajectory looks like fast, steady progress. OpenAI’s GPT-5.1-Codex-Max system card, published November 18, 2025, describes a model natively trained to operate across multiple context windows through a compaction mechanism; independent reporting on that card puts its SWE-bench Verified score at 77.9% and its Terminal-Bench 2.0 score at 58.1% ^{[1, 2]}. Six days later, Anthropic announced Claude Opus 4.5, describing it as state-of-the-art on real-world software engineering, evaluated with a 64,000-token thinking budget and results averaged over five independent trials; at its highest effort setting the card reports it exceeding Claude Sonnet 4.5 by 4.3 percentage points, while at medium effort it matches Sonnet 4.5’s best score using 76% fewer output tokens ^[3]. Contemporary reporting put the headline SWE-bench Verified figure at 80.9% ^[4].

By July 24, 2026, Anthropic had launched Claude Opus 5, describing it as coming close to the frontier intelligence of its top-tier Fable 5 model at roughly half the price and as new state-of-the-art on its own Frontier-Bench and GDPval evaluations ^[5]. Launch-day reporting on the accompanying system card cited a SWE-bench Verified score of 96.0%, alongside 79.2% on the harder SWE-bench Pro variant ^[6]. A second outlet’s independent write-up of the same launch quotes “around 97%” for the same model on the same benchmark, in a sentence that itself argues differences of a point or two between frontier models are noise, not a meaningful gap ^[7]. Two publications reading the same vendor’s own disclosure landed on different numbers before any cross-vendor question was even asked.

OpenAI’s own most recent preview card, for GPT-5.6, published June 26, 2026, treats the picture differently still: it lists SWE-bench Verified only as one component task inside an internal controllability evaluation suite, without publishing a headline resolve-rate for it in that document at all ^[8]. Anthropic headlines a SWE-bench Verified percentage as a marketing number; OpenAI’s most recent preview card does not use it that way. That asymmetry alone should make a reader suspicious of any table that lists both companies’ “SWE-bench score” side by side as though it were collected the same way.

Same name, different measurement: the scaffold confound

The deeper problem predates any of the specific numbers above. A peer-reviewed audit of SWE-bench’s own leaderboards, accepted at ICSE-SEIP 2026, examined every submission to the Lite and Verified boards and found that “submissions are not required to disclose how the reported results were obtained” — most entries are self-reported by their authors rather than independently re-executed by a third party ^[12]. The same paper documents that identical scaffolds paired with different underlying models are catalogued as unrelated leaderboard rows, and vice versa, which means a leaderboard’s headline ranking conflates two separate design choices — which model, and which harness around it — into one indistinguishable number ^[12].

The size of that confound is measurable, and it is not small. An open-source scaffold called Live-SWE-agent, run against Claude Opus 4.5, scored 79.2% on SWE-bench Verified — 1.7 percentage points behind Anthropic’s own manually engineered internal scaffold running the identical model ^[17]. Scale AI’s own analysis, reported independently in June 2026, found something more extreme on the harder SWE-bench Pro set: three different agent harnesses run against the same Claude Opus 4.5 model produced scores from 50.2% to 55.4%, a 5.2-point spread from harness choice alone, and the same source attributes to Scale AI a general finding that scaffold changes — context management, retry logic, tool integration, error-feedback formatting — can move a score by ten to twenty points without the model changing at all ^[18].

That range is worth writing as a simple decomposition, because it clarifies exactly what a published score is actually a measurement of:

where $\theta$ is the model’s weights, $H$ is the harness or scaffold wrapped around it, $e$ is the reasoning-effort or thinking-budget setting, $\tau$ is the strength of the test oracle used to grade the output, and $\phi$ is the model’s likely prior exposure to the benchmark’s specific tasks during training. A score gap between two systems is informative about $\theta$ — the thing “OpenAI versus Claude” is supposed to mean — only when $H$, $e$, $\tau$, and $\phi$ are held fixed across both measurements. The evidence above shows that, on the leaderboards actually in public use, none of the four is reliably held fixed.

The one comparison that holds the harness fixed

There is one genuine exception, and it is worth dwelling on because it is rare. Artificial Analysis runs its own independent evaluation of Terminal-Bench v2.1, using a single fixed agent harness — Terminus 2 — inside standardized sandboxed environments, scoring pass@1 averaged over three repeats per task, and states explicitly that “all evaluations are conducted independently by Artificial Analysis” rather than accepted as vendor submissions ^[11]. Run that way, with the harness held constant and only the model and its effort setting varied, GPT-5.6 Sol at extra-high reasoning effort scores 89.5%, and Claude Opus 5 at maximum effort scores 89.1% ^[11]. The gap is 0.4 percentage points.

Set that beside the official, community-submitted Terminal-Bench 2.1 leaderboard, where entries supply their own harness and the page notes only that “a Terminal-Bench team member ran the evaluation and verified results” rather than standardizing the harness itself ^[10]. There, the top three entries as verified are Claude Code paired with Claude Fable 5 at 83.8%, OpenAI’s Codex paired with GPT-5.5 at 83.1%, and the Terminus 2 harness paired with Fable 5 at 80.4% — different models entirely from the Artificial Analysis pairing, a different rank order, and costs ranging from $438.64 to $2,059.19 to produce a single leaderboard row ^[10]. Both pages are legitimately called “Terminal-Bench 2.1.” They are not measuring the same thing, and a reader who quotes one number from each to build a table has silently mixed harness effects into what looks like a model comparison.

The Artificial Analysis result is the closest thing available to a controlled, apples-to-apples reading of current frontier agentic-coding capability, and what it shows is a gap smaller than the scaffold effects documented in the previous section. That is itself the finding: under the one measurement that isolates the model, OpenAI’s and Anthropic’s current flagships are close enough that the comparison does not support a confident ranking either way.

Higher effort is not free performance

A second complication sits inside every one of the numbers above: reasoning effort is not a dial that only goes up. The Holistic Agent Leaderboard project ran 21,730 agent rollouts across nine models and nine benchmarks, spanning coding, web navigation, science, and customer service, at a cost of roughly $40,000, using a standardized evaluation harness built specifically to remove implementation inconsistencies between runs ^[13]. Its headline result on effort is counterintuitive: higher reasoning effort reduced accuracy in the majority of runs studied, rather than improving it ^[13].

That finding cuts against a common shortcut in vendor comparisons, which is to quote each model at its own maximum available effort setting on the assumption that “max effort” means “best possible score.” If accuracy peaks before the top of the effort range on a meaningful share of tasks, then a maximum-effort-versus-maximum-effort comparison — like the two headline figures in the Terminal-Bench section above — is not comparing either vendor’s best number; it is comparing two numbers chosen by a rule that neither vendor’s own evaluation data necessarily supports.

What “solved” means, and how fragile it is

A resolve rate is only as trustworthy as the test suite deciding what counts as resolved, and the strongest independent challenge to SWE-bench Verified’s numbers attacks exactly that layer rather than any model. SWE-ABS, published in early 2026, built an adversarial testing framework combining coverage-driven program-slicing with mutation-driven adversarial patch generation, specifically to probe whether existing SWE-bench test suites were strong enough to catch subtly wrong “solutions” ^[14]. Applied across 500 instances, it strengthened just over half the test cases in the set and, doing nothing else, rejected 19.71% of patches that had previously been counted as passing across the top submissions studied — one in five nominally “solved” patches on the top-30 agents turned out to be semantically wrong, passing only because the original test suite never exercised the code path where the error lived ^[14]. The formerly top-ranked agent’s score fell from 78.80% to 62.20% under the strengthened suite, and it dropped out of first place into fifth ^[14].

The size of that drop is close to what a simple deflation model predicts. If $s_{\mathrm{reported}}$ is the score under the original oracle and $\mathrm{fp}$ is the false-positive rate the adversarial suite exposes, then

gives $0.7880 \times (1 - 0.1971) \approx 0.633$ — within a percentage point of the measured 0.622 ^[14]. That closeness is a coincidence of rounding as much as a proof of the model, since rejected patches are not independent of task difficulty, but the approximation is useful precisely because it names the assumption plainly: a benchmark score is a joint statement about the system under test and the strength of the judge grading it, and when the judge gets stricter without the model changing at all, the rank order of a leaderboard can change with it. Neither vendor’s headline SWE-bench Verified number has been re-run through this specific adversarial suite in public as of this writing, which means neither number’s resistance to the effect SWE-ABS documents is currently known.

A frozen public test set has a shelf life

A separate and more slow-moving problem is contamination. SWE-bench Verified’s 500 instances have been public, widely discussed, and presumably present in some form in web-scale training data since the set was released. SWE-bench-Live exists specifically to route around that: it maintains frozen “lite” and “verified” splits for comparability alongside an automatically curated, continuously updated split built from GitHub issues opened after each model’s likely training cutoff, refreshed on a monthly cycle with dedicated Docker environments for reproducible execution ^[16]. The frozen split stays useful for tracking a single system’s progress over time; it is a weaker instrument for a cross-vendor snapshot, because a model’s performance on a well-known, unchanging 500-task set reflects some unknown mixture of genuine capability and years of cumulative exposure to that exact set’s structure, discussion, and solutions.

This is not an accusation against either vendor specifically — it is a structural property of any static, famous, years-old benchmark, and it is one more reason the specific percentage point separating two frontier models on the original Verified split deserves less weight than the same gap would deserve on a freshly generated task set neither model could have seen.

A second, complementary axis: how long a task can run unattended

Resolve-rate benchmarks share a blind spot: they score whether a fixed-size task was completed, not how much autonomous work a model can sustain before it needs a human to intervene. METR, a research nonprofit that describes itself as operating independently of the labs it evaluates, measures a different quantity: the length of task, benchmarked against how long a skilled human professional takes to complete it, at which a model succeeds with 50% and 80% reliability ^[15]. Its methodology contracts professionals with software engineering, machine learning, or cybersecurity experience to establish human-time baselines, then fits a logistic curve to each model’s success probability as a function of that baseline duration ^[15].

METR’s tracked model list spans both companies directly and without a single ranking table standing in for the comparison: OpenAI’s GPT-5 series, o3, and o4-mini sit alongside Anthropic’s Claude 3.7 Sonnet through Opus 4.6 and the Mythos preview ^[15]. The organization’s broader published finding — that the task duration frontier AI agents can complete autonomously has been doubling roughly every seven months over a six-year trend, with a faster roughly four-month doubling visible in the most recent stretch — describes an industry-wide trajectory rather than crowning either vendor, and the time-horizons page itself offers no head-to-head verdict between the two companies ^[15]. That is a feature of the measurement, not a gap in it: because it is baselined against human task duration rather than a fixed public task list, it is far harder to move by scaffold tuning on one known benchmark than a SWE-bench Verified percentage is, even though it still says nothing about cost, price, or which vendor’s product a given team should buy.

What would make a comparison trustworthy

Pulling the mechanisms above together, a genuinely informative OpenAI-versus-Claude agentic-coding comparison would need, at minimum: one fixed harness run against both models rather than each vendor’s own scaffold; a disclosed and matched reasoning-effort or cost budget rather than each vendor’s self-selected maximum; a test oracle that has been adversarially hardened, or at least is known not to have the failure rate SWE-ABS documented; a task set recent enough that contamination is not a live concern; and multiple trials with reported variance rather than a single run. Of everything reviewed here, only the Artificial Analysis Terminal-Bench v2.1 reproduction clears the harness and repeat-count bars, and even it does not address effort-matching, oracle strength, or contamination. That is the honest state of the evidence, not a gap in this article’s research.

Predictions, and what would falsify them

Two forecasts, horizon August 2028, kept explicitly separate from the sourced findings above.

One. Independent, harness-fixed reproduction services — following the model Artificial Analysis has already built for Terminal-Bench — will become the primary citation for agentic-coding comparisons in technical reporting, displacing vendor system-card percentages as the default quoted figure. Assumption: the scaffold-driven variance documented here (ten to twenty points on identical models) becomes widely enough known that self-reported comparisons lose credibility. Indicator: independent reproduction leaderboards gaining citation share in technical press over vendor announcement pages. Disconfirmed if, by 2028, vendor system-card figures are still the primary numbers quoted in mainstream technical coverage without an independent harness citation alongside them.

Two. At least one adversarially hardened variant of SWE-bench Verified, in the lineage SWE-ABS opened, will become a standard companion score published alongside the original Verified number on the official leaderboard. Assumption: the field treats a 19.71% false-positive rate on a widely used benchmark as disqualifying for headline-only reporting once it is well known. Indicator: the official SWE-bench project adopting or endorsing an adversarially strengthened variant as a companion metric. Disconfirmed if the original, unhardened Verified split remains the sole SWE-bench figure vendors headline through 2028 with no adversarially strengthened companion in common use.

What the independent evidence actually supports

Set the vendor percentages side by side and OpenAI and Claude look like they are trading a fast-moving lead: 77.9% to roughly 81% to the high-90s within a single year of headline SWE-bench Verified numbers, plus a Terminal-Bench pair separated by 89.5% to 89.1% under the one genuinely controlled rerun available. That last gap — the only one produced by a harness held fixed across both vendors — is well within the range that scaffold choice alone moves a single model’s score, which is the central finding of this piece: at the one point where the comparison is actually controlled, it does not support calling a winner.

Everything else reviewed here is a reason for caution rather than a reason to prefer one company. Most SWE-bench leaderboard entries are self-reported rather than independently re-executed. The same model can be pushed ten to twenty points by scaffold choice alone. Higher reasoning effort does not reliably raise accuracy. A stricter, independently built test oracle has already knocked a leaderboard leader from first place to fifth without touching a single model weight. A frozen, years-old public benchmark carries an unknown and unfalsifiable contamination discount. None of that is a reason to distrust either vendor specifically — Anthropic and OpenAI are each publishing what their own system cards say they measured, honestly labeled as such. It is a reason to distrust the exercise of ranking them from those numbers alone, and to read every “OpenAI leads” or “Claude leads” agentic-coding headline as a claim about one harness, one effort setting, and one test suite on one day — not a claim about which model is better at writing and fixing code.