What Actually Happens Inside a Very Long Claude Context Window

The number on the box is not the number that matters

Anthropic states context window sizes as fixed engineering facts. As of current documentation, Claude Opus 5, Claude Sonnet 5, and several recent Opus and Sonnet models expose a one-million-token context window on the API at standard pricing, with no beta header required; older or smaller models such as Claude Sonnet 4.5 remain at two hundred thousand tokens ^[3]. Anthropic first opened a one-million-token window to beta customers on Claude Sonnet 4 in August 2025, at a price premium over the standard tier for anything past two hundred thousand tokens ^[2]; that premium has since been removed, and the larger window is now the default on current Sonnet and Opus models ^[3].

Read plainly, a token count like that invites the same mental model as a hard drive’s capacity: a container that either holds a file or it doesn’t. That is the wrong model, and Anthropic’s own current documentation says so in as many words, using a specific term for the failure mode it names directly: “as token count grows, accuracy and recall degrade, a phenomenon known as context rot” ^[3]. That sentence sits in first-party developer documentation, not in a critical third-party report, which makes it a reasonable place to start rather than end an investigation.

This article is about the space between those two claims — “the window can technically hold this many tokens” and “the model correctly used what was placed in them” — and what documented, independently verifiable research actually says fills that space. It works through the needle-in-a-haystack test that produced the headline recall numbers, the newer benchmarks built specifically to find out how far that test’s results generalise, the position effects that make some parts of a long context less reliable than others, Anthropic’s own architectural account of why any of this happens, how the whole approach differs mechanically from retrieval-augmented generation, and what a working engineer should actually do with a long window rather than simply filling it because it is available.

The test that built the marketing claim

The most-cited evidence for “long context works” traces back to a test an engineer named Greg Kamradt built and released as open source in 2023: take a corpus of ordinary text long enough to fill a context window, insert one out-of-place sentence — the needle — at a chosen depth, then ask the model to find it ^[6]. Vary the depth and the total length and the result is a two-dimensional grid: whether the model retrieves the needle, as a function of where it was buried and how much text surrounded it. The design’s appeal is obvious. It produces a single clean number, it scales cheaply to any context length a lab wants to advertise, and every major model provider adopted some version of it.

Anthropic’s own version, run for the Claude 3 family’s launch, was already more careful than the original: thirty different needle-and-question pairs tried against a diverse, crowdsourced corpus rather than one fixed pair, reducing the chance that a single lucky or unlucky sentence pair drove the headline number ^[1]. On that test, Anthropic reported that Claude 3 Opus achieved recall surpassing ninety-nine percent up to the model’s then-current two-hundred-thousand-token window ^[1]. The same announcement records an anecdote that later circulated widely on its own: in one run, Opus’s answer did not just supply the buried fact, it flagged that the sentence looked “very out of place and unrelated to the rest of the content,” speculating it might have been “inserted as a joke or to test if I was paying attention” ^[1].

That anecdote is worth treating as two separate claims, because the distinction is exactly what this whole article turns on. As a fact, it is accurately reported by Anthropic and independently repeated by outside coverage. As evidence of something like self-awareness or robust general recall, it is much weaker than it sounds: transcripts of needle-in-a-haystack tests, including their planted out-of-place sentences, are themselves now part of the public discourse that frontier models train on, so a model recognising the pattern “this sentence looks like an inserted test artifact” is at least as plausible an explanation as the model reasoning about its own situation from first principles. Anthropic’s own material reports the behaviour without asserting the stronger interpretation; that framing was added afterward, by commentary rather than by the model card. Near-perfect scores on the standard needle test are a real, verified, and narrow result. They are not, by themselves, evidence about recall or reasoning more broadly — and three separate benchmarking efforts were built specifically to test how far that narrow result actually travels.

Why passing the needle test proves less than it seems

Researchers at NVIDIA built a benchmark called RULER specifically to check whether “passes the needle test” predicts “handles a long-context task well” ^[8]. RULER keeps the needle-retrieval format but adds variations that most production use cases actually require: multiple needles instead of one, multi-hop tracing where the answer requires chaining facts together rather than locating one string, and aggregation tasks that require processing many relevant pieces of the context rather than filtering everything else out. Testing seventeen long-context language models against thirteen such tasks, the authors report that models achieving “nearly perfect accuracy” on the vanilla version of the needle test show “large performance drops as the context length increases” once the harder task variants are introduced, and that of the models publicly claiming support for contexts of thirty-two thousand tokens or more, only half maintained what the authors call satisfactory performance at that length ^[8]. The vanilla needle test, in other words, was measuring something closer to “can the model find one string that differs sharply from its surroundings” than “can the model reason over everything it was given.”

A second line of research pushed on a different assumption baked into the original design: that the needle and the question about it share obvious vocabulary. Real queries rarely offer that gift. NoLiMa replaced needles that could be located by literal string matching with ones requiring an inferred association not present in shared wording — a fact stated in a way that never repeats the words used to ask about it ^[9]. Across twelve widely used models advertised to support at least a hundred and twenty-eight thousand tokens of context, accuracy held up well below one thousand tokens of input, then fell steadily as length grew; at thirty-two thousand tokens, ten of the twelve models dropped below half of their own short-context accuracy, and even the strongest performer in the study, GPT-4o, fell from 99.3 percent accuracy at short context to 69.7 percent at thirty-two thousand tokens ^[9]. Separately, a team at Google DeepMind built a benchmark called Michelangelo, constructed so that the answer cannot be read off any single span of text but must instead be assembled from the shape of relationships across the document — closer to reasoning over a data structure than to finding a fact — and reported that frontier long-context models had clearly improved at literal retrieval while continuing to struggle specifically on this structural class of task ^[11].

None of these three studies was built to single out Claude, and none should be read as a Claude-specific finding — this is exactly the kind of cross-vendor result that should not be flattened into a ranking of one lab’s models against another’s. What they establish generally, across the field, is a gap between two different skills that a single needle-in-a-haystack score conflates: locating one sharply distinct string, and actually using everything in a long context the way a realistic task requires. A model can be excellent at the first while being materially worse at the second, and the standard test as originally built cannot tell the two apart.

Where the model looks, and where it stops looking

A separate and earlier finding constrains recall along a different axis entirely — not whether a fact can be found, but where in the context it happens to sit. In a widely cited study later published in the Transactions of the Association for Computational Linguistics, Nelson Liu and colleagues tested how well models used information placed at different positions across long inputs, including on multi-document question answering ^[7]. The result is now usually summarised as a single phrase: lost in the middle. Performance was consistently highest when the relevant information sat at the very start or the very end of the context and fell substantially — in some cases below what a much shorter context achieved — when the same information was placed in the middle, producing a U-shaped curve across position rather than a flat one. The effect held even in models built and marketed specifically for long-context use ^[7].

That result predates the current generation of million-token windows, so it is fair to ask whether newer, larger models simply grew out of it. A July 2025 technical report from Chroma, a company that builds retrieval infrastructure and therefore has a commercial interest in the answer, tested eighteen current models — including the Claude 4 family alongside GPT, Gemini, and Qwen models — across several tasks that extend the needle-in-a-haystack design, plus a conversational-recall benchmark and a task requiring exact replication of repeated text ^[10]. Chroma’s own name for the general finding is context rot: performance degrades measurably as input length grows even on simple tasks, the degradation is uneven rather than a single clean threshold, and it depends on factors an ordinary needle test does not vary — how closely the planted fact resembles the question asked, how many distractor sentences sit nearby, and how much internal logical structure the surrounding text has ^[10]. One specific finding runs counter to intuition and is worth reporting on its own: haystacks built from shuffled, logically disconnected sentences were retrieved more reliably across every tested model than haystacks built from coherent, logically ordered text, which the report’s authors attribute to coherent surrounding text creating more plausible distractors for a model to be misled by ^[10].

The same report isolates a Claude-specific behavioural pattern worth stating on its own rather than folding into the general trend. Across the Claude 4 family, hallucination rates under distraction were the lowest of any model family tested, with Claude Sonnet 4 and Opus 4 tending to state explicitly that no answer could be found rather than guessing; Opus 4 additionally refused the exact-replication task outright in 2.89 percent of attempts, citing its own concerns about the request, while still showing the slowest rate of accuracy decline among the models compared on that specific task ^[10]. That pattern is a third-party research finding rather than an Anthropic disclosure, and it describes a behavioural tendency, not a guarantee against error at any given length. It is reported here as one documented data point about how one model family fails, not as a settled property of the family going forward.

Mechanism: why more tokens is not free attention

Anthropic’s own engineering guidance, published in September 2025 as advice for developers building long-running agents, gives the clearest available first-party account of why context rot happens architecturally rather than treating it as an unexplained empirical curiosity ^[4]. The explanation rests on the transformer’s core mechanism: every token attends to every other token in the context through self-attention, so the number of pairwise relationships the model must represent grows with the square of the sequence length. For a context of $n$ tokens, the compute spent by self-attention within a single layer scales as

where $d$ is the model’s hidden dimension. Doubling the context does not double the work of relating everything in it to everything else; it roughly quadruples that specific cost. Anthropic’s guidance frames the practical consequence in the language of a finite resource: models have what it describes as a limited attention budget, and every additional token spends a sliver of it, so a model’s capacity to represent all the pairwise relationships in a very long input “gets stretched thin” well before the window is technically full ^[4]. The guidance names two further contributing factors distinct from the raw attention-cost argument: models see comparatively few long sequences during training relative to short ones, leaving fewer specialised parameters for context-wide dependencies, and the position-encoding schemes that let a model handle sequences longer than anything seen in training necessarily reduce the model’s precision about exactly where in a long sequence a given token sits ^[4]. Put together, Anthropic characterises context rot as “a performance gradient rather than a hard cliff” — a real, current-generation Claude behaviour, described in its own engineering team’s words, not a defect specific to a competitor’s system, and not a problem a strictly bigger window automatically fixes ^[4].

How this differs, mechanically, from retrieval-augmented generation

It is worth being precise about what “just use a longer context” actually replaces, because the alternative is not vague. Retrieval-augmented generation, as originally formalised by Patrick Lewis and colleagues, pairs a parametric model with a separate non-parametric retrieval step: a retriever selects a small number of passages likely to be relevant to a query from a much larger corpus, and only those selected passages, not the whole corpus, are placed in the model’s context for generation ^[14]. This publication’s own reporting on retrieval-augmented systems covers that mechanism, its indexing choices, and its failure modes at length elsewhere; the point relevant here is narrower and purely architectural. Long-context use skips the selection step entirely: the entire candidate material goes into the window, and the burden of finding what matters is placed on attention at inference time rather than on an index built in advance. Writing $n$ for the number of tokens placed directly in context and $k$ for the number of tokens a retriever would instead have selected, the quadratic cost derived above means the two approaches sit in different complexity regimes for handling the same underlying material:

That gap is not an incidental efficiency detail. It is the reason retrieval-augmented systems existed as a technique before million-token windows did, and it is why the comparison between the two approaches remains a live, actively contested research question rather than a settled one, even after context windows grew by orders of magnitude.

The evidence on that comparison genuinely disagrees, and it should be characterised that way rather than resolved artificially. A 2024 study from researchers at Google ran a systematic comparison of retrieval-augmented generation against long-context use of the same underlying models across public benchmarks and found that, “when resourced sufficiently,” long-context use consistently outperformed retrieval-augmented generation on average — while noting that retrieval’s dramatically lower computational cost remained a real and distinct advantage, and proposing a hybrid method, Self-Route, that uses a model’s own judgment of a query’s difficulty to send easy queries through cheap retrieval and route only harder ones to full long-context processing, reporting that this hybrid approached long-context accuracy at a fraction of the compute ^[12]. A separate 2024 study from researchers at NVIDIA took closer aim at the “long context wins on average” framing directly, arguing that spreading a model’s attention across an entire long context measurably reduces its focus on the material that actually answers a given question, and proposing an order-preserving retrieval method that traced an inverted-U relationship between the number of retrieved chunks and answer quality in their tests — quality rose, then fell, as more material was added — identifying a sweet point where their tuned retrieval approach matched or beat feeding the model the entire long context, using substantially fewer tokens to do it ^[13].

Both results can be true at once, because they are not measuring the same thing. One line of evidence says long context wins on raw average accuracy given sufficient compute; the other says a well-built retrieval step can match or beat it more cheaply, and that placing unfiltered material into a long context has its own failure mode distinct from cost — attention spent on irrelevant material is attention not spent on relevant material, which is a restatement, from the retrieval side of the research community, of the same attention-budget mechanism Anthropic describes from the model side. Where experts disagree here, the honest summary is that they disagree about magnitude and about which axis to optimise, not about whether either mechanism is real.

Engineering guidance: when long context is the right tool

None of the above supports “never use a long context” any more than it supports “context length is the only thing that matters.” It supports being specific about what a long window is good for and building around its documented weaknesses rather than around its advertised ceiling.

Long context earns its cost when material genuinely cannot be split without losing what makes it answerable: a single codebase where a change in one file depends on conventions defined in another far away, a contract that must be read as one coherent document rather than clauses in isolation, or an agentic session where a model’s own prior tool calls and their outputs are the material it needs to keep reasoning coherently, and where no fixed index could have been built in advance because the relevant material is generated turn by turn. Anthropic’s own current prompt-engineering guidance is built around exactly this profile of task and gives concrete, specific advice rather than a general endorsement: for inputs above roughly twenty thousand tokens, place long documents near the top of the prompt, above the instructions and the query, and ask the model to extract and quote the specific passages it will rely on before answering — reported to improve response quality on complex, multi-document inputs, and useful independently as a way to audit what the model actually used rather than trusting an unstated internal search ^[5]. For agentic work specifically, the same team’s guidance is not “extend the window further” but closer to the opposite: treat context as a finite, precious resource, actively compact and summarise older turns as a session runs long, keep external structured notes an agent can return to rather than relying on everything staying live in the window, and split work across specialised sub-agents rather than letting one agent accumulate an ever-longer single context ^[4].

Long context earns its cost less clearly, and a retrieval step should usually be preferred instead, when the underlying material is large, heterogeneous, and growing — a support knowledge base, a document store spanning many unrelated topics, a corpus where most of any given window would be irrelevant to most given queries. In that regime the quadratic cost above is not an abstraction; it is paid on every request, whether or not the request needed most of what was loaded, and the position effects and literal-matching sensitivity documented above mean a critical fact can go unretrieved even when it was technically present the whole time. A fact whose absence would be costly belongs at the start or end of a prompt, or behind an explicit retrieval step that can be tested and audited on its own, not buried mid-context and trusted to surface by attention alone. The practical rule that falls out of everything above is not a permanent preference between two techniques; it is that “the window is big enough to hold this” and “the model will reliably use what is placed in it” are different claims, verified by different tests, and a system built on the first without checking the second is building on an assumption the model’s own maker has explicitly warned against.

Predictions, with the observations that would falsify them

These are forecasts, clearly separated from the sourced findings above. Horizon: 12 August 2028.

One. Position-aware evaluation — reporting accuracy as a function of where the relevant fact sits within the context, rather than a single number blended across positions — becomes a standard part of published long-context benchmarks, on the assumption that a single blended score has already been shown, repeatedly, to hide the specific failure this article documents. Disconfirmed if the field’s major public long-context leaderboards in 2028 still report only single blended accuracy figures with no position breakdown.

Two. Hybrid retrieval-plus-long-context architectures, in the Self-Route style, become the default production pattern for large, heterogeneous document collections, with a full long-context pass reserved for queries a cheaper first pass flags as difficult, on the assumption that the cost asymmetry between $O(k^2)$ and $O(n^2)$ processing does not disappear as windows grow further. Disconfirmed if, by 2028, production systems handling large heterogeneous corpora have converged on feeding full context on every request as the standard approach, with no routing step.

Three. Context window sizes continue to grow past the current million-token figure, but published recall on realistic, non-needle tasks at the maximum advertised length continues to lag the advertised figure by a wide margin, on the assumption that the quadratic-attention and position-encoding mechanisms described above are architectural rather than incidental to any single model generation. Disconfirmed if a future frontier model demonstrates recall on structural, non-literal-matching tasks, in the style of NoLiMa or Michelangelo, at its full advertised context length that is statistically indistinguishable from its short-context recall.

What to take away

A long context window is a real, measured, and genuinely useful capability, and none of the research surveyed here says otherwise. What it is not is a single number that settles the question of whether a given task will work. The needle-in-a-haystack test that produced the headline recall figures measures a narrow and comparatively easy skill — finding one sharply distinct string — and three independent benchmarking efforts, built specifically to test how far that skill generalises, found it does not generalise as far as the headline number implies. A separate line of research found that recall is not even flat across a context of a given length: position matters, sometimes as much as length does. Anthropic’s own engineering account explains why in architectural terms rather than leaving it as an observed curiosity, and its own current guidance already reflects the finding: treat a long context as a finite, costly resource to be curated, not a bucket to be filled. The question worth asking about any given task is not how large the window is, but whether the specific evidence available — needle tests with their narrow scope, position-sensitive benchmarks, and the model’s own architecture — supports trusting it with the material this task actually needs recalled.