Cognitive Bias Was the Label; Attention Sink Is the Suspect

Xiaobo Guo and Soroush Vosoughi published “Serial Position Effects of Large Language Models” in Findings of the Association for Computational Linguistics: ACL 2025, held in Vienna, Austria, after first posting the work as an arXiv preprint in June 2024 ^[1]. The study is thorough by any reasonable standard: thirteen model checkpoints spanning three architecture families — four GPT-series models, five from the Llama2 lineage including two Solar variants, four from the T5 lineage — tested across eight datasets, five for classification and three for summarization, with primacy dominant in seventy-three of the resulting hundred and four configurations. Guo completed the work as a PhD candidate in Dartmouth College’s Department of Computer Science, defending a dissertation titled “Leveraging Large Language Models for Enhancing Well-Being in the Digital Age” in 2024 before spending 2024–2025 as a researcher at Amazon PXT ^[13]; nothing in the public record this article could reach confirms where he has worked since, and a secondhand claim of a further move is left out of this account for that reason. Vosoughi, the senior author, has since been promoted to Associate Professor of Computer Science at Dartmouth, where he continues to lead the “Minds, Machine, and Society” research group ^[12].

The paper opens by calling serial position effects, “including the primacy and recency effects,” well-documented cognitive biases in human behavior, citing Solomon Asch’s 1946 impression-formation experiments for the first and Alan Baddeley and Graham Hitch’s 1993 account for the second ^{[1, 6, 7]}. That framing sets up the sentence this article is built around, which sits at the very end of the introduction’s bulleted list of findings, one paragraph before the paper turns to related work: “The comparison between LLM behavior and human cognition suggests that LLMs’ sensitivity to the order of prompts may extend beyond mere engineering challenges and relate to the fundamental nature of their attention mechanisms” ^[1]. The hedge is real — “may extend,” “may relate” — and it deserves to be read as a hedge rather than an assertion. But a hedged sentence still does work. It tells a reader where to look for an explanation, and it points away from the one explanation that was already sitting in the literature when the sentence was written, fully capable of producing the identical measured curve without borrowing a single term from the psychology of memory.

Seventy-Three Trials Are Not a Coin Flip

Before any of that, credit belongs where the paper actually earned it, and the measurement is the place it earned it most cleanly. For each classification query, the paper’s own protocol produces two prompts with identical content and different label orders, records which label the model selects under each, and repeats this across three thousand randomly sampled queries per dataset — “designed to eliminate any potential bias in dataset sampling,” in the paper’s own description ^[1]. A primacy effect is declared when the first third of a label list accounts for more than forty percent of the model’s selections, aggregated across those three thousand trials; recency and middle effects use the same forty-percent rule against the last and middle thirds respectively, and their absence is scored as no effect at all. The intensity of whatever effect is found is then quantified separately, as the Jensen–Shannon divergence between the observed selection distribution and a uniform reference distribution — the paper’s own SPEM statistic, $\mathrm{SPEM} = \mathrm{JS}(\hat P \,\|\, R)$, where $\hat P$ is the predicted label distribution and $R$ is the reference. None of that construction is casual, and none of it is unique to one model family: thirteen checkpoints across three architecturally distinct lineages, eight datasets ranging from a seventy-seven-intent banking corpus to five-article news summaries, and a headline count — seventy-three of a hundred and four tested configurations showing primacy — that is exactly what falls out of multiplying thirteen models by eight datasets, not a rounded or approximate figure.

A skeptical reader’s first move should be to ask whether that headline number could be sampling noise dressed up by a generous threshold. It is worth actually running that check rather than asserting an answer to it. Under a null model in which a classifier’s selections are entirely independent of label position — no cognitive bias, no architectural quirk, nothing but content — the expected share of selections landing in any one third of an evenly split label list is $p_0 = 1/3$, and with $n = 3000$ independent trials per dataset the standard deviation of that observed share is $\sigma = \sqrt{p_0(1-p_0)/n} \approx 0.0086$, well under one percentage point. The paper’s own forty-percent threshold sits $0.40 - 0.3333 \approx 0.0667$ above that null mean, which is $z \approx 7.75$ standard deviations out — a one-sided tail probability, by the normal approximation to the binomial (valid here since $np_0(1-p_0) \approx 667$, far above the usual rule-of-thumb minimum), of roughly $4.7 \times 10^{-15}$. At three thousand trials per dataset, in other words, a classifier with no position sensitivity of any kind would essentially never cross this threshold by chance, in either direction. Whatever explains the measured pattern, it is not noise inflated by a lenient bar. This computation does not touch the paper’s causal claim in either direction — it only confirms that the thing being explained is real, which sharpens rather than weakens the actual dispute: the seventy-three-configuration majority is not the part anyone should contest. What follows contests only which sentence gets to explain it.

The measurement’s reach extends past multiple-choice selection, too, which matters for how seriously it should be taken. For the three summarization datasets, the paper cannot ask which discrete label a model “chose,” so it instead measures each source article’s normalized BERTScore against the model’s generated summary and treats the resulting per-position differences as a proxy for where the model’s attention actually landed while composing its output ^[1]. Recency shows up more often than primacy in that setting, and more clearly in Summ5’s five-article lists than in Summ20’s twenty-article lists — the paper’s own reading is that as prompt length grows, whatever residual capacity a model has for tracking the far end of a list gets overwhelmed, leaving the front-loaded sink as the more reliable survivor. That length-dependence is itself a data point worth flagging in advance of the countermodel below: it is exactly the signature a fixed-position, capacity-limited attention mechanism would produce, and not an obvious prediction of a psychological account built from human list-recall experiments, where recency in free recall famously survives long lists rather than fading with them.

A Second Explanation Was Already on the Record

The mechanism at issue is built into the architecture these thirteen checkpoints all share. Every model the paper tests, decoder-only or encoder-decoder, computes attention the way Ashish Vaswani and seven coauthors specified in 2017: a query vector compared against every visible key, the resulting scores passed through a softmax so they sum to exactly one, then used to weight a sum over value vectors ^[11]. That normalization constraint is the whole story. A softmax cannot output “attend to nothing” — every unit of attention mass has to land somewhere, on every single query, at every layer, whether or not anything in the visible context is actually relevant to that query.

Roughly nine months before the arXiv preprint that would become this paper, Guangxuan Xiao and four coauthors published “Efficient Streaming Language Models with Attention Sinks,” accepted at ICLR 2024, documenting what that constraint produces in practice: in a causally masked transformer, the first few tokens of a sequence absorb a disproportionate share of attention mass from every later token, almost regardless of what those first tokens actually contain ^[2]. A model trained under the softmax constraint above learns to route a nontrivial fraction of “nothing here is relevant” mass onto the earliest available positions, because those are the only positions guaranteed to be visible to every later query in the sequence — a dumping ground the normalization itself forces into existence. The tokens themselves are close to interchangeable — swapping the first few tokens for unrelated text does not eliminate the sink, only relocates which specific positions absorb the mass. A follow-up paper, “When Attention Sink Emerges in Language Models: An Empirical View,” accepted as an ICLR 2025 Spotlight, traced the phenomenon further: sink strength develops over the course of pretraining, varies with how a model is trained rather than only with how large it is, and — critically for what follows — the same research group built and released a matched pair of training pipelines, one using ordinary softmax attention and one using a sigmoid-based attention variant that does not force outputs to sum to one and does not develop the same sink ^[3]. That toolkit exists, publicly, on GitHub, built for exactly the kind of comparison this article is about to describe.

Layer a second architectural fact on top of the sink and the paper’s own headline result — primacy dominant more often than recency, but recency still present in a meaningful minority of configurations — stops needing any appeal to memory at all. Relative positional encoding schemes, of which Rotary Position Embedding is the dominant example in the model families this paper actually tested, are constructed so that a query attends more strongly to keys that are nearby in sequence position, with attention decaying — not vanishing, but weakening — as relative distance grows ^[10]. Put an attention sink at the front of the sequence and a relative-decay bias toward the back, and a list of options gets pulled from both ends at once by two independent, well-documented geometric properties of the architecture, with whatever is left in the middle receiving the leftover attention neither end absorbed. That is precisely the shape Guo and Vosoughi measure and describe as an effect “attributed to factors such as diminished attention” when characterizing the human literature ^[1] — except here diminished attention is not a metaphor for a cognitive process. It is a literal, computable property of a softmax normalization and a rotation matrix, present before a single training example teaches the model anything about language at all. A closely related U-shaped curve shows up at a completely different scale of the same architectures: Nelson Liu and six coauthors found that models asked to retrieve a fact from a long document perform best when the fact sits at the very start or the very end of the context and measurably worse in the middle, a pattern they name “lost in the middle” and trace to exactly the same sink-plus-decay geometry, just stretched across thousands of tokens instead of dozens of shuffled labels ^[5]. Two papers, two task families, one architectural cause showing up at two different length scales is a stronger pattern than either paper alone would suggest, and neither one needed a theory of human memory to find it.

The Appendix That Asked the Right Question and Got No Answer

The paper’s own appendix, in a section titled “The Predictability of Serial Position Effects,” comes close to running exactly the kind of test this article is arguing for — and its result is more interesting, and more damaging to any confident reading in either direction, than either the main text or the paper’s abstract lets on. The authors fit a logistic regression predicting which type of serial position effect appears — primacy, recency, middle, or none — from four candidate features: model size in parameters, task accuracy, the rate at which a model’s predicted label changes when the list is reshuffled, and model architecture family, encoded as a set of dummy variables ^[1]. This is, in substance, an attempt to ask whether something about a model’s construction — exactly the question an attention-sink account would want answered — predicts the effect. On the MASSIVE dataset, the primacy-effect regression reports a Model Size coefficient of $-0.0345$ with a standard error of $0.103$ and $p = 0.738$: a small, statistically unremarkable coefficient, cleanly estimated, that shows no relationship. Read alone, that looks like real evidence against any scale-dependent architectural account.

It should not be read alone. The companion regressions for the middle and recency effects, reported directly beside it in the same appendix table set, show a different and more telling pattern. The middle-effect regression reports a Model T5 coefficient of $-6.2258$ carrying a standard error of $3.19 \times 10^{4}$, and a Model Llama2 coefficient of $-7.5794$ carrying a standard error of $1.07 \times 10^{4}$ — standard errors between one and five thousand times the size of the coefficients they attach to. That ratio is not statistical noise; it is the textbook signature of quasi-complete separation, the condition in which a logistic regression’s maximum-likelihood estimate fails to converge because some category has too few outcomes on one side of the split for the model to be identified at all. The authors’ own conclusion, stated plainly in the same section, is admirably honest about the ambiguity: “This could be either due to our limited sample size or show that these factors are not predictive of the SPE in LLMs” ^[1]. That sentence is doing more work than it appears to. A regression that cannot converge cannot rule an architectural account out; at best it fails to rule it in, on a design — thirteen models from three architecturally distinct families, each contributing a single data point per dataset, with architecture family and parameter count strongly collinear by construction — that was never going to have the statistical power to separate “this doesn’t matter” from “this couldn’t be tested.” The question the appendix asks is the right one. Its own numbers show the tool it reached for could not answer it.

A Sentence Sitting Between Two Bibliographies

The disputed sentence’s Related Work section, read directly from the published camera-ready PDF rather than assumed from the earlier preprint, cites twelve prior papers by name: two studies of multiple-choice selection bias, a study of sensitivity to option order, two studies of in-context example ordering, two studies of adversarial and spurious prompt features — including Melanie Sclar and three coauthors’ demonstration that open-source models can swing by dozens of accuracy points on nothing but prompt formatting ^[9] — three studies measuring primacy or recency directly in specific model families, among them Yiwei Wang and four coauthors’ direct measurement of a primacy effect in ChatGPT itself ^[8], and one study of how reinforcement learning from human feedback might modulate the effect ^[1]. Every one of those twelve is an empirical NLP paper about prompt sensitivity — not one is a cognitive-psychology citation. The cognitive-psychology references that give the paper its title’s framing — Asch’s impression-formation work, Baddeley and Hitch on recency, plus three further citations on primacy-recency mechanisms — appear earlier, in the introduction, motivating the analogy before Related Work ever begins ^{[6, 7]}. Xiao and colleagues’ attention-sink paper is absent from both lists. So is the empirical follow-up on when sink formation emerges. So is any purely architectural account of position sensitivity. This was checked directly against the ACL Anthology’s published version of record — not the June 2024 arXiv preprint, which carries the identical gap — specifically because a camera-ready revision is exactly the kind of place a gap like this would ordinarily get closed during peer review, and in this case it was not.

That absence is not a discredit to the authors, and treating it as one would be a strawman of exactly the kind this article is obligated not to build. The attention-sink paper and the paper under discussion here were developed on close to the same timeline, in adjacent but not identical subfields — one working from mechanistic interpretability and long-context inference efficiency, the other from prompt-sensitivity measurement and its parallels to cognitive science — and citation gaps between adjacent subfields moving at the pace NLP moved in 2023 and 2024 are common enough that their absence proves nothing about anyone’s diligence. What the absence does show is narrower and more useful: the sentence claiming the effect “may extend beyond mere engineering” was written, and later republished unchanged through peer review, without ever being tested against the one engineering account built to produce exactly this pattern. A sentence can be honestly hedged and still be doing unearned work if the hedge was never checked against the strongest available alternative.

What a Sink and a Rotation Predict Without a Mind

Independent evidence that this is more than a plausible-sounding architectural story already exists, on a different experimental paradigm than the one under examination here. A 2025 preprint from Rutgers University–New Brunswick, training small transformer models from scratch on synthetic tasks modeled directly on human free-recall paradigms, reports that identifying attention layers exhibiting sink behavior and disrupting them through targeted dropout removed the primacy effect from the model’s outputs, with the largest measured performance drop concentrated specifically at the first-token position ^[4]. The same preprint explicitly engages Guo and Vosoughi’s results in its discussion, treating the reduction in position sensitivity as model complexity increases as consistent with what it independently measured. This is not the identical experiment the argument above calls for — it uses purpose-built small models trained on synthetic memory tasks, not the specific Llama2, GPT, T5, and Solar checkpoints this paper tested on real classification and summarization data — but it is a real, executed ablation, on a cognate architecture and a related task family, and it points the same direction the sink account predicts: remove the sink mechanically, and the primacy effect goes with it, with no appeal to anything resembling a memory system required at any point in the removal or the measurement.

The countermodel gains a second kind of support from a place that looks at first like it should count against it: the paper’s own successful mitigations. Guo and Vosoughi report that directive prompts explicitly instructing a model to focus on a particular third of the list, and Chain-of-Thought prompting that forces the model to consider every option before answering, both measurably reduce the effect’s magnitude, though inconsistently across models and tasks ^[1]. A purely psychological reading has no particular reason to expect a change in surface instructions to move a bias rooted in “the fundamental nature of attention mechanisms” — if the mechanism really were analogous to a human memory constraint, telling a person to “pay attention to the middle” only partially overrides encoding limits that operate below conscious control, and the paper’s own inconsistent success rate is at least compatible with that reading. But it is at least as compatible, and arguably more directly explained, by the architectural account: an instruction like “please pay attention to the middle N labels” adds tokens near the point where the model must produce its answer, changing the relative-position distances between the label list and the query — exactly the lever RoPE-style decay operates on — without touching the fixed early-position sink at all. Chain-of-Thought’s inconsistent success fits the same picture: forcing the model to restate each option before answering effectively re-inserts every option close to the query position, partially neutralizing the decay term for options RoPE would otherwise have discounted, while doing nothing to the sink itself, which is exactly why the mitigation the appendix’s own data shows working best is imperfect rather than complete. Neither account is falsified by the mitigation data. But an account that requires no appeal to memory, encoding depth, or rehearsal strategies to explain why prompt engineering has partial, inconsistent success is the simpler one standing on the same evidence.

A Fraction That Would Have to Be Measured, Not Argued

Everything above is a case for treating the architectural account as at least as well supported as the psychological framing. It is not, on its own, a decisive discriminator, and this article’s one genuine contribution is naming and specifying the measurement that would be. Call it the sink-attributable fraction, $\phi_{\mathrm{sink}}$, defined as

where $\mathrm{SPEM}_{\mathrm{softmax}}$ is Guo and Vosoughi’s own Jensen–Shannon-divergence effect magnitude, measured by their own published protocol, on an ordinary sink-forming model, and $\mathrm{SPEM}_{\mathrm{sigmoid}}$ is the identical measurement on an architecture- and data-matched twin trained with sigmoid attention instead of softmax — the exact contrast Gu and colleagues’ released training pipeline was built to produce ^[3]. The quantity is dimensionless, bounded in the neighborhood of zero to one for any model pair where the sink-free twin is not somehow more position-sensitive than its sink-forming counterpart, and it answers a single, sharp question: of the effect magnitude the original paper measured, how much of it disappears when the one specific architectural mechanism this article names is removed and nothing else changes?

I have not measured $\phi_{\mathrm{sink}}$ for the checkpoints this paper actually tested, and no claim in this article depends on pretending otherwise. Doing so would require training matched softmax and sigmoid-attention models at Llama2 scale — a run measured in GPU-days this article does not have access to — and then rerunning Guo and Vosoughi’s own classification protocol on both. What can be stated precisely is the protocol and the threshold that would settle the question. Train the matched pair at each of the three Llama2 parameter counts the original paper tested — seven billion, thirteen billion, seventy billion — using the released sail-sg/Attention-Sink pipeline; reimplement Guo and Vosoughi’s own label-shuffling protocol on Banking77, GoEmotions, MASSIVE, TACRED, and RE-TACRED, unchanged; compute $\phi_{\mathrm{sink}}$ at each scale. A value clustering above roughly $0.7$ across the size sweep would mean the sink accounts for most of what was measured, leaving little for “beyond mere engineering” to explain. A value clustering below roughly $0.2$ would mean the sink is close to irrelevant to this particular effect, the hedge in the original sentence would be vindicated, and something the architectural account does not currently supply would be operating instead. Either outcome is worth knowing, and neither has been measured by anyone yet, including the authors of either paper this article draws on.

The Hedge the Countermodel Cannot Touch

The strongest response to all of this does not require disputing a single number above; it requires only reading the original sentence as carefully as it was written. “May extend beyond mere engineering challenges” is not “does extend,” and a paper that hedges correctly should not be penalized for a critique constructed as though it hadn’t. Guo and Vosoughi’s own limitations section is candid about exactly the kind of uncertainty this article has been describing, acknowledging that their identification thresholds were “derived from empirical observations, given the lack of standardized criteria for SPE in the current literature” and that the boundary between effect types is not always clean ^[1]. A paper this careful about its own measurement’s edges was, if anything, more careful than most in this literature about not overclaiming a mechanism.

What survives every argument above, untouched, is the paper’s actual empirical contribution. The measurement itself — that primacy and recency sensitivity is widespread across architecturally distinct model families, that it is not confined to decoder-only transformers since the encoder-decoder T5 and Flan-T5 models showed comparable effects, that it persists into summarization tasks via BERTScore-measured focus rather than only multiple-choice selection, and that mitigation through prompting is real but unreliable — does not depend in any way on which mechanism turns out to explain it. A reader deciding whether to trust an LLM-as-judge system on an unlabeled ranking task needs to know that position matters and how much, not why it matters, and the paper answers the “how much” question rigorously regardless of how the $\phi_{\mathrm{sink}}$ measurement eventually comes out. Nor does the countermodel explain everything the paper reports: the appendix’s own finding that neither model size nor architecture family significantly predicts which specific type of effect appears is, on a well-behaved sub-model like the MASSIVE primacy regression, a genuine data point that an architecture-only story has to sit with rather than explain away, even granting that the same appendix’s other two regressions were too unstable to trust either direction.

The Number That Would End This, Stated Before It Exists

Sorting what stands on which footing is worth doing plainly rather than blending it into a closing flourish. Observed: the paper’s measured seventy-three-of-a-hundred-and-four primacy dominance, its SPEM protocol, its appendix regression coefficients and standard errors, and the exact absence of any attention-sink citation in the published camera-ready’s Related Work section or bibliography, all drawn from the primary source and confirmed directly against it. Derived: the binomial null-model calculation showing the measured effect is not a threshold artifact, computed once from the paper’s own stated sample size and threshold rule using nothing but arithmetic. Proposed: the sink-attributable fraction $\phi_{\mathrm{sink}}$, defined with explicit units, a stated protocol built entirely from already-released tools, and a numeric threshold on either side of which the dispute resolves — but not computed, because computing it needs compute this article does not have. Not performed: any training run, any ablation on the exact checkpoints this paper used, and any direct measurement of how much of Guo and Vosoughi’s own reported effect a sink accounts for.

None of the three outcomes above requires anyone to take this article’s word for anything. The datasets are public, the checkpoints are open where they are not proprietary APIs, and the training pipeline that would produce the matched pair is already released under an open license by the same lab that documented sink emergence in the first place. The only genuinely scarce input is compute, and it is worth naming the actual bill rather than waving at it: training three pairs of models at 7B, 13B, and 70B parameters, softmax and sigmoid variants each, then running an already-specified evaluation protocol on five datasets, is a project sized for a well-resourced lab over a period of weeks, not a research program measured in years. That is a modest price for closing a gap this specific.

What changes if $\phi_{\mathrm{sink}}$ comes back high is not a correction to a footnote. It is a reclassification of what kind of problem serial position sensitivity is for anyone building an evaluation pipeline around LLM judgment: not a psychological quirk requiring behavioral workarounds borrowed from survey design, but an engineering property with an engineering fix — sink-aware attention variants, sink-token insertion at fixed positions, or architectures that never form the sink at all — sitting one training run away from the labs that already build these models. What changes if it comes back low is smaller but still real: it means the architectural account this article has spent its length building was not sufficient, the authors’ hedge was earned rather than merely careful, and whatever does explain the residual effect remains to be found by someone willing to run the training pair this article could only specify. The measurement that would decide between those two futures does not exist yet. Its absence, not any number in this article, is the actual finding.