The Benchmarks That Don't Need the Image

The blind rerun

In 2015, the paper that introduced visual question answering as a research task published its own results table, and buried inside it is one of the cleanest demonstrations this field has ever produced of the problem this article is about. Alongside models that received both an image and a question, the authors also evaluated a model that received only the question — no pixels, nothing to look at — and reported its accuracy on the open-ended real-image task at 50.39%, against 57.75% for the best question-and-image model tested, and 83.30% for a human answering with both available ^[1]. A benchmark built specifically to require sight closed most of the distance to human performance without any access to the image at all.

That number should unsettle anyone who treats a multimodal benchmark score as self-evidently a measurement of multimodal ability, and a decade of follow-up work keeps finding the same shape of result on new benchmarks, in new modalities, under new architectures: a model scores well on a task built to require seeing an image, hearing audio, or watching video, without ever needing the channel the task claims to test. This is usually called unimodal bias, after the name a 2019 debiasing method gave it explicitly ^[5]. It is not a curiosity confined to one dataset but a structural property with a name in the broader machine-learning literature: shortcut learning, the tendency of a model trained by gradient descent to find whatever decision rule solves the objective at the lowest cost, whether or not that rule generalizes to the capability the benchmark’s authors intended to test ^[6].

This article is about what counts as rigorous evidence against that possibility. Not “does the benchmark sound like it requires the image” — VQA sounds like it requires the image, and did not, for a text-only baseline that scored well above chance. The actual standard is modality ablation: rerun the benchmark, or the trained system, with one input channel deliberately removed, degraded, or held fixed while another is perturbed, and treat the resulting gap as the real evidence of what the system used. What follows works through specific, documented forms of that test — text-versus-image, frame-versus-video, answer position, counterfactual image-question pairs — and what applying it to named, widely cited benchmarks has actually found.

What “unimodal bias” means, precisely

It is worth being exact about the claim, because “the model ignored the image” is not quite it. The precise claim is narrower: on a specific benchmark, a specific model achieves a score close to, or in some cases above, its score with the full multimodal input, when one modality is withheld. That is a statement about a benchmark-model pair under a stated protocol, not a general property of “multimodal AI.” A benchmark can be substantially unimodal-solvable for one model family and not another; a model can be robust on one benchmark and shortcut-prone on the next. The unit of analysis has to be the pairing, measured directly, every time — which is why ablation has to be a standing practice rather than a one-off finding people cite from memory.

The mechanism is well understood at a general level, independent of modality. Geirhos and colleagues, writing about deep-learning failures broadly rather than multimodal systems specifically, define a shortcut as “a decision rule that performs well on a standard benchmark but fails to transfer to more challenging conditions,” and argue that many of the field’s most persistent failure patterns — texture bias in image classifiers, spurious correlations in language-inference datasets, adversarial fragility — are symptoms of one dynamic: optimization finds the cheapest sufficient statistic in the training distribution, and a benchmark’s construction determines what that statistic happens to be ^[6]. Unimodal bias is a specific instance of that general pattern, not a separate phenomenon. When a benchmark’s language surface — its phrasing, its answer distribution, its option ordering — correlates strongly enough with the correct answer, gradient descent will exploit that correlation preferentially, because it is cheaper to learn than genuine cross-modal integration and the training objective does not distinguish between the two.

The rigor standard: ablate the channel, not the question

Two distinct engineering moves both count as modality ablation in the broad sense used across this literature, and it is worth separating them because they answer different questions.

The first is measurement-time ablation: take an already-trained system, rerun the exact same benchmark items with one channel removed at inference, and compare. This is what the original VQA paper did with its question-only baseline ^[1], and it is what a 2024 audit of vision-language benchmarks did formally, by defining two paired metrics from three separately measured accuracies. Let $S_v$ be a model’s accuracy on a benchmark with the image present, $S_{wv}$ its accuracy on the same items with the image withheld, and $S_t$ the accuracy of that same model’s underlying text-only language backbone, evaluated on its own before any multimodal training touched it. The audit’s two metrics are then

where $MG$, Multimodal Gain, is how much the image actually added once everything else is held constant, and $ML$, Multimodal Leakage, is the part of the no-image score that exceeds what the model’s own text backbone could already do unaided — evidence not that the benchmark is answerable from language in general, but that this specific model’s multimodal training pipeline let benchmark answers leak in through some channel other than genuine visual grounding ^[10]. The distinction matters because the two problems have different remedies: a benchmark with low $MG$ needs harder, more vision-dependent items; a model with high $ML$ needs a cleaner training pipeline. Collapsing both into a single “the model cheated” headline erases exactly the information a fix would require.

The second move is training-time ablation, where a unimodal branch is not just measured after the fact but built into the objective to actively suppress the bias it detects. RUBi trains an auxiliary question-only branch alongside the main vision-language model, uses that branch’s own confidence as a per-example signal for how answerable a question already is from language alone, and dynamically down-weights the main model’s loss on exactly the examples the question-only branch already gets right, forcing the remaining learning signal to come from cases where the image does real work ^[5]. Both moves rest on the same test — isolate one channel, measure what it alone can do — but one is diagnostic and the other corrective, and a mature evaluation culture needs both.

Case files: benchmarks that were solved blind

Applying measurement-time ablation to specific, named, widely used benchmarks has produced a consistent pattern of findings, worth walking through as documented cases rather than a single generic warning.

The 2024 audit that introduced Multimodal Gain and Multimodal Leakage tested current vision-language models directly on established benchmarks with the image removed. It reports that GeminiPro reached 42.9% accuracy on MMMU, a graduate-level, multi-discipline benchmark built around expert-level images, with no visual input at all, and that across six widely used benchmarks, models evaluated without images outperformed the random-choice baseline by more than 24 percentage points on average — meaning a very large share of what looks like multimodal reasoning is recoverable from the question text and answer options alone ^[10]. The audit separates that finding from a sharper one: Sphinx-X-MoE reached 43.6% on MMMU without accessing images, 17.9 points above what its own underlying language-model backbone achieved directly — a positive Multimodal Leakage reading, meaning the gap was not just “the benchmark is guessable from text” but “this model’s multimodal training introduced exploitable signal beyond what plain language competence predicts” ^[10].

MMMU’s own authors responded to exactly this class of finding by rebuilding the benchmark rather than defending the original. MMMU-Pro strips out questions an earlier audit found answerable from text alone, expands each question’s answer options to reduce guessing, and adds a vision-only condition in which the question and options are rendered as an image the model must read directly, closing off the possibility that an accompanying text transcript does unacknowledged work. Across the models tested, performance on the redesigned benchmark fell between 16.8 and 26.9 percentage points below the original MMMU score for the same model ^[11] — confirmation, from the benchmark’s own maintainers, that much of the original headline numbers reflected exactly the shortcut this article describes.

The pattern predates the current generation of vision-language models by most of a decade. Goyal and colleagues showed in 2017 that visual question answering models of that era were exploiting language priors — the tendency of a question like “what color is the banana” to have an overwhelmingly likely answer regardless of the specific image — and built a fix that did not rely on withholding the image at all: VQA v2.0 pairs every question with two similar images that have two different correct answers, so a model answering from language alone is mathematically capped near 50% on the paired subset by construction. State-of-the-art models tested against that rebalanced set performed significantly worse than on the original, confirming they had learned to exploit exactly the priors the redesign targeted ^[2]. A year later the same group went further with VQA-CP, reorganizing the train and test splits so the answer distribution for each question type differs between them — a model that memorized “bananas are yellow” from training-set frequency is punished specifically because the test-time base rate has been deliberately changed ^[3]. Separately, Manjunatha and colleagues used automated rule mining over a trained model’s own attention patterns and answer statistics to extract specific, human-readable shortcut rules — of the literal form “if the question mentions grass, answer green” — turning an aggregate accuracy gap into a named, inspectable mechanism rather than a suspicion ^[4].

Read together, these are not one finding repeated five times. They are five instruments — a blind rerun, a formal leakage metric, a benchmark redesign, a paired-image reconstruction, a distribution-shifted split, a rule-extraction audit — converging on the same diagnosis from different angles, a considerably stronger form of evidence than any one alone.

The same test on a different axis: time

Video benchmarks add a second axis to the same problem, because “video understanding” implicitly claims not just that a model uses visual information, but that it uses visual information across time — that motion, sequence, and causal ordering between frames matter to the answer, not merely the presence of some recognizable object somewhere in the clip. That claim has its own ablation test, and it has been applied with results that echo the text-versus-image finding closely.

Buch and colleagues built what they call an atemporal probe: a model deliberately restricted to select and reason over a single frame from a video, used as a formal upper bound on what pure image-level, non-temporal understanding alone can achieve on a task nominally requiring video. Applied to established video-language benchmarks, the single-frame probe matched or exceeded contemporary multi-frame, genuinely temporal models on several of them: on the multiple-choice variant of MSR-VTT it reached higher accuracy than VideoCLIP and MERLOT, both built to reason over sequences of frames; on the How2QA split of the VALUE benchmark it exceeded the previous best multi-frame result; and on NExT-QA, built specifically to test causal and temporal reasoning, it scored close to a dedicated temporal-reasoning model despite having access to only one frame ^[7]. The authors’ conclusion, stated directly, is that genuine understanding of event temporality is often not necessary to achieve strong or state-of-the-art performance on benchmarks built to require it ^[7].

The methodological point generalizes cleanly. Deleting all but one frame from a video clip is functionally the same operation as deleting the image from a text-plus-image pair: both ask whether the benchmark’s nominal reasoning requirement survives once the channel that supposedly supplies it is reduced to the minimum needed to establish a floor. Buch and colleagues used their probe constructively rather than only diagnostically, building “ATP-Hard,” a filtered subset containing specifically the items the single-frame probe could not solve, isolating each benchmark’s genuinely temporal residue ^[7]. That is the same move MMMU-Pro made for text-answerable questions, applied to a different modality boundary: once ablation identifies which items are shortcut-solvable, the fix is to keep only the items that are not.

Position is a modality too

A less obvious but equally well-documented shortcut exploits not which modality is present, but the order in which answer options are presented within a single item — and the fix for it is itself a form of ablation, applied to the answer key rather than to the image or the text.

MMBench introduced an evaluation protocol called CircularEval specifically to close this loophole. A multiple-choice question is not scored from a single pass; it is put to the model $N$ times, once for each of its $N$ answer options, with the options cyclically shifted into a new order on each pass, and the item counts as solved only if the model answers correctly on every single rotation ^[8]. The effect of switching from a single ungated pass to this all-or-nothing protocol was substantial: scores dropped roughly ten to twenty-four percentage points across most evaluated models, and for one open model, OpenFlamingo v2, the collapse was dramatic — from 36.7% under a single pass to 2.6% under CircularEval ^[8]. A model that had learned to favor a particular letter or position regardless of content will pass a single-pass evaluation whenever that position happens to hold the right answer, and fail almost every rotation once it does not.

It is worth seeing precisely why an all-or-nothing rule across permutations is so stringent, and where the intuition needs a caveat. If a model’s answer on each rotated pass were an independent uniform guess among $k$ options, the probability of passing all $N$ rotations by chance would be

which for a four-option item run through all four rotations is $(1/4)^4$, under half a percent. That is not a real prediction of model behavior — actual models are not memoryless guessers, and a model with a fixed positional bias will do considerably worse than this floor once the correct answer rotates away from its favored slot, while a model with genuine content understanding will pass reliably regardless of $k$ or $N$. The independence assumption is a deliberate simplification, useful only for showing why the two behaviors CircularEval separates — answering from content, and answering from position — land so far apart in practice, exactly as the OpenFlamingo v2 result illustrates.

Counterfactual pairs: ablation without deleting anything

Every method described so far ablates by subtraction: remove the image, remove all but one frame, remove the freedom to keep answering from the same position. A complementary and, in some ways, more surgical technique ablates by substitution instead — hold one channel fixed and perturb the other, then check whether the answer changes when it logically should, or stays the same when it logically should not.

HallusionBench operationalizes this as a diagnostic suite built around deliberately constructed question-image control groups: for a given real image and question, the benchmark also includes a modified variant — a subtly altered image paired with the original question, or a question that presupposes something the image does not show — designed so a model relying on a memorized prior rather than the actual pixels gives the same answer to both members of the pair, revealing itself in the disagreement between what it said and what the modified input contains ^[9]. The design separates two failure modes a simple accuracy number conflates: language hallucination, where the model answers from a linguistic prior regardless of what the image shows, and visual illusion, where it over-trusts a superficial visual pattern against context that should override it ^[9]. Scoring is unforgiving in the same spirit as CircularEval: a control group only counts as correct if every question tied to it is answered correctly. Across fifteen evaluated models, the best performer, GPT-4V, reached a question-pair accuracy of 31.42%; every other model tested fell below 16% ^[9].

The contribution here is distinct from outright channel deletion. Deleting the image entirely, as the VQA and MMMU audits do, answers “how much of the score survives with zero visual information.” Perturbing one input while holding the question fixed, as HallusionBench does, answers a more precise question: does this specific piece of visual content change the model’s answer the way it should. A model can pass a blind-rerun ablation — clearly using some visual signal — and still fail a counterfactual-pair test, because the signal it uses is a coarse pattern rather than the specific content difference the pair was built to isolate. Rigor is not one test; it is a family of tests isolating non-interchangeable failure modes, and a system can be innocent under one and guilty under another.

Why passing every audit still is not real-world robustness

Everything above describes ways to determine whether a system is exploiting a shortcut on a fixed, curated, already-collected set of benchmark items. That is a meaningful, increasingly well-instrumented question, and a model that survives blind reruns, leakage checks, temporal ablation, answer-order permutation, and counterfactual pairing on a given benchmark has cleared a real bar. It has not, however, answered a harder question: whether the same system is robust to inputs an adversary or the open world chooses deliberately, rather than inputs a benchmark’s authors happened to include.

Zhao and colleagues tested exactly that gap directly, rather than assuming it. Using open vision-language encoders such as CLIP and BLIP as surrogate models, they crafted targeted adversarial image perturbations designed to steer a system’s output toward an attacker-chosen response, then transferred those perturbations, without further access, to a range of other large vision-language models, including MiniGPT-4, LLaVA, UniDiffuser, BLIP-2, and Img2Prompt. The transfer-based attacks achieved a high success rate at inducing the targeted response through black-box transfer alone, and limited query access to the target model improved that success rate further ^[12]. None of the systems tested were unusually weak by the standards of curated benchmark evaluation; several belong to model families that score competitively on exactly the kind of vision-language benchmarks discussed throughout this article. The two facts sit side by side without contradiction, because they measure different things: a benchmark audit asks whether a model uses the image on the items in front of it, while an adversarial-robustness study asks whether an adversary free to choose the image can control what the model reports seeing at all. A high score on the first gives no guarantee whatsoever about the second.

This is the same distinction Geirhos and colleagues make when they define a shortcut as a rule that “performs well on a standard benchmark but fails to transfer to more challenging testing conditions” ^[6] — and VQA-CP made the identical point for language priors specifically, years before adversarial robustness became a mainstream concern for multimodal systems, simply by changing the test-time answer distribution and watching in-distribution accuracy collapse ^[3]. A curated benchmark, however carefully audited, is still a fixed sample from a known distribution. Passing every ablation in this article demonstrates that a model is not taking the specific shortcuts those tests were built to detect. It does not demonstrate that no shortcut exists, only that the ones looked for were not found — and an adversary, unlike a benchmark author, is not obligated to test only the failure modes someone already thought to check.

Predictions, with the observations that would falsify them

These are forecasts, kept explicitly separate from the sourced findings above. Horizon: August 2028. Assumptions: continued public disclosure of benchmark construction methodology by major academic venues, no single multimodal benchmark becoming a de facto standard without a published ablation study, and continued academic incentive to publish audits of widely cited benchmarks.

One. Reporting a blind or unimodal-ablation baseline alongside the headline score will become close to a submission norm for new multimodal benchmarks at top vision and language venues, following the pattern set by the MMStar and MMMU-Pro audits. Indicator: the share of newly introduced multimodal benchmarks at CVPR, NeurIPS, and ACL reporting an explicit no-image or no-audio control. Disconfirmed if a majority of such benchmarks at these venues in 2028 still report only full-input accuracy.

Two. Grouped, all-or-nothing scoring in the style of CircularEval and HallusionBench’s paired control groups will spread from image-text question answering into audio and video benchmarks, since the underlying shortcut-exploitation dynamic is not specific to one modality pairing. Disconfirmed if audio- and video-focused benchmarks introduced in 2028 still predominantly score independent single items with no paired or grouped condition.

Three. Single-frame or few-frame temporal-ablation baselines will become a standard reported control for new video-understanding benchmarks, mirroring how text-only baselines became expected for image benchmarks after this decade of audits. Disconfirmed if 2028’s leading video-understanding benchmarks report headline scores with no frame-reduced control anywhere in the paper.

Four. The measured gap between best-reported curated-benchmark accuracy and independently verified adversarial or black-box robustness on the same model families will not close, because benchmark curation and adversarial red-teaming remain distinct research efforts on different timelines optimizing different objectives. Disconfirmed if independent, non-vendor adversarial-robustness studies on 2028’s frontier multimodal systems report attack success rates comparable to those against untrained baselines.

What to take away

A multimodal benchmark score, reported on its own, is not evidence that a system integrated the modalities the benchmark claims to test. It is a number that becomes evidence only after it survives specific, named forms of scrutiny: a rerun with the image deleted, a formal decomposition into genuine gain and leaked advantage, a rerun with all but one video frame deleted, a rerun across every permutation of the answer key, and a rerun against a counterfactual pair that changes exactly one thing. Every one of those tests has been applied to real, widely cited benchmarks in the published record, and in nearly every documented case, a meaningful share of the reported score turned out to be reachable without the capability the benchmark was built to measure.

None of this is an argument that multimodal benchmarking is broken beyond use, or that reported scores should be dismissed. It is an argument for a specific discipline: before treating a multimodal number as a claim about multimodal ability, ask which of these ablations, if any, were actually run, and read the gap they expose rather than the headline they sit beside. And even a benchmark that has survived every ablation in this article has answered only the question of whether it can be solved by a known shortcut on its own fixed item set — not whether the system behind it is robust to an image, question, or frame nobody thought to include. Those are different claims, established by different evidence, and this publication’s own editorial standard applies here as much as anywhere: report which test was run, and do not let a score answer a question it was never built to answer.