Five places the claim can be tested
Multimodal leaderboard averages have climbed for three years running, and it is tempting to read that climb as a single competence approaching completion. It is not, and the reason for saying so has to be stricter than a general sense that language models sometimes get things wrong. This article holds itself to one rule: every claim that a specific capability remains unsolved is backed by a benchmark paper’s own reported finding about its own instrument, not by a demo failure, an anecdote, or a general appeal to skepticism. Where a paper reports a limitation, disagreement, or a specific accuracy gap, that number is quoted and attributed. Where evidence is mixed, that is stated as mixed.
Five capabilities recur across the current benchmark literature as places where the “multimodal is basically solved” reading breaks down: reasoning about temporal structure in long video, associating a sound with the correct source in a visually busy scene, inferring three-dimensional structure and uncommon viewpoints from a training distribution that is overwhelmingly flat images, describing visual or audio content with confidence that outruns what is actually present, and generalising to combinations of modalities that a training corpus rarely paired together. Each has its own dedicated benchmark, built specifically because an earlier, easier evaluation was not actually measuring the capability it claimed to measure. That pattern — a harder benchmark built to remove a shortcut the easier one allowed — is itself informative, and the article returns to it at the end.
Long video: temporal reasoning that degrades before the context runs out
The first problem is not that models cannot process long video. Context windows have grown enormously, and a system can now ingest an hour of footage without technically failing to run. The problem is that the quality of temporal reasoning over that footage degrades well before any hard limit is reached, and two benchmarks built specifically to isolate temporal structure from everything else document this directly.
TemporalBench constructs roughly ten thousand video question-answer pairs from around two thousand fine-grained human annotations, with negative captions designed to probe exactly the distinctions a single frame cannot resolve: action frequency (“twice” against “three times”), event order (“then” against “before”), and directionality (“pushes” against “pulls”) [1]. These are not obscure or contrived distinctions; they are the ordinary content of a sentence describing what happened. The paper’s own reported result is that a leading model of its testing period answered only 38.5% of these questions correctly, a gap of roughly 30 percentage points below human performance on the same items [1]. That gap is not explained by video length in the everyday sense — many TemporalBench clips are short — which is itself the finding: fine-grained temporal understanding is a distinct skill from long-context ingestion, and it lags even at the scale of individual clips before duration is added as a second axis of difficulty.
LongVideoBench adds that second axis directly, building a benchmark of videos up to an hour long with subtitles, and introducing a “referring reasoning” task in which a question references a specific span of context that the model must locate and use [2]. The paper’s central reported finding is that model performance improves specifically as a function of how many frames a system is able to process, meaning current gaps are attributable in real part to how much of a long video a model actually samples rather than an even, length-independent competence [2]. Put together, the two benchmarks describe a compounding problem: fine-grained temporal distinctions are hard even in short clips, and coverage of a long clip is uneven and sampling-dependent, so a system’s report about anything late in an hour-long recording is systematically less trustworthy than its report about the opening minutes.
There is a structural reason this compounds rather than merely adding up, and it is worth making explicit because it exposes an assumption rather than a vague sense of difficulty. A vision transformer front end tokenises an image of height
Audio-visual grounding: hearing and seeing without connecting them
A system that processes both an image stream and an audio stream is not thereby a system that correctly associates the two — that a given sound belongs to a given visible source in a scene with more than one plausible candidate. Two benchmarks purpose-built to isolate this specific competence report that it fails in ways more basic than the “complex scene” framing might suggest.
AV-Odyssey Bench assembles 4,555 problems that require a model to draw on both visual and audio input jointly, and includes a component the authors call the “DeafTest”: pairs of questions asking only which of two sounds is louder, or which of two sounds has the higher pitch [3]. These are not audio-visual association tasks in the sense of locating a sound’s source in a scene — they are prior, more basic judgments about the audio signal alone, paired with the finding that current leading multimodal systems including GPT-4o, Gemini 1.5 Pro, and Reka Core “struggle with simple tasks humans find trivial” on exactly this kind of comparison [3]. If a system cannot reliably judge which of two sounds is louder, its report about which visible object a given sound belongs to in a cluttered scene is being built on an unreliable primitive, whatever the visual half of the system is doing correctly.
SAVVY addresses the association question more directly, targeting spatial reasoning that requires combining a moving visual scene with its accompanying audio over time. The paper’s own framing of the gap is direct: 3D spatial reasoning in dynamic, audio-visual environments “remains largely unexplored” by existing audio-visual large language models, which the authors report “predominantly focus on static or 2D scenes” rather than the moving, jointly-cued environments the task actually requires [4]. That is a statement about the state of the field’s training and evaluation focus, made by researchers building a benchmark specifically because the capability had not been measured, not a claim that models attempt and fail at dynamic audio-visual localisation — the more basic finding is that it had barely been tested until this benchmark existed. Cross-modal hallucination research adds a further, related data point: audio-visual systems have been found to “struggle to discern subtle relationships between audio and visual signals,” a difficulty attributed specifically to cross-interactions between the two modalities rather than to either one alone [5]. The pattern across all three papers is consistent — audio-visual grounding fails not primarily at the hard cases the phrase “complex scene” evokes, but at simple comparisons and at the basic task of connecting a moving sound to a moving source over time, both of which remain open by the field’s own account.
Spatial and 3D reasoning: a 2D prior meeting a 3D question
Most image training data is a flat photograph taken from an ordinary, camera-height, forward-facing viewpoint. Two benchmarks built to test spatial reasoning specifically — rather than object recognition, captioning, or general visual question answering — report that model performance depends heavily on how close a test image is to that ordinary viewpoint, which is direct evidence that the underlying representation is a learned photographic prior rather than a general grasp of three-dimensional structure.
BLINK evaluates fourteen classic computer-vision tasks reframed as multiple-choice questions, including relative depth estimation and multi-view correspondence — tasks the paper’s authors describe as solvable by humans “within a blink.” Humans average 95.70% accuracy on the benchmark. The paper’s own reported result for the best-performing systems at the time of testing was 51.26% for GPT-4V and 45.72% for Gemini, each only 13.17 and 7.63 percentage points above random guessing respectively [7]. A model scoring barely above chance on relative depth and multi-view correspondence is not making sophisticated errors at the margins of 3D understanding; it is largely guessing on tasks that require no domain expertise to solve correctly, only genuine access to spatial structure in the image.
3DSRBench narrows in on exactly why, with 2,772 manually annotated questions across height, location, orientation, and multi-object 3D reasoning. The paper’s stated finding is that a wide range of open-source and proprietary multimodal systems show weaknesses across all four categories, and — the detail that matters most for a training-distribution explanation — that performance is specifically “degraded on images with uncommon camera viewpoints” [6]. That is close to a controlled experiment on the hypothesis this section opened with: if 3D reasoning were a general capability the model had acquired, viewpoint should matter little; because it degrades exactly where the training distribution is thin, the more defensible reading is that these systems have learned strong photographic priors for canonical viewpoints and comparatively little of the underlying 3D structure a canonical photograph happens to make legible. Depth, orientation and multi-object position are recoverable from a single 2D image only with strong assumptions about scene geometry, and the benchmark evidence indicates those assumptions are not reliably encoded.
Cross-modal hallucination: confidently describing what was never there
Hallucination in text-only language models is well studied. Cross-modal hallucination is the specific case where a system describes visual or audio content — an object, a sound, a relationship between them — that is not actually present in the input it was given, and it is documented across image, video, and audio-visual systems with methodologically distinct benchmarks.
POPE, the earliest of the three cited here, evaluates object hallucination by asking binary questions of the form “is there a [object] in the image?” across random, popular, and adversarial object samples, and finds that representative large vision-language models “mostly suffer from severe object hallucination,” with the tendency worse for objects that occur frequently in training data or that co-occur with objects actually present in the image [8]. That co-occurrence finding is important: it indicates the failure is not simple noise but a systematic bias toward reporting objects the model’s prior expects to see, given what else is in the scene, independent of the actual image content.
HallusionBench extends this to a controlled, paired-question design across 346 images and 1,129 expert-written questions, distinguishing “language hallucination,” where a model’s answer is driven by linguistic priors rather than image content, from “visual illusion,” where a genuine misperception of the image drives the wrong answer. The paper’s reported accuracy is stark: the best-performing system tested, GPT-4V, reached 31.42% question-pair accuracy, and every other evaluated system scored below 16% [9]. A question-pair metric, which requires both the original and a matched control question to be answered correctly together, is a considerably harder bar than the single-question accuracy more commonly reported, and the size of the drop when that bar is applied is itself evidence about how much of ordinary benchmark performance rests on shortcuts that a paired design removes.
AVHBench extends the same underlying question into audio-visual systems specifically, reporting that the failures it measures are driven by “cross-interactions between modalities” — that is, hallucination that arises specifically from the difficulty of jointly interpreting audio and visual signals together, rather than from a weakness in either channel alone [5]. Taken together, these three benchmarks describe hallucination not as a single glitch but as a family of related failures with a common shape: a model reports content driven by what it expects, whether from language priors, from visual misperception, or from an unreliable audio-visual join, rather than content actually verified against the input.
Compositional generalization: combinations training never rehearsed
The fifth problem is different in kind from the first four. It is not that a specific pairwise combination — image and text, or audio and video — fails at some task. It is that systems trained predominantly on certain modality pairings generalise poorly to combinations that occur less often in training, including the joint combination of three modalities at once.
OmniBench tests exactly this: it requires a model to recognise, interpret, and reason across visual, acoustic, and textual input simultaneously, with questions constructed so that an accurate answer genuinely requires integrating all three rather than relying on any two [10]. The paper’s own reported finding is direct — open-source omni-language models show “critical limitations in instruction-following and reasoning capabilities within tri-modal contexts,” and most baseline systems perform below 50% accuracy even when given textual descriptions in place of the raw image or audio, which should make the task easier rather than harder [10]. The authors’ own explanation for the result is a training-data argument, not a capacity argument: the ability to construct a consistent context from text, image and audio together is, in their words, “often overlooked in existing MLLM training paradigms” [10]. That is a direct claim about what training distributions actually contain — heavily paired data for common combinations, and comparatively little data that forces a model to weigh three modalities against each other at once — offered by the researchers who built the benchmark specifically to surface the gap.
This connects back to the audio-visual grounding evidence from earlier in the article rather than standing apart from it. SAVVY’s authors made essentially the same point about dynamic audio-visual 3D reasoning: the capability is undertested because the combination it requires is undertrained [4]. A model can be excellent at image-plus-text and competent at audio-plus-text while still failing badly the moment a task requires all three simultaneously, or requires a pairing — video length beyond a few minutes joined with fine-grained audio timing, for instance — that existing datasets rarely construct. This is a genuinely different failure from the other four: it is not that a specific modality is poorly represented internally, but that the joint handling of specific combinations was never well rehearsed, and there is no way to know in advance which untested combination will behave like OmniBench’s tri-modal case rather than transferring cleanly.
What the five failures share
Line the five problems up and a pattern emerges that is more useful than treating each as an isolated gap. Every one of the ten benchmarks cited above was built because an earlier, easier evaluation of the same general area was not actually measuring the capability being claimed. TemporalBench and LongVideoBench exist because ordinary video question-answering benchmarks conflate general knowledge and single-frame recognition with genuine temporal reasoning. AV-Odyssey and SAVVY exist because audio-visual benchmarks had rarely isolated basic auditory judgment or dynamic 3D localisation from easier joint tasks. BLINK and 3DSRBench exist because general visual-question-answering accuracy does not separate spatial competence from object recognition. POPE and HallusionBench exist because free-form caption quality metrics do not surface systematic hallucination the way a binary or paired-control question does. OmniBench exists because pairwise multimodal benchmarks cannot detect a failure that only appears when three modalities must be weighed together.
In every case, the harder benchmark did not report a new kind of error. It reported the same kind of error the field already suspected, made impossible to average away. That is a meaningfully different claim from “multimodal AI has rough edges,” and it is also a more optimistic one in a specific, narrow sense: each of these five problems has a documented instrument for measuring it, which is the necessary precondition for tracking whether a future system has actually closed the gap rather than merely improved its score on an easier proxy.
Predictions, with the observations that would falsify them
These are forecasts, kept separate from the sourced findings above. Horizon: 15 August 2028. Assumptions: no architectural discontinuity comparable to the arrival of the transformer, continued publication of specialised benchmarks rather than a return to pooled multimodal averages, and continued public release of benchmark construction details sufficient to check for shortcut solutions.
One. Cross-modal hallucination on static images will narrow measurably faster than the other four problems, because it is the one where mitigation techniques — the paired-control design itself, plus decoding-time interventions already published against POPE and HallusionBench — directly target the failure mode rather than requiring new training data. Indicator: published question-pair accuracy on HallusionBench-style paired evaluations rising substantially above the 2024 range. Disconfirmed if leading systems in 2028 still score below 40% on a comparably strict paired-control hallucination benchmark.
Two. Fine-grained temporal reasoning in long video will remain the slowest-narrowing of the five, because the token-budget trade described above is a structural property of patch-and-frame tokenisation rather than a training-data gap that more data alone resolves. Indicator: continued large gaps between short-clip and long-clip accuracy on fine-grained temporal benchmarks even as raw context length grows. Disconfirmed if a general-purpose system matches human-level accuracy on TemporalBench-style ordering and frequency questions at durations beyond ten minutes without a task-specific temporal architecture.
Three. Tri-modal and other under-represented combinations will continue to underperform pairwise-trained baselines specifically on questions engineered to require joint use of all inputs, because the underlying cause is a training-distribution gap that grows, not shrinks, as the number of possible modality combinations increases. Disconfirmed if a 2028 omni-modal system reaches the accuracy of its best single-pair baseline on a tri-modal-only benchmark comparable to OmniBench.
Four. Audio-visual grounding in dynamic, multi-source scenes will remain measurably behind static audio-visual matching, because the additional requirement — tracking correspondence as both the sound and its source move — has only recently acquired a dedicated benchmark and no widely adopted training objective yet targets it directly. Disconfirmed if a general-purpose system published before 2028 reports DeafTest-level accuracy on basic auditory comparisons at or above 90% alongside strong performance on a dynamic source-localisation benchmark such as SAVVY-Bench.
What to take away
None of the five problems in this article is evidence that multimodal AI is not working; each is evidence that a specific claim about it is not yet true, made by the researchers who built the instrument capable of showing that. Long video degrades in temporal fidelity well before it exhausts context. Audio and vision can each be processed competently while their join remains unreliable, down to basic auditory comparisons. Spatial and 3D reasoning tracks how close an image is to an ordinary photographic viewpoint rather than reflecting a general grasp of geometry. Systems describe content that is not present with a confidence that these benchmarks show is unwarranted. And combinations of modalities a training set rarely paired together are not a safe extrapolation from combinations it paired constantly.
The discipline this article tried to hold to generalises past these five cases. A claim that a capability is solved should be checked against the hardest, most recently built benchmark specifically designed to isolate it — not against a pooled average that a shortcut can quietly dominate. Where that benchmark exists and reports a gap, the gap is the finding. Where it does not yet exist, that absence, as several of the papers above note about their own subject before they built their instrument, is itself informative.
