Ten Failure Modes That Define Multimodal AI Systems

Ten case files, one filing cabinet

A multimodal system’s failures rarely announce themselves. The model still answers. It still describes the picture, still transcribes the clip, still proposes the next click in a browser session — and the sentence it returns is grammatical, specific, and delivered with the same even confidence whether it is right or not. What breaks is not the sentence. What breaks is the correspondence between that sentence and whatever was actually in the image, the audio, the depth map, or the second document the model was supposed to be checking against the first. Reading the output tells you almost nothing about whether that correspondence held.

This article catalogues ten distinct, independently documented ways that correspondence comes apart in deployed and research-grade multimodal systems. Each is drawn from a benchmark paper, a security study, or a training-dynamics analysis, not from anecdote, and each has its own detection method and its own root cause — they are not restatements of one underlying problem. Two caveats before the list. First, “multimodal” here covers vision-language, audio-visual, and agentic systems that consume more than one channel; the evidence is strongest for vision-language pairs because that is where the most benchmark infrastructure exists, and this article says so wherever the evidence for another pairing is thinner. Second, these are documented tendencies with measured rates on specific benchmarks, not universal constants — a model’s exposure to any one of them depends on its training data, its fusion architecture, and the task distribution it is actually run on.

1. Cross-modal hallucination: describing what is not there

The most-studied failure is also the most basic: a model generates a confident description of an object, sound, or event that the input does not contain. Rohrbach and colleagues gave this a name and a metric in image captioning — CHAIR, which checks generated object mentions against a verified ground-truth object list — and their central finding cuts against intuition: models that score best on standard fluency and similarity metrics do not reliably hallucinate less, and captions with more hallucinated content are disproportionately driven by language priors rather than by visual misclassification ^[2]. That is the mechanism in miniature. The model is not failing to see; it is completing a caption the way captions in its training distribution usually go, and the completion is frequently correct because co-occurring objects usually do co-occur — until they do not.

Li and colleagues extended the diagnosis with POPE, a polling-based evaluation that asks a model direct yes/no questions about whether specific objects are present, precisely to route around the confound that free-form caption metrics reward fluency as much as accuracy. Their finding was that hallucination is not a rare edge case but a widespread property of the large vision-language models they tested, concentrated on objects that appear frequently in training data or that habitually co-occur with what is actually in the image ^[1]. The practical consequence: a caption reading like a careful visual description can be, in measurable part, a language model’s prior about what usually accompanies what it did see, asserted with the same syntax as an observation.

2. Modality collapse: answering from the prior, not the input

A subtler and architecturally deeper failure is that a jointly trained multimodal model can systematically under-use one of its input channels — not because that channel is uninformative, but because gradient descent found it easier to solve the training objective through the other one. Peng and colleagues documented this directly: in multimodal discriminative training with a single uniform objective across modalities, one modality routinely dominates optimization — their examples are sound over vision when wind is blowing, or vision over sound in a drawing task — leaving the other modality’s encoder measurably under-optimized relative to what it could achieve on its own ^[3]. Their fix, on-the-fly gradient modulation that monitors each modality’s relative contribution to the loss and rebalances updates accordingly, is itself evidence for the diagnosis: if imbalance were not real and correctable, monitoring and rebalancing gradients per modality would not have produced a measurable gain.

This differs from cross-modal hallucination in an important way. Hallucination is a symptom visible in the output — a wrong noun in a caption. Modality collapse is a property of the trained weights themselves: the pathway carrying one channel’s information into the shared representation was never driven hard enough during training to matter much at inference, regardless of what that channel actually contains on a given input. A model in this state can pass a large fraction of a benchmark by pattern-matching on the dominant modality while the nominally multimodal channel does almost no work — which is a different failure from the model actively inventing content, even though both can produce the same wrong answer.

3. Audio-visual misalignment in generated and processed content

Where two modalities must be produced or processed in lockstep — a mouth shape and a phoneme, a footstep and a frame of contact, a musical downbeat and a cut — small timing errors are much easier to generate than to notice automatically, because most quality metrics score each channel’s plausibility in isolation rather than their joint timing. Liu and colleagues built JavisDiT specifically to close that gap for text-to-audio-video generation, introducing both a hierarchical spatio-temporal synchronization mechanism inside the model and a dedicated benchmark, JavisBench, along with a purpose-built synchrony metric, because — as their own framing states — existing evaluation approaches were not adequate to catch audio-visual misalignment in generated content ^[4]. That a dedicated architecture, benchmark, and metric were judged necessary is itself the documentation: joint audio-video generation does not get synchronization for free from strong per-channel quality, and the authors report their approach outperforming prior joint-generation methods on the synchrony measure they built to detect exactly this failure ^[4].

The general shape of the problem extends past generation into any pipeline that processes audio and video as loosely coupled streams — dubbing, transcription-driven captioning, and multi-camera event reconstruction all inherit the same risk, because nothing in a two-tower architecture with separately timed encoders guarantees that a claim made about the audio track and a claim made about the video track are talking about the same instant. This is the one failure mode in this catalogue where the evidence base is newer and thinner than the others; treat the mechanism as well-established and any single number about its frequency in a specific deployed product as unverified until that product publishes it.

4. Spatial reasoning errors from 2D training applied to 3D tasks

Vision-language models are trained overwhelmingly on flat, single-view images paired with captions that describe what is visible, not the three-dimensional geometry that produced that view. Three independent benchmarks converge on the same finding: models handle a canonical, common viewpoint tolerably and degrade sharply once genuine spatial or depth reasoning is required. Fu and colleagues built BLINK around fourteen classic computer-vision perception tasks — including relative depth and spatial relations — that “humans can solve within a blink,” and found a human average of 95.70% accuracy against 51.26% for GPT-4V and 45.72% for Gemini, both only marginally above the 38.09% random-guessing baseline ^[5]. Liu and colleagues’ Visual Spatial Reasoning dataset, covering sixty-six types of spatial relation, found a human ceiling above 95% against roughly 70% for the best models, and — the more structurally important result — by-relation performance showed little correlation with how many training examples of that relation the model had seen, meaning the gap does not close simply by adding more paired data of the same kind ^[6]. Ma and colleagues’ 3DSRBench went further by explicitly separating common from uncommon camera viewpoints of the same scene, and found large multimodal models showing significantly degraded 3D spatial reasoning specifically on the uncommon-viewpoint condition ^[7].

Read together, these results support a specific interpretation, offered here as analysis rather than as a finding any one paper states outright: a model trained on the photographer’s-eye-level, canonical-angle distribution of internet images can learn surface statistics that mimic spatial understanding on the views it has seen, without acquiring anything that generalizes to genuine 3D geometry. Turn the object, change the camera height, or ask about occluded relations, and the mimicry runs out.

5. Generation artifacts specific to multimodal outputs

A still image either looks right or does not, on its own. A generated video adds a dimension standard per-frame quality checks were never built to catch: whether an object stays the same object across time. Huang and colleagues built VBench because evaluating video generation on a single aggregate score conflates dimensions that fail independently, and they explicitly break quality into sixteen separate dimensions, including temporal flickering and motion smoothness, so that a model’s strengths and weaknesses across dimensions are visible rather than averaged away ^[8]. That a comprehensive benchmark needed sixteen separable dimensions rather than one is itself evidence that per-frame plausibility and cross-frame consistency are different properties that current models do not automatically deliver together — a clip can consist of individually convincing frames whose subject subtly changes shape, colour, or identity from one frame to the next, a failure invisible to any metric scored frame-by-frame in isolation.

6. Benchmark-gaming via unimodal shortcuts

A benchmark score that looks like evidence of multimodal competence can be, in substantial part, a text-only score wearing a multimodal label. Chen and colleagues audited widely used vision-language benchmarks for exactly this and found it was not a marginal effect: GeminiPro reached 42.9% on MMMU with no visual input at all, models with images removed beat random-choice baselines by more than 24% on average across six benchmarks, and Sphinx-X-MoE scored 43.6% on MMMU without accessing images — 17.9 points above its own language-model backbone run alone, evidence of leakage rather than mere language-prior guessing ^[9]. Their response was MMStar, a benchmark hand-curated for genuine visual necessity.

The finding motivates a simple, checkable decomposition rather than a new metric: for any multimodal benchmark, let $s_{\text{multi}}$ be the score with all modalities present and $s_{\text{text}}$ be the same benchmark scored with the non-text modality removed. The quantity

is the actual multimodal increment — what the removed channel contributed beyond what text alone already bought. A large $s_{\text{multi}}$ with a small $\Delta$ is not evidence of multimodal competence; it is evidence that the benchmark, not the model, is doing something unimodal. Reporting $\Delta$ alongside $s_{\text{multi}}$ costs one extra evaluation run and turns an unfalsifiable headline number into a checkable one.

7. Brittleness to modality-specific adversarial perturbations

Adding an image or audio channel to a language model does not just add a capability; it adds an attack surface with its own physics, one that safety training conducted primarily on text does not automatically cover. Qi and colleagues showed that a single adversarially optimized image — imperceptible or near-imperceptible perturbation layered onto an otherwise ordinary picture — could universally jailbreak an aligned vision-language model, inducing it to comply with a wide range of harmful text instructions it would refuse without the image present, with the effect generalizing across instructions rather than merely reproducing one memorized harmful example ^[10]. Bailey and colleagues generalized the attack beyond jailbreaking into arbitrary behavioral control: their “image hijacks,” produced with a Behaviour Matching algorithm, achieved reported success rates over 80% against LLaVA across several distinct attack objectives — forcing specific outputs, leaking context, and overriding safety behavior — using source images from generic datasets unrelated to the target behavior, which the authors present as evidence the attack does not depend on any special relationship between the image and what it is being used to force ^[11].

The structural point, not specific to either paper, is that continuous-valued channels such as pixels or audio waveforms admit gradient-based optimization in a way discrete text tokens mostly do not, so an attacker with white-box or even partial access to a vision-language model has an optimization surface for control that a text-only interface denies them. Defending the language side of a multimodal model without separately hardening the added channel leaves that surface open.

8. Degraded performance on low-resource modality combinations

Multimodal degradation is not evenly distributed across language, and the gap is not really about the image or audio encoder being worse — it is about how thin the paired training data is for a given modality combination. Geigle and colleagues built Babel-ImageNet by translating ImageNet labels into a hundred languages without relying on machine translation, then evaluated eleven public multilingual CLIP models against it, finding performance gaps between English and other high-resource languages, and a further, steeper drop for low-resource languages specifically — a gap partially closable with parameter-efficient, language-specific training but present as a baseline condition across the models tested ^[12]. The image encoder itself is not the bottleneck; the same pixels are being classified regardless of the label language. What is thin is the paired image-language training data for anything other than a handful of high-resource languages, so the alignment between the vision representation and a given language’s text representation is only as good as how much of that pairing existed to train on.

The consequence generalizes past image-language pairs to any modality combination that is individually well-served in isolation but rare in combination — a common spoken language paired with an uncommon dialect’s script, or a well-covered object category photographed in an underrepresented setting. A system’s overall multimodal competence, reported as one number, can conceal an uneven map in which some channel pairings are solid and others are barely trained at all.

9. Failing to detect conflicting information across modalities

A model that generates fluent, internally consistent text about one channel is not the same as a model that checks whether two channels actually agree, and current systems are demonstrably weak at the second, distinct task. Popordanoska and colleagues built CLASH by pairing real images with captions containing controlled, deliberately introduced object- or attribute-level contradictions, then tested whether state-of-the-art models could catch them; the models showed substantial limitations in recognizing cross-modal conflicts, along with systematic modality-specific biases — meaning some models default toward trusting the text, others toward trusting the image, rather than flagging the disagreement — and targeted fine-tuning on the benchmark measurably improved detection, which is itself evidence that ordinary training does not teach this behavior ^[13]. Tian and colleagues’ CrossCheck-Bench, built independently and around the same time, reached a compatible conclusion with a different design: evaluating thirteen state-of-the-art models across three tiers of reasoning complexity, most models performed well at isolated entity recognition but showed a consistent, tier-by-tier performance drop as the task moved from perceptual matching toward genuine logical contradiction detection, and standard chain-of-thought prompting produced only marginal improvement ^[14].

The distinction from cross-modal hallucination matters. Hallucination is the model asserting something false about one channel considered alone. Conflict-detection failure is the model holding two channels that plainly disagree with each other and generating a fluent answer anyway, without anything in its processing flagging the contradiction — a failure mode invisible to any evaluation that only checks a single-channel claim against ground truth, because both channels can individually be read as plausible.

10. Compounding errors in downstream agentic pipelines

The most expensive version of these failures is the least visible one at the moment it occurs: a multimodal perception error that feeds a multi-step agentic pipeline does not cost what it would cost in isolation, because every downstream step that trusts it inherits the mistake and can amplify it. Zheng and colleagues’ SeeAct system, built around GPT-4V as a web agent, makes the point with an unusually clean natural experiment: the same model’s high-level plans succeeded on 51.1% of live-website tasks when grounded manually — that is, when a human converted the model’s textual description of the intended action into the actual click or keystroke — but the authors report that grounding, translating a correct visual-and-textual plan into a correct action on the actual page, “remains a major challenge” when performed automatically, with substantial gaps persisting between every automated grounding strategy tested and this manual-grounding ceiling ^[15]. The plan can be right and the task can still fail, because the failure was never in the reasoning; it was in the translation of a multimodal reading into a concrete action, one link in the chain the reasoning never gets a chance to verify or repair.

That structure — a correct-sounding intermediate judgment, an unverified translation step, and no mechanism for a later stage to catch an earlier stage’s mistake — is what makes this failure mode compounding rather than merely additive. A single-turn captioning error is bounded: it produces one wrong sentence. The same category of error inside an agentic loop that clicks, reads a new screen, and clicks again can steer an entire multi-step trajectory away from the intended task before anything downstream has a reason to doubt the step that caused it.

What actually catches each one

None of these ten is found by re-reading the model’s output more carefully; each requires a specific check built for it. Cross-modal hallucination is caught by object-presence polling against verified ground truth, not by grading caption fluency. Modality collapse is caught by measuring each modality’s marginal contribution to the loss or to accuracy, not by reading which channels the prompt mentions. Audio-visual misalignment is caught by a dedicated synchrony metric operating on the joint signal, not by grading each channel’s standalone quality. Spatial-reasoning failure is caught by testing uncommon and canonical viewpoints of the same scene side by side. Frame-level generation artifacts are caught by dimension-separated video metrics, not one aggregate score. Unimodal shortcuts are caught by the ablation in Section 6 — run the benchmark with the channel removed and report the difference, not the headline. Adversarial brittleness is caught by red-teaming the added channel specifically, with the same seriousness given to text-based jailbreak testing. Low-resource degradation is caught by breaking a single aggregate score out by language or modality pairing before publishing it. Conflict blindness is caught by benchmarks built with deliberately injected contradictions, since organically contradictory pairs are rare in curated data. And compounding agentic error is caught only by instrumenting the pipeline itself — logging what each stage was given, not just what the final action was — because the transcript of a failed task reads exactly like the transcript of a successful one until the one step where they diverge.

Predictions, with the observations that would falsify them

These are forecasts, kept separate from the sourced analysis above. Horizon: 15 August 2029. Assumptions: continued commercial deployment of vision-language and audio-video systems at current or greater scale, no architectural discontinuity that eliminates learned continuous-valued perception encoders, and continued public benchmark publication.

One. Text-only and image-removed ablation scores will become a standard reported line for vision-language benchmarks, following the trajectory MMStar and MMMU-Pro-style audits started. Indicator: major new vision-language benchmarks publishing an ablated baseline by default. Disconfirmed if leading 2029 benchmarks still report a single headline multimodal number with no unimodal control condition available.

Two. Dedicated cross-modal contradiction and conflict-detection benchmarks, following CLASH and CrossCheck-Bench, will multiply faster than general perception benchmarks, because conflict detection is the capability most clearly absent rather than merely weak. Indicator: a measurable rise in published contradiction-focused multimodal benchmarks year over year. Disconfirmed if the count of such benchmarks plateaus or declines relative to general vision-language benchmarks over the horizon.

Three. Grounding — translating a correct multimodal judgment into a correct action — will remain the dominant bottleneck in agentic pipelines relative to perception or reasoning quality, because SeeAct’s gap between planned and grounded success has a structural cause (translating continuous visual layout into discrete actions) rather than a data-scale cause. Indicator: published agent benchmarks continuing to show large gaps between oracle-grounded and fully automated success rates. Disconfirmed if that gap closes to within a few points for a general-purpose visual web or GUI agent without a task-specific grounding model bolted on.

Four. Adversarial robustness work on the added modality will remain behind text-based safety work by a measurable margin, because the continuous optimization surface Section 7 describes is a standing structural asymmetry, not a temporary gap in research attention. Indicator: published attack success rates against the visual or audio channel of aligned models remaining substantially higher than published rates against text-only jailbreak attempts on the same models. Disconfirmed if red-teaming reports show the two converging to comparable difficulty for the attacker.

What to take away

Ten failure modes, and not one of them is fixed by making the model more fluent, because fluency was never the thing that was broken. Each is a specific, checkable place where a claim generated from one or more channels stops corresponding to what those channels actually contained, and each has its own instrument: an ablation, a viewpoint pair, a dimension-separated metric, an injected contradiction, a pipeline log. Treat a multimodal system’s headline competence the way this catalogue treats it — as ten separable, separately falsifiable claims rather than one — and ask, for whichever of the ten matters to your use case, which specific check was run and what it found. A system that has not been checked for a given failure mode has not been shown not to have it.