One Model, Many Modalities: What Multimodal Systems Actually Share

The join is the claim

A four-colour print is not one picture. It is four monochrome impressions, each produced on its own plate from its own separation, laid one over another on the same sheet. Every plate is a complete and self-consistent object, and none of them is the image. The image appears only when the four register, and the difference between a good print and a ruined one is often a fraction of a millimetre at the pins.

Contemporary multimodal systems are built the same way, and the analogy is closer than it is comfortable to admit. There is an encoder or tokeniser for each modality, trained on data of that modality with objectives suited to it. There is an alignment procedure that puts two of those encoders into correspondence. There is a fusion point, somewhere in the stack, where the representations meet. And there is a claim, made on specification sheets and in launch posts, that the result is one model that understands text, images, audio and video.

That claim is worth taking apart, because it bundles at least four separable propositions: that non-text inputs can be represented in a form the model can consume; that those representations are meaningfully aligned with the text representation; that competence acquired in one modality transfers to another; and that the resulting system can be measured as a whole. The first is straightforwardly true and well understood. The second is true in a narrower sense than the phrase “shared representation space” suggests. The third is largely false in the strong form people assume. The fourth is where most published evidence quietly breaks down.

This article goes through the stages in order — tokenisation, alignment, fusion, transfer, evaluation, and the separate question of generation — and ends where the printing works ends: with the observation that registration is a per-job achievement, not a property of the press.

Tokenising what was never a sequence

A transformer consumes a sequence of vectors. Text arrives nearly in that form already; the byte or subword stream is discrete, ordered, and finite. Nothing else does. The first engineering problem in any multimodal system is therefore to manufacture a sequence out of something that is not one.

For images the dominant answer is patchification, introduced at scale by the Vision Transformer: cut the image into a grid of non-overlapping square patches, flatten each, and project it linearly into the model’s embedding dimension, so that a pure transformer applied directly to sequences of image patches performs competitively with convolutional networks when pre-trained on enough data ^[2]. The token count follows immediately from the geometry,

for an image of height $H$ and width $W$ at patch size $P$. The quadratic relationship is the entire practical story of image tokenisation. Halving the patch size quadruples the sequence, and attention cost grows faster still. Every deployed system therefore resolves a three-way tension between input resolution, patch size, and context budget, and it resolves it by throwing away detail.

For audio the answer is usually a learned discrete codec. The vector-quantised variational autoencoder established the pattern: an encoder produces continuous vectors, each is replaced by the nearest entry in a learned codebook, and the resulting indices are a discrete sequence that an autoregressive model can treat exactly as it treats text ^[3]. Modern neural audio codecs extend this with residual quantisation and adversarial training, producing streaming encoder–decoder architectures with a quantised latent space that reconstruct speech and music at low bitrate ^[4]. The alternative path treats audio as an image — a log-mel spectrogram fed to a sequence model — which is how large-scale weakly supervised speech recognition has been trained at the scale of 680,000 hours of multilingual audio ^[5].

Video is the case where the arithmetic becomes punishing rather than merely inconvenient. A sequence of frames multiplies the per-image token count by frame count, so practical systems subsample frames aggressively, pool across time, or compress spatio-temporally before the model sees anything. Whatever the method, the model is not watching the video. It is reading a summary of it produced by a component that was not trained to know what the user was going to ask.

What the tokeniser has already decided

This is the part of multimodal engineering that is systematically underdiscussed, and it deserves stating plainly: the tokeniser fixes an upper bound on what the model can ever notice, and it does so before any language model is involved.

Three losses are structural rather than incidental. A patch grid imposes a fixed spatial quantisation, so any distinction finer than a patch survives only if the linear projection happens to preserve it. A codebook is lossy by construction — quantisation replaces a continuous vector with its nearest codebook entry, and everything separating them is discarded, permanently, with no downstream component able to recover it ^[3]. And any subsampling in time removes events shorter than the sampling interval, which is why questions about brief transients in long video are answered badly by systems that answer questions about static scenes well.

There is direct evidence that these bounds bite in deployed systems. Tong and colleagues identified what they call CLIP-blind pairs — images a CLIP encoder embeds as similar despite obvious visual differences — assembled them into a benchmark of basic visual patterns, and reported that state-of-the-art multimodal systems including GPT-4V struggle with straightforward questions built on them, with a notable correlation between the visual patterns that challenge CLIP and those that challenge the multimodal models built on top of it ^[19]. That is the clearest available demonstration that the visual front end’s invariances propagate upward as the whole system’s blind spots. The authors’ proposed remedy — mixing self-supervised visual features in alongside the contrastively trained ones — is itself an admission that no single encoder’s representation was sufficient ^[19].

Contrastive pretraining, and what it actually optimises

Alignment is the step that made cross-modal retrieval work, and it has an unusually clean formulation. Contrastive language–image pretraining takes a batch of $B$ image–text pairs, encodes each side separately into a normalised vector, and trains both encoders so that matched pairs score higher than mismatched ones. In its softmax form the objective is

with image embedding $u_i$, text embedding $v_i$, and a learned temperature $\tau$. Radford and colleagues showed that this simple pre-training task, applied to 400 million image–text pairs collected from the internet, matched the accuracy of the original ResNet-50 on ImageNet zero-shot without using any of the 1.28 million training examples that network was trained on ^[1]. A later variant replaces the softmax with a pairwise sigmoid loss that does not require a global view of the batch’s pairwise similarities for normalisation, and the authors report training a model to 84.5% ImageNet zero-shot accuracy in two days on four TPUv4 chips, while also finding that the benefit of larger batches saturates around 32,000 examples ^[6].

Read the objective again and notice what it does not say. It constrains a ranking within a batch. It says nothing about the internal geometry of either embedding space, nothing about how the two spaces should be positioned relative to each other beyond making matched pairs closer than mismatched ones, and nothing at all about the compositional structure of the caption. Whatever features suffice to win the in-batch ranking are the features that get learned.

The consequences are visible and reproducible. Yuksekgonul and colleagues built a benchmark of more than 50,000 test cases probing attribution, relation and word order, found that vision-language models show severe deficiencies in relational understanding and a severe lack of order sensitivity, and — crucially — showed that it is possible to perform well on retrieval over existing datasets without using composition or order information at all ^[15]. Winoground makes the same point with a harder instrument: two images and two captions containing an identical set of words in different orders, matched correctly or not. Of the models tested, none did much better than chance ^[14].

These are not marginal failures at the frontier of the task. They are the objective working exactly as specified, on a task that rewarded bag-of-words matching because bag-of-words matching was sufficient.

Where the representations actually meet

Given aligned or alignable encoders, a system designer chooses a fusion point, and the choice determines almost everything about cost, modularity and capability.

Late fusion with cross-attention keeps both towers frozen and inserts trainable layers that let the language model attend to visual features. Flamingo is the reference design: it bridges powerful pretrained vision-only and language-only models and handles sequences of arbitrarily interleaved visual and textual data, without retraining either tower ^[7]. Late fusion through a learned bridge compresses further. BLIP-2 connects a frozen image encoder to a frozen large language model through a lightweight querying transformer trained in two stages, and its authors report outperforming Flamingo80B by 8.7% on zero-shot VQAv2 with 54 times fewer trainable parameters ^[8]. Late fusion through a projection plus instruction tuning is simpler still: LLaVA maps visual features into the language model’s token space and trains on machine-generated multimodal instruction data, an approach whose appeal is that most of the work is done by data rather than architecture ^[9].

Early fusion refuses the separation. Chameleon is described by its authors as a family of early-fusion token-based mixed-modal models capable of understanding and generating images and text in any arbitrary sequence, requiring — in their account — a stable training approach from inception and an architectural parameterisation tailored to the mixed-modal setting ^[10]. That framing is worth reading carefully: the emphasis on stability from inception is a statement that early fusion is harder to train, not merely more expensive.

The trade is legible once stated. Late fusion is cheap, modular, and lets each tower be upgraded independently — but everything the language model learns about the image must pass through the bridge, and the bridge has finite channel capacity. A fixed number of query vectors is a hard bound on how much visual information reaches the text side, no matter how good the encoder was. Early fusion removes that bottleneck and pays for it in training cost, data requirements, and instability, and it forfeits the ability to swap a component.

Both are engineering choices, not discoveries about cognition. A system’s specification sheet reports which modalities go in. It rarely reports where they meet, and where they meet is what determines whether fine visual detail can influence a textual answer at all.

Is the shared representation space a real claim?

“Shared representation space” is the phrase that does the most rhetorical work in multimodal marketing, and it is worth asking whether it survives contact with measurement.

The strongest direct evidence says it needs qualification. Liang and colleagues documented what they named the modality gap: in models like CLIP, different data modalities are embedded at arm’s length in their supposedly shared representation, occupying separate regions rather than intermingling. They attribute this to two causes — deep network representations are confined to a narrow cone at initialisation, so the modalities begin apart, and contrastive optimisation preserves that separation at a distance influenced by the temperature — and they show that deliberately changing the gap distance measurably affects downstream zero-shot performance and fairness metrics ^[12]. A space in which two modalities occupy disjoint cones, and in which moving them changes behaviour, is better described as two spaces with a learned correspondence than as one space.

The opposing view is not that the gap is absent but that it is superficial relative to a deeper convergence. Huh and colleagues argue that representations in deep networks are converging over time and across domains, that vision and language models measure distance between datapoints in increasingly similar ways as they grow, and they hypothesise a shared statistical model of reality toward which training pressures push — while explicitly discussing limitations and counterexamples to their own analysis ^[13].

These positions are less contradictory than they first appear, and characterising the disagreement precisely matters more than picking a side. The convergence claim is about relational structure — whether two models induce similar distances between pairs of concepts. The modality-gap finding is about absolute placement — where each modality’s embeddings sit. Two encoders can agree closely on which things are similar to which while still occupying separate regions of the vector space. The honest summary is that a converging similarity structure is a genuine and interesting empirical finding, while a single unified space in which an image and its caption land in the same place is not what current contrastively trained systems produce. Anyone who needs the second property should verify it in their own embeddings rather than assume it from the phrase.

What transfers, and what demonstrably does not

Separating the transfer question into two kinds resolves most of the confusion around it.

What transfers reliably are methods. The transformer architecture transfers. So does the recipe of tokenise-then-predict-the-next-token, the finding that weak supervision at very large scale beats curated supervision at small scale — demonstrated for speech at 680,000 hours with zero-shot transfer competitive with fully supervised results ^[5] — and the general machinery of scaling, optimisation and instruction tuning. This is real and it is why progress in one modality has repeatedly accelerated progress in others.

What does not transfer reliably is competence. There is no published evidence that a model’s linguistic compositional ability confers visual compositional ability. The evidence points the other way: models with strong language capabilities still fail visio-linguistic compositional matching at close to chance ^[14], still behave like bags-of-words on attribution and relation ^[15], and still fail elementary visual discriminations that their language competence cannot rescue ^[19]. Nor does perceptual acuity transfer downward: a strong vision encoder does not make a weak language model reason better about what it sees.

The asymmetry has a practical consequence. When a system is upgraded and its text benchmarks improve, the reasonable prior is that its visual discrimination has not changed at all, because the visual tower and the bridge may be identical. That prior should be tested, not assumed — but it is the right starting point, and it is the opposite of what a version number implies.

The evaluation problem is not a detail

If one section of this article should change a reader’s behaviour, it is this one. A substantial fraction of the multimodal benchmark literature measures something other than multimodal ability, and the effect is large enough to invert conclusions.

The problem was identified early. Goyal and colleagues observed that bias in language is a simpler signal for learning than visual modalities, producing models that ignore visual information, and rebuilt the visual question answering dataset so that each question is paired with two similar images yielding two different answers. All models performed significantly worse on the balanced dataset, confirming that they had learned to exploit language priors ^[16]. That was 2017, and the lesson did not stick.

The most direct recent measurement is Chen and colleagues, who audited current benchmarks for two failure modes. First, visual content is unnecessary for many samples — they report that GeminiPro achieves 42.9% on MMMU with no visual input at all, and that models without images outperform the random-choice baseline across six benchmarks by over 24% on average. Second, unintentional data leakage: Sphinx-X-MoE reaches 43.6% on MMMU without accessing images, exceeding its own language-model backbone by 17.9%. Their response was to hand-select 1,500 vision-indispensable samples into a new benchmark ^[17]. Independently, the MMMU authors rebuilt their own benchmark by filtering text-only-answerable questions, augmenting the option sets, and adding a vision-only condition; model performance on the revised version is substantially lower, ranging from 16.8 to 26.9 percentage points below the original across models ^[18].

Take those two results together. A headline multimodal score is, in part, a text score — and the text component can be large enough to dominate comparisons between systems. This is exactly the situation of a proof in which the black channel alone reproduces most of the picture: the print looks nearly right with one plate, and grading the four-colour job tells you mostly about that one plate.

Three rules follow, and they are cheap to apply. Run the text-only ablation. Feed the benchmark to the system with the images removed and report that number beside the headline; the difference, not the headline, is the multimodal gain. Prefer benchmarks with a published vision-indispensability audit, since these now exist ^{[17, 18]}. Never rank vendors across differently constructed multimodal suites, because the shortcut fraction differs per suite and a ranking built from incomparable benchmarks measures benchmark construction as much as it measures models.

Generation and understanding are not one capability

The last conflation is between producing a modality and comprehending it, and the field’s own architectures argue against treating them as one thing.

A contrastive encoder can support retrieval and classification over images while being structurally incapable of producing one ^[1]. A bridged language model can describe an image in detail while having no image decoder anywhere in the stack ^{[8, 9]}. A codec-plus-autoregressive-model stack can synthesise audio without any component trained to answer questions about it ^{[3, 4]}. Unified token-based systems such as Chameleon are the attempt to fold both into one sequence model, and their authors present understanding and generation as jointly achieved rather than automatically shared, alongside a training recipe developed specifically to make that joint objective stable ^[10].

There is also a representational argument for why the two pull apart. Generation rewards a tokenisation that reconstructs faithfully, since anything the codebook discards cannot be synthesised. Understanding rewards a representation that discards nuisance variation, since invariance is what makes classification and description robust. These are opposing pressures on the same encoder, and the mixture-of-features remedy proposed for visual shortcomings is a direct response to exactly that tension ^[19].

For a reader assessing a product, the operational form is simple: an “image capability” may mean input, output, or both, and a system’s competence at one says close to nothing about the other. The same holds for audio, where speech recognition, speech synthesis, and non-speech audio understanding are three separate capabilities routinely described with one word.

Predictions, with the observations that would falsify them

These are forecasts, separated deliberately from the sourced analysis above. Horizon: 8 August 2028. Assumptions: no architectural discontinuity comparable to the arrival of the transformer, continued commercial pressure toward broad modality coverage, and the continued public availability of benchmark construction details.

One. Text-only ablations will become a standard reported line in multimodal evaluations, in the way that held-out contamination checks became standard for language benchmarks. Indicator: major system cards reporting a no-image baseline beside each multimodal score. Disconfirmed if leading 2028 multimodal system cards still publish headline benchmark numbers with no image-ablated control.

Two. Fine-grained visual discrimination will remain the weakest axis of general multimodal systems relative to their language ability, because the tokenisation bottleneck and the contrastive objective’s insensitivity to fine detail are both structural. Indicator: continued large gaps on perception-focused benchmarks even as reasoning benchmarks saturate. Disconfirmed if a general-purpose system reaches human-level accuracy on a vision-indispensable perception benchmark without a task-specific visual component.

Three. Early-fusion token-based designs will grow as a share of newly announced systems, driven by generation requirements rather than by understanding requirements. Indicator: new systems that generate images natively describing a unified token stream rather than a bolted-on decoder. Disconfirmed if the frozen-encoder-plus-bridge pattern remains the dominant published architecture for new generative multimodal systems in 2028.

Four. The phrase “shared representation space” will continue to be used descriptively while the measured modality gap persists in production contrastive encoders. Disconfirmed if published measurements show the gap closing to within embedding noise in widely deployed image–text encoders.

None of these requires a breakthrough or a collapse. They follow from the structure already visible: separate front ends, a bounded join, and an evaluation culture that has only recently begun to check whether the image was needed.

What to take away

The vendor claim, when it is made carefully, is a claim about an interface: this endpoint accepts images, audio and video as well as text ^[11]. That is a real and useful engineering achievement, and it should be credited as one. It is not a claim that a single competence spans the modalities, and it is usually not presented as one — the elision happens downstream, in procurement documents and in summaries.

So treat “multimodal” the way a printer treats a four-colour job. Ask which plates exist, at what screen ruling, and how much detail each one can actually hold. Ask where they register and how much slack there is at the pins. Ask whether the proof was judged with all four channels or whether one of them was carrying the picture. And ask separately whether the press can lay a channel down at all, because reading a separation and printing one are different operations performed by different equipment.

The image is never in the plates. It is in the registration, and registration is checked one job at a time.