Adapters, Native Pretraining, and Unified Tokens: The Main Multimodal Architectures, Compared

Four recipes, one word doing too much work

“Multimodal foundation model” names a destination, not a road. It says that a system accepts more than one kind of input, or produces more than one kind of output, and it says nothing at all about how that was built. In the published literature there are at least four structurally different answers, each with its own training objective, its own cost profile, and its own documented failure mode, and none of them is a minor variation on another.

The first answer keeps two expensive, separately trained networks intact and connects them with a small trained bridge — the recipe behind LLaVA, BLIP-2 and Flamingo. The second trains one network on several modalities together from the first gradient step, discarding the idea of a pretrained unimodal backbone at all — the recipe Google describes for Gemini and Microsoft for Kosmos-1. The third refuses even the idea of modality-specific components at the input level, converting everything — words, image patches, robot joint angles — into entries in one shared vocabulary that a single autoregressive model predicts token by token, as Gato and Chameleon do. The fourth abandons next-token prediction entirely for generation and instead learns to reverse a noising process, the family that produced Stable Diffusion, DALL-E 2, Imagen and CoDi.

This article takes the four in turn, states what each one’s own authors say the design bought them, what it cost, and what it failed at, then closes on why stacking these four onto one ranked list would misrepresent all four papers. They optimise different objectives, under different constraints, reported by different measures. The right question is never “which is best” — it is “which cost was this team willing to accept, and which did they refuse to pay.”

The modular adapter: reuse two frozen networks and train the seam

The oldest and cheapest recipe treats a vision encoder and a language model as finished, expensive artefacts that should not be touched. Flamingo’s own framing is explicit about why: the goal is architecture that can “bridge powerful pretrained vision-only and language-only models” and handle arbitrarily interleaved sequences of images and text without retraining either one ^[3]. The vision and language backbones are frozen; new gated cross-attention layers are inserted between the language model’s existing blocks and trained from scratch to let the frozen language model attend to visual features. Flamingo’s own ablations quantify why the freeze matters rather than merely asserting it: training the whole system from scratch cost 12.9 percentage points of performance, and even fine-tuning the pretrained language model — not freezing it, just adjusting it — cost 8.0 points, which the authors attribute to catastrophic forgetting of what the language model already knew ^[3]. Freezing is not a shortcut taken for convenience. It is reported as the difference between keeping a capability and losing it.

BLIP-2 pushes the same logic further and makes the resulting efficiency the headline result rather than a side finding. Its bridge, a lightweight Querying Transformer, is trained in two stages — first to learn vision-language representations from a frozen image encoder, then to learn vision-to-language generation from a frozen large language model — and the paper reports outperforming Flamingo80B by 8.7 percentage points on zero-shot VQAv2 while training 54 times fewer parameters ^[2]. That figure is the clearest documented statement in this literature of what the modular approach is actually purchasing: not a better model in some abstract sense, but dramatically cheaper training for comparable or better task performance, because almost the entire parameter count of both towers is inherited rather than learned.

LLaVA simplifies the bridge itself down to a single linear projection matrix mapping CLIP visual features into the language embedding space, and makes the case that most of the necessary work can be done with data rather than architecture: the projection is trained on machine-generated visual instruction data produced by prompting GPT-4 with image captions and bounding boxes ^[1]. LLaVA’s own two-stage recipe is worth stating precisely because it shows the freeze is not absolute even within one paper. In the first, feature-alignment stage, both the vision encoder and the language model stay frozen and only the projection matrix trains. In the second stage the vision encoder stays frozen but the language model itself is unfrozen and updated end to end ^[1]. The “modular” label describes stage one; stage two already spends real compute retraining the language side, and the paper reports this trade explicitly rather than hiding it.

None of these three papers claims the resulting bridge is unlimited in what it can carry. LLaVA’s own error analysis documents a case where the model answers confidently that strawberry-flavoured yoghurt is present in a fridge that in fact contains only yoghurt and strawberries, which the authors read as the model treating the image “as a bag of patches, failing to grasp the complex semantics within the image” ^[1], and they separately note that recognising a specific product brand would require higher input resolution than the system uses. The bridge has a fixed information-carrying capacity, whether it is BLIP-2’s small set of learned query vectors or LLaVA’s single projection matrix, and everything the language side ever learns about the image has to fit through it.

This is the formal shape of “freeze the backbones”: the gradient update is masked to zero everywhere except the bridge parameters $\theta_{\text{bridge}}$, so $|\theta_{\text{bridge}}|$ — a few hundred million parameters in BLIP-2’s case, dramatically fewer than either tower — is the entire quantity being optimised, and $\theta_{\text{vision}}$ and $\theta_{\text{LM}}$ contribute a forward pass but never a gradient.

Native pretraining from scratch: train every modality into the same weights

A second lineage rejects the premise that a unimodal backbone should be inherited at all. Google’s technical report describes Gemini models as “natively multimodal,” trained jointly across image, audio, video and text data from the outset rather than assembled by joining single-modality components, and states that Gemini models can accept text interleaved with a wide range of audio and visual inputs and can natively output images using discrete image tokens without relying on an intermediate natural-language description ^[4]. The rationale, as the paper frames it, is architectural: joint training lets cross-modal correlations be learned at every layer of the network rather than only at a late bridge, and it removes the ceiling a frozen encoder’s fixed representation would otherwise impose.

That design choice is expensive in a way the modular approach specifically avoids. Gemini’s report documents training the largest model, Gemini Ultra, on a large fleet of TPUv4 accelerators deployed in “SuperPods” of 4,096 chips across multiple Google datacenters, and separately discusses infrastructure work — such as redundant in-memory model copies for fast recovery — needed to keep goodput on the largest training run above roughly 97 percent ^[4]. None of that infrastructure is optional overhead layered on top of a cheap idea; it is the direct cost of training one very large network jointly from initialisation on several modalities simultaneously, rather than reusing two networks someone else already paid to pretrain separately.

Microsoft’s Kosmos-1 is a genuinely instructive intermediate case, and it is worth being precise about where it sits rather than filing it neatly into one bucket. The paper states plainly that the model is trained “from scratch on web-scale multimodal corpora, including arbitrarily interleaved text and images, image-caption pairs, and text data,” and its roughly 1.3-billion-parameter multimodal language model backbone is randomly initialised, not fine-tuned from an existing pretrained text-only checkpoint ^[5]. But the paper also documents that Kosmos-1 obtains its visual input features from a frozen pretrained CLIP ViT-L/14 vision encoder ^[5] — meaning the language-modelling core is native, jointly trained, while the visual front end is inherited exactly as in the modular recipe. Kosmos-1 sits on the boundary between the two families this article treats as distinct, and that boundary is real: “native pretraining” and “frozen adapter” describe a spectrum of how much of the system was jointly optimised from initialisation, not two hermetically sealed categories.

The documented limitation of native pretraining in this literature is not architectural instability — that problem belongs to the next section — but capability gaps that persist despite the joint training. Kosmos-1’s authors introduce a nonverbal Raven-style IQ test specifically to probe reasoning that is not mediated by language, and report that the model reaches only 26 percent accuracy against a random baseline of 17 percent, describing “a large performance gap between the current model and the average level of adults” even as they credit the model with demonstrating the basic capability ^[5]. Joint training from scratch buys deeper cross-modal integration; it does not, on this evidence, buy strong abstract reasoning over visual patterns for free.

All of $\theta$ here is shared and randomly initialised at $t=0$, and no term is held fixed while another is optimised. That is the formal content of “native”: contrast it with the modular recipe, where the vision and language terms are each minimised separately, over separate data, to convergence, long before the two are ever combined.

Unified tokenization: one vocabulary, one loss, every modality

A third lineage keeps a single autoregressive next-token objective — the workhorse of language modelling — and instead does all of the multimodal work at the tokeniser, converting every input into entries in one shared discrete vocabulary before the model ever sees it. DeepMind’s Gato is the most literal statement of this idea, because it is not only multimodal but multi-task and multi-embodiment: the same 1.2-billion-parameter decoder-only transformer plays Atari, captions images, chats and operates a real robot arm, deciding at each step “whether to output text, joint torques, button presses, or other tokens” depending on context ^[6]. Its tokenisation scheme is stated precisely: text is encoded with SentencePiece into the integer range zero to 32,000; images are cut into non-overlapping 16-by-16 patches in raster order in the manner of a Vision Transformer; and discrete or continuous control values, from Atari button presses to joint torques, are separately discretised — continuous values mu-law encoded and then binned — into a further range of 1,024 tokens ^[6]. Whatever arrives, it becomes an integer, and the model never has to know which kind of integer it is.

Gato’s own reported limitation is not a training-stability failure but a deliberate, disclosed capability trade. On Atari, a specialist agent trained on one game outperforms the generalist Gato, and the authors write that this “suggests that scaling Gato may result in even better performance,” immediately followed by the statement that they “purposely restricted Gato’s size such that it can be run in real-time on a real robot” ^[6]. The single shared vocabulary and single shared network are not reported as free; the authors identify capacity itself as the binding constraint, and record having chosen deployability over the scaling headroom they believe was available.

Meta’s Chameleon commits to the same principle at a much larger scale and is candid that stability, not capability, was the hard engineering problem. It describes itself as “a family of early-fusion token-based mixed-modal models” that interleaves image and text tokens in one sequence, generating either kind at any position, and its authors state directly that this “requires a stable training approach from inception, an alignment recipe, and an architectural parameterization tailored for the early-fusion, token-based, mixed-modal setting” ^[7]. The paper documents the failure mode this addresses concretely: without a query-key normalisation step controlling the growth of attention logits, Chameleon-7B diverges after roughly 20 percent of a single training epoch, a failure the authors attribute to the softmax operation behaving badly across tokens of very different entropy once image and text tokens share one sequence ^[7]. The 34-billion-parameter variant needed a second, distinct intervention — reordering where layer normalisation sits within the transformer block — and both model sizes were trained with an added z-loss regularisation term penalising drift in the softmax normaliser ^[7]. The reported scale is substantial on its own terms: 4.4 trillion tokens, with the 7-billion-parameter run consuming roughly 856,000 GPU-hours and the 34-billion-parameter run roughly 4.28 million GPU-hours ^[7]. Meta’s earlier CM3 model is the more modest ancestor of the same idea, treating structured web documents — text, hyperlinks and VQVAE-encoded image tokens — as one causally-masked token stream so that a single model can perform captioning, zero-shot text-to-image generation and entity-linking without task-specific heads ^[8].

The same cross-entropy objective a text-only language model uses is the entire training objective here — the only thing that changed is what $z_t$ is allowed to denote. Whether index $z_t$ names a subword, a 16-by-16 image patch code, or a discretised robot joint angle is invisible to the loss function; the difficulty this section documents is not in the objective but in keeping training numerically stable once the vocabulary spans quantities of such different statistical character.

Diffusion: learn to reverse noise, and let a frozen text model steer it

The fourth lineage answers a different question than the first three. Adapters, native pretraining and unified tokenization are all, in the end, recipes for understanding or jointly representing multiple modalities; diffusion is a recipe for generating one, typically conditioned on another, and it does not use next-token prediction at all. A diffusion model learns to reverse a fixed process that gradually adds Gaussian noise to data, training a network to predict the noise component at each step:

where $c$ is a conditioning signal — most often a text embedding — and generation runs the process in reverse, starting from pure noise and repeatedly subtracting a predicted noise estimate.

Rombach and colleagues’ latent diffusion work, the architecture behind Stable Diffusion, addresses the single largest documented cost of that recipe directly: the paper states that training the most powerful diffusion models “often takes hundreds of GPU days,” citing 150 to 1,000 V100-days as typical, and its stated fix is to run the denoising process not over raw pixels but inside the compressed latent space of a separately pretrained autoencoder, reporting roughly four times fewer training resources than comparable pixel-space diffusion models while adding cross-attention layers so the same architecture can condition on text or other inputs ^[9]. The authors are explicit that this is a real trade, not a free win: because generation happens in a compressed latent space, “the use of LDMs can be questionable when high precision is required,” since the autoencoder’s own reconstruction fidelity becomes a bottleneck for tasks needing fine-grained pixel accuracy ^[9].

OpenAI’s DALL-E 2 combines this generative machinery with an existing contrastively trained embedding space rather than training a text encoder from scratch: a prior model maps a text caption to a CLIP image embedding, and a diffusion decoder then generates an image conditioned on that embedding ^[10]. The design rationale is stated as inheriting CLIP’s joint embedding space specifically because it supports “language-guided image manipulations in a zero-shot fashion” — edits and style-preserving variations follow from moving through an embedding space that was never trained for image generation at all ^[10].

Imagen makes a related but distinct architectural bet and reports an unusually clean empirical result for it: rather than training any dedicated text encoder, it feeds text through a frozen, off-the-shelf large language model, T5-XXL, then generates a low-resolution image with one diffusion model and upsamples it through a cascade of two further diffusion super-resolution models ^[11]. The paper’s central finding is that scaling the size of that frozen text encoder improves both sample fidelity and text-image alignment substantially more than scaling the image diffusion model does ^[11] — an echo, inside a completely different architecture family, of the modular lesson that a large pretrained unimodal model can be reused rather than re-learned. Imagen’s authors document a specific, named limitation rather than a generic disclaimer: “Imagen’s limited ability to generate photorealistic people,” which they trace in part to limitations in the training data used ^[11].

CoDi generalises the diffusion recipe furthest, aiming at any combination of input and output modalities — text, image, video and audio — rather than one fixed direction such as text-to-image. Its method aligns separately trained, modality-specific diffusion models in a shared conditioning space so that modalities can be generated in parallel and, when interrelated, synchronised, such as producing a video and its matching soundtrack together ^[12]. Where Chameleon and Gato achieve generality by forcing every modality through one shared discrete vocabulary, CoDi achieves a comparable generality by aligning several continuous diffusion processes with each other — a structurally different route to the same word, “any-to-any,” and worth naming as different rather than treating as interchangeable.

Why these four do not resolve into a ranking

Lay the four recipes side by side and a pattern emerges that argues directly against ranking them. Each one fixes a specific cost as non-negotiable and lets a different one float.

The modular adapter fixes pretraining cost near zero, by construction: BLIP-2’s 54-times reduction in trainable parameters relative to Flamingo80B is only possible because both towers already existed ^[2]. What floats is representational bandwidth — a bridge of fixed size is a hard ceiling on how much visual detail can ever reach the language side, documented directly in LLaVA’s own “bag of patches” failure case ^[1].

Native pretraining fixes representational bandwidth instead — nothing is squeezed through a late bridge, because the whole network was jointly trained — and lets infrastructure cost float upward, into TPU SuperPods and multi-datacenter training runs ^[4]. Kosmos-1’s own design shows this is a matter of degree: the paper freezes exactly the piece, the vision encoder, that the fully native approach would also train, and pays a correspondingly smaller infrastructure bill for it ^[5].

Unified tokenization fixes generality — one architecture, one loss, arbitrarily many modalities and even non-linguistic control signals, as in Gato’s shared control-and-perception vocabulary ^[6] — and lets training stability float, sometimes badly: Chameleon’s authors needed three separate interventions to keep training from diverging at all ^[7]. Diffusion generation fixes sample fidelity for continuous, high-dimensional outputs — image and audio quality that autoregressive token prediction has not matched on the same tasks — and lets inference cost float, since a usable sample requires many sequential denoising passes rather than one forward pass, a cost latent diffusion reduces without eliminating ^[9].

None of these trades is dominated by another. A team with two strong pretrained unimodal models and a small compute budget has a documented reason to choose the adapter route; a team that needs the model itself to reason jointly over interleaved modalities, and can fund TPU SuperPods, has a documented reason to choose native pretraining; a team building one generalist controller across dissimilar task types has a documented reason to choose unified tokenization even knowing it will fight instability; and a team whose deliverable is a generated image, not a text answer about one, has a documented reason to choose diffusion over all three. Building a single cross-architecture leaderboard from these four papers would compare systems evaluated on different benchmarks, at different scales, against different costs, for different purposes — exactly the kind of incommensurable comparison this publication’s editorial standards rule out.

What to take away

Four papers, four training objectives, four different things held fixed while something else was allowed to grow expensive. The adapter recipe pays almost nothing to pretrain and pays in bridge capacity. Native pretraining pays in infrastructure and buys depth of integration. Unified tokenization pays in training stability and buys a single architecture that spans modalities and task types most systems treat as separate problems. Diffusion pays in iterative inference cost and buys sample fidelity nothing else here matches on the same generative tasks.

None of the original papers claims to have solved multimodality in general; each claims to have solved it under a specific, stated constraint, and reports exactly where that constraint bites. Reading a system’s marketing page for the word “multimodal” answers almost nothing. Reading which of these four training recipes it used, and what that recipe’s own authors said it cost and where it broke, answers a great deal more.