A record of separately solved problems
Open a current foundation-model announcement and “multimodal” reads as a single, continuous capability, as though a system that writes and a system that sees had always been one thing that simply grew larger. The documented record does not support that reading. For most of the last decade, a network that could describe a photograph and a network that could answer a question about one were separate research efforts, trained on separate objectives, evaluated on separate benchmarks, built years apart, and only later joined to each other and to large language models by several distinct engineering strategies — each one solving a specific limitation the previous strategy left standing.
This article follows that record rather than a folk timeline, through five documented turning points: image captioning and visual question answering built with convolutional and recurrent networks before the transformer existed; the 2021 shift from fixed label taxonomies to contrastive alignment against raw internet text at scale; two independent lineages of image generation — discrete-token autoregression and denoising diffusion — that started apart and were later combined; the bridging architectures that joined large pretrained language models to large pretrained vision encoders with a comparatively small trained connector rather than retraining either; and the recent move, described by the labs building them, toward models pretrained jointly across modalities from the beginning. Each turn is attributed to a specific paper or release that can be checked, not to a general impression of the field’s progress. Where a claim comes from a vendor’s own description rather than an independently verified fact, that distinction is kept explicit throughout: “trained jointly from the start” and “trained on one modality then extended” can look identical in a demonstration and mean very different things underneath.
Before the transformer: describing what a network could barely see
The earliest systems to put an image and a sentence in the same computation predate the transformer by two full years, and they inherited their basic shape from machine translation rather than from anything specific to vision. Vinyals, Toshev, Bengio and Erhan’s “Show and Tell” trained a convolutional network to produce a fixed-length image embedding and fed that embedding as the initial state of a recurrent network trained to generate a caption word by word, describing the whole system explicitly as an image encoder paired with a language decoder in direct analogy to the encoder-decoder architectures then used for translating between languages [1]. The image and language sides were not learned as one undifferentiated system: the convolutional encoder was pretrained separately for image classification and largely held fixed, and only the recurrent decoder and a projection into its input space were trained specifically for captioning.
Karpathy and Fei-Fei took the same encoder-decoder shape and pushed it toward finer structure. Their “Deep Visual-Semantic Alignments” system used a pretrained region-detection network to propose object regions within an image, embedded each region and each sentence fragment into a shared space with a ranking objective that pulled matching pairs together and pushed mismatched pairs apart, then trained a separate multimodal recurrent network on the resulting aligned pairs to generate novel descriptions of new images and image regions [2]. The pattern visible in both 2014 papers is one this article returns to repeatedly: a vision component pretrained on a vision-only objective, a language component trained on a language objective, and a comparatively narrow joining step — a projection, a shared embedding space, an initial hidden state — that does the work of bringing the two together. Neither paper trained pixels and words inside one undifferentiated network from a blank start; both assembled a system from two pieces that had already learned something on their own.
That assembly pattern was not a stopgap chosen for lack of compute. It reappears, essentially unchanged in spirit, in the bridging architectures built eight years later once the components being joined were a hundred times larger — worth holding onto, because the field’s own later announcements sometimes present each new joining strategy as a departure from everything before it.
A harder test: answering questions instead of describing scenes
Captioning rewards a system for producing a plausible sentence about an image; it does not require attending to any particular detail the way a specific question does. Malinowski and Fritz made this distinction explicit in 2014 with a system built around what they called a multi-world approach: rather than assuming a single correct segmentation of a scene, their model represented uncertainty over the scene’s semantic content in a Bayesian framework and combined that uncertain perception with a semantic parser over the question, so that an answer to how many chairs surround a table had to draw on an actual, if uncertain, accounting of chairs rather than a generically plausible response [3]. The paper framed this explicitly as a step toward a visual Turing test, evaluated on a small dataset of real indoor scenes with a correspondingly modest vocabulary of answers.
Antol and colleagues scaled the same task by an order of magnitude the following year with the VQA dataset: roughly a quarter of a million images paired with open-ended, freely typed questions and multiple human-written answers per question, designed, in the authors’ own framing, to require detailed understanding of the image and complex reasoning beyond what captioning demands [4]. The reported baselines were unglamorous by later standards — fixed convolutional image features combined with recurrent question features, feeding a classifier over a closed vocabulary of common answers — but the dataset proved durable, and it exposed a problem that recurs throughout the rest of this history: a benchmark built to require vision can still be substantially solvable from the question text alone, because textual regularities in how people phrase questions are themselves a strong signal about the likely answer. That failure mode was formalized more precisely in later work outside this article’s scope, but it was already latent in how respectably simple, largely language-driven baselines performed on the original VQA numbers.
Both lines of work through 2015 shared a structural ceiling. Every visual representation available to them came from a convolutional network trained for classification, holding a fixed, closed vocabulary of categories. Whatever a caption or an answer needed to say about an image, it could only say through features the classifier had already learned to notice. Escaping that ceiling — decoupling visual representation from any fixed label set — was the specific problem CLIP was built to solve, six years later.
What changed with contrastive alignment at scale
Radford and colleagues’ 2021 paper, “Learning Transferable Visual Models From Natural Language Supervision,” is usually remembered simply as CLIP, and the compression loses the specific thing that made it a turning point rather than an incremental improvement. The method itself is not exotic: encode an image, encode its paired caption, and train both encoders so that the true pairing scores higher than every mismatched pairing drawn from the same batch. For a batch of
an image-to-text and a text-to-image cross-entropy averaged together, each treating every other pairing in the batch as a negative. Nothing in that objective is architecturally new; matching objectives had been used for retrieval before CLIP. What changed was scale, and what the scale was spent on: 400 million image-text pairs collected from the public internet, with no hand-assigned category label anywhere in the pipeline [5].
The consequence the paper reports is the one that mattered downstream: a CLIP model applied with no task-specific training at all matched the accuracy of a supervised ResNet-50 on ImageNet, without using any of the 1.28 million labelled training images that network had needed [5]. That result directly removed the ceiling both captioning and VQA had been building against. A visual representation was no longer a fixed set of classifier categories a language head had to work around; it was a point in a space already organised by the free text people had actually written about images, addressable at inference time with an arbitrary sentence rather than a label chosen from a closed list.
CLIP’s influence over the rest of this history is direct: its image encoder became the default front end bridged into later vision-language models, and its joint embedding space became the conditioning signal one major text-to-image system was built directly on top of, discussed next. The objective’s own limits are worth stating plainly, because later sections depend on them: an in-batch ranking objective only has to learn whatever features are sufficient to win that particular ranking. It does not, by construction, guarantee that the resulting space correctly represents word order, spatial relations, or compositional structure — a limitation later, more targeted benchmarks went on to document directly, though tracing that critique belongs to a different article. What matters here is narrower: CLIP changed what a visual representation was addressable by, and that change is what the rest of this history builds on.
Two lineages of image generation, and their merger
Generating an image from a description turned out to have two largely independent origins, and the fact that they later merged into the same systems obscures how differently each one started.
The first lineage treated an image exactly the way a language model treats a sentence: as a sequence of discrete tokens to be predicted one at a time. Ramesh and colleagues’ “Zero-Shot Text-to-Image Generation” — the paper behind the system usually called DALL-E — first compressed each training image into a 32-by-32 grid of tokens drawn from a learned codebook of 8,192 possible entries, using a discrete variational autoencoder, then trained a single autoregressive transformer to model one combined stream of up to 256 text tokens followed by the corresponding 1,024 image tokens, exactly as a language model models a sentence [6]. The image was never modelled as pixels directly; it was modelled as a sequence over a fixed, learned vocabulary — the prior success of large-scale autoregressive language modelling, transplanted onto a differently tokenised sequence.
The second lineage did not start from language modelling at all. Ho, Jain and Abbeel’s “Denoising Diffusion Probabilistic Models” defined a fixed process that gradually corrupts an image with Gaussian noise over many steps, and trained a network to reverse it — to predict, at a randomly chosen step, the noise that had been added, so that running the network backward from pure noise reconstructs a plausible image [7]. The original results were unconditional, generated from noise with no text guiding the process, and the paper’s contribution was establishing that a simplified denoising objective could produce sample quality competitive with the adversarial training methods that had dominated image synthesis until then [7]. Diffusion, at its founding, was a solution to unconditional image synthesis, not to text-to-image generation.
The two lineages merged explicitly the following year. Ramesh, Dhariwal, Nichol, Chu and Chen’s “Hierarchical Text-Conditional Image Generation with CLIP Latents” — the paper behind DALL-E 2 — abandoned discrete-token autoregression for a two-stage design: a prior network maps a CLIP text embedding to a corresponding CLIP image embedding, and a diffusion decoder generates the final image conditioned on that CLIP image embedding [8]. Every component in that description belongs to work already covered above: CLIP supplies the aligned embedding space, diffusion supplies the generative mechanism, and DALL-E 2’s specific contribution was the architecture that let one condition the other.
Rombach and colleagues then made diffusion practical at deployment scale. “High-Resolution Image Synthesis with Latent Diffusion Models” runs the denoising process inside the compressed latent space of a separately trained autoencoder rather than on raw pixels, and adds cross-attention layers so the same architecture accepts flexible conditioning — text among other signals — at substantially lower training and sampling cost than pixel-space diffusion [9]. That architecture, released with open weights, is the one the public generally encountered as Stable Diffusion, and its efficiency argument is a distinct claim from either DALL-E 2’s use of CLIP or DDPM’s denoising objective, even though a casual retelling tends to compress diffusion into one undifferentiated idea. By 2022, image generation was not one method the field had converged on; it was a merger of three separately published results, each solving a different piece of the same problem.
Bridging pretrained giants instead of retraining them
Generation and captioning are, up to this point, two different problems solved by two different families of systems. The next turning point is about neither in isolation: it is about how the field connected large pretrained language models — expensive, general, already competent at reasoning and writing — to large pretrained vision encoders, without retraining either from scratch.
Alayrac and colleagues’ Flamingo, from DeepMind, made the case for this approach explicitly. The paper’s stated goal was to bridge powerful pretrained vision-only and pretrained language-only models, inserting new, trainable gated cross-attention layers between the frozen layers of an existing large language model so it could attend to visual features extracted by a frozen vision encoder, trained on large-scale web data of images and text interleaved in their natural reading order [10]. Both towers had already been trained, at considerable cost, on their own single-modality objectives before Flamingo touched them; what Flamingo trained from scratch was the comparatively small set of connecting layers between them. The reported payoff was genuine few-shot in-context learning on vision-language tasks — adapting to a new task from a handful of prompt examples, a behaviour previously associated with language models alone, now shown to transfer to tasks involving images once the bridge was in place.
Li and colleagues’ BLIP-2 pushed the same idea toward a smaller, cheaper connector: a lightweight Querying Transformer, a small set of learned query vectors, extracts a fixed number of visual features from a frozen image encoder and hands them to a frozen large language model, trained in two stages — first vision-language representation learning, then generative learning [11]. The authors report outperforming an 80-billion-parameter Flamingo configuration on zero-shot VQAv2 by 8.7 percentage points while training 54 times fewer parameters [11] — evidence that the connector, not the raw size of either frozen tower, was the binding constraint on earlier designs.
Liu and colleagues’ LLaVA simplified the connector again, down to a single trainable linear layer projecting visual features from a CLIP-derived encoder directly into an existing open language model’s input embedding space, trained on instruction-following examples generated by prompting GPT-4 with detailed image descriptions and asking it to produce plausible multimodal conversations about them [12]. LLaVA’s contribution was less about architecture — a linear projection is about as simple a bridge as exists — and more a demonstration that the training data teaching the bridge mattered as much as any structural choice.
Across all three designs, the constant is what does not change: the vision encoder and the language model are each pretrained once, separately, and stay largely or entirely frozen while a small connecting mechanism is trained to translate between them — the direct descendant of the 2014 encoder-decoder pattern, reappearing at roughly a hundred times the scale.
Systems built multimodal from the first training step
The bridging pattern has one structural property worth stating precisely, because it is exactly what the next generation of systems is described as departing from: whatever the language model learns about an image must pass through a connector of bounded capacity, trained after both towers already had fixed representations shaped entirely by their own single-modality pretraining. A frozen vision encoder trained only to align with captions cannot, after the fact, acquire the fine-grained detail a downstream task might need; the bridge can only pass along what the encoder had already extracted.
The systems that followed are described by their developers as removing that constraint at the source. Google’s Gemini paper presents the model family as natively multimodal, trained jointly across image, audio, video and text data from the beginning, rather than as a text-only model subsequently fitted with a vision or audio component [13]. That is a claim about the training process, not merely the interface: on its developers’ own account, the system learns its representations of images and text together from the start, rather than learning one and teaching a small bridge to translate toward it. The paper reports state-of-the-art results across a broad range of text, image, audio and video benchmarks for models trained this way [13], though the precise training procedure that makes joint pretraining different in practice from a very thorough bridge is not something an outside reader can independently verify from the paper alone; it is the paper’s own description of its method, and should be read as exactly that.
OpenAI’s public releases let the contrast between the two eras be dated precisely, since both the earlier and later design were released under documented names. GPT-4V, described in OpenAI’s September 2023 system card, extended GPT-4 to accept image inputs [14] — consistent, in spirit if not disclosed technical detail, with the bridging pattern above: an existing language model gaining a vision capability. GPT-4o, announced the following May, was described differently: OpenAI stated that a single new model was trained end-to-end across text, vision and audio, with all inputs and outputs processed by the same network, in explicit contrast to the voice interaction that had preceded it, described as a pipeline of three separate models — one transcribing audio to text, a language model processing that text, a third converting it back to audio — that could not perceive tone, background sound or multiple speakers, and could not itself produce laughter or song [15]. The measurable consequence reported alongside that description was a large drop in interaction latency: audio responses averaging 320 milliseconds, with a fastest reported case of 232 milliseconds, close to human conversational response time [15] — a figure at least consistent with removing a three-model relay, whatever the full underlying architecture turns out to be once more is disclosed.
What the sequence itself argues
Laid end to end, the five turns are not independent events; each is a response to a specific limitation the previous one left standing. Captioning and VQA systems built through 2015 were bounded by classifiers trained on fixed label sets, because that was the only visual representation available. CLIP removed that bound by training on raw text instead of labels, at a scale no earlier paper had attempted, but a contrastive ranking objective is not, by itself, a generative one — it can align an image and a sentence without producing either from the other. Image generation solved a problem CLIP’s objective does not touch, through two lineages that started independently and were combined only once both had matured on their own. None of the first three turns addressed how to give such a system access to the reasoning and world knowledge inside a large pretrained language model; the bridging architectures solved specifically that, at the cost of routing all visual information through a connector of fixed, finite capacity. The native multimodal systems are presented by the labs building them as a response to that specific cost — training jointly so nothing is routed through a bottleneck fitted after the fact, at the cost, for an outside reader, of a training process that is largely undisclosed and cannot be independently checked against the bridging alternative it claims to improve on.
That last gap is worth naming rather than smoothing over, because it is where this history is currently unfinished. Whether joint training from the start actually produces qualitatively different cross-modal reasoning than an extremely well-bridged system, or whether the practical difference is smaller than the marketing distinction suggests, is not something the publicly available papers resolve, because the training procedures on both sides of that comparison are not published in enough detail to compare directly. Readers should treat “natively multimodal” the way this article has treated every other vendor description: as a claim about method, worth taking seriously, and not yet independently verified against a disclosed alternative.
Predictions, with the observations that would falsify them
These are forecasts, kept deliberately separate from the sourced history above. Horizon: 12 August 2028. Assumptions: no architectural discontinuity comparable in scale to the arrival of the transformer itself, continued competitive pressure to disclose enough about training to support marketing claims without disclosing enough to be independently checked, and continued publication of comparative benchmarks by parties outside the labs building frontier systems.
One. Bridging architectures will not disappear even as native multimodal pretraining becomes more common at the frontier, because a frozen-encoder-plus-connector design remains far cheaper to build and update than joint pretraining, independent of which approach reasons better. Indicator: continued publication of new bridging designs, particularly from teams without frontier-scale pretraining budgets. Disconfirmed if the bridging pattern effectively disappears from newly published vision-language systems by the horizon date, with joint pretraining becoming the only design in active use.
Two. Independent evaluation will increasingly separate “trained jointly from the start” from “extended with a very well-trained bridge” by testing for the specific failure this article traces to bridging — information loss at a fixed-capacity connector — rather than relying on either label. Indicator: benchmarks specifically designed to probe fine-grained cross-modal binding in a way a bounded connector would be expected to fail even when overall accuracy looks strong. Disconfirmed if by 2028 no such targeted evaluation exists and the distinction remains something only vendor documentation asserts.
Three. Diffusion and autoregressive-token generation will not fully converge into one dominant method; both will continue to appear in new systems, chosen for different reasons — tokens for unifying generation with the sequence-modelling machinery used for text, diffusion where sample quality and controllability matter more than architectural unification. Disconfirmed if one lineage is essentially abandoned in new frontier releases by the horizon date.
Four. The failure mode identified in early VQA systems — a benchmark solvable, in part, from the prompt text without the accompanying image or audio actually being needed — will keep recurring in new multimodal benchmarks as modalities are added, because it is a property of how benchmarks get built under time pressure, not something specific to 2015-era vision. Disconfirmed if newly introduced multimodal benchmarks are consistently released already accompanied by a modality-ablation audit that few or no models pass without the full input.
What to take away
The record does not describe one capability that grew. It describes five separately solved problems, each attributed to a specific, checkable paper or release, joined to the ones before it because the join solved a limitation the previous approach left standing: fixed label sets, given up for contrastive alignment against raw text; alignment without generation, filled in by two independent generative lineages that later merged; generation and understanding without access to a large language model’s reasoning, filled in by connectors trained to bridge frozen pretrained towers; and, most recently, the bounded capacity of that connector, addressed — on the builders’ own account, not yet independently verified — by training jointly across modalities from the first step.
Reading a current release against this record is mostly a matter of asking which of these five turns it actually represents, and on whose authority. A system described as multimodal might mean any of them. The specific claim being made — aligned but not generative, generative but not reasoning-linked, bridged, or genuinely joint from the start — determines what the system can actually be expected to do, and only some of those claims can currently be checked against a published paper rather than a product announcement.
