How Multimodal Models Actually Handle Video, Audio, and Space

Two representations, and a third kind of problem

A companion article in this series worked through what image-text multimodal systems share: a patch tokenizer that cuts a picture into a grid, a contrastive objective that puts two encoders into loose correspondence, and a fusion point where a bridge lets a language model read visual features it was never trained to produce. That account stops at the edge of the picture. It says nothing about a clip that moves, a waveform that has no spatial grid to cut, or a room a robot has to act inside rather than merely describe.

Those three are not the same problem wearing different clothes, and treating them that way is where a lot of “multimodal” marketing goes wrong. An image is a fixed spatial array; cutting it into patches is a solved, if lossy, engineering problem. Video adds a dimension that does not behave like the other two — time has a direction, a duration that can run to hours, and events shorter than any fixed sampling interval simply vanish. Audio has no spatial grid at all; it is a one-dimensional signal sampled tens of thousands of times a second, and it must be forced into a discrete vocabulary before a transformer can treat it the way it treats a word. And a 3D scene or an embodied environment is not a sequence in any natural sense to begin with — it is a continuous field of geometry and appearance that has to be sampled, or a stream of proprioceptive and control signals that has to be tokenized on both the input and the output side.

This article works through each in turn: the temporal-tokenization problem that makes video a spacetime volume rather than a longer picture, the discretization schemes that turn a waveform into codes, what “spatial representation” means as a technical claim rather than a slide-deck phrase, the specific machinery embodied interfaces add on top of everything already built for image and text, and where current evaluation of all three currently overstates what has been measured.

The temporal-tokenization problem: why video is not a bigger image

The image-text tokenization scheme has an exact, well-understood cost. An image of height $H$ and width $W$, cut into square patches of size $P$, produces

tokens. Video adds a third factor. If a model attends over $T$ sampled frames, each cut into the same spatial grid, and groups $P_t$ consecutive frames per temporal patch, the token count becomes

Every added frame multiplies the spatial cost rather than adding to it. A minute of video at even a modest frame rate is not “a bigger picture” in the way a higher-resolution photograph is; it is a volume, and attention cost over a transformer sequence grows faster than the sequence itself. No system deployed today attends densely over every raw frame of a long clip, because nothing could afford to.

The two working answers both throw detail away, deliberately, at different points in the pipeline. One is to compress before tokenizing at all. Sora’s technical report describes turning video into a unified representation by first compressing videos into a lower-dimensional latent space, then decomposing that latent representation into spacetime patches that a diffusion transformer treats as its token sequence ^[1] — the compression and the patchification are two separate lossy steps, not one. The other is to build a tokenizer whose vocabulary is designed for video specifically. MAGVIT-v2 is described by its authors as a video tokenizer built to generate concise and expressive tokens for both videos and images from a common vocabulary, using a lookup-free quantizer rather than the codebook-lookup scheme earlier video and image tokenizers relied on ^[3]. VideoPoet then treats video generation itself as a language-modeling task over exactly this kind of token stream, using a single decoder-only transformer across the video, image, audio and text tokens it is given ^[2]. The common thread across all three: nothing in this pipeline lets the model see a raw frame. Everything it reasons over has already survived two rounds of compression before a single attention weight is computed.

What frame sampling decides before the model ever runs

The consequence that matters practically is that a system’s temporal-tokenization budget is set once, in advance, and it silently caps what kinds of questions the system can ever answer well — a fast, brief event either survives the sampling interval or it does not, and no amount of downstream reasoning recovers it if it does not.

Two results make this concrete rather than hypothetical. Gemini 1.5’s technical report frames its central claim as a context-length one: the model family can recall and reason over fine-grained information across millions of tokens of context, including hours of video and audio, with near-perfect retrieval reported above 99% up to at least ten million tokens — a figure the authors compare directly against the 200,000-token context of Claude 3.0 and the 128,000-token context of GPT-4 Turbo at the time of writing ^[4]. That is a vendor’s own reported comparison, made in the vendor’s own technical report, and it should be read as exactly that: a claim about one system’s stated context capacity, not an independent measurement, and not evidence about accuracy at that length rather than mere token admission.

Video-MME supplies the independent check on what a long temporal window actually buys. The benchmark assembles 900 videos totalling 254 hours, spanning clips from 11 seconds to a full hour across six visual domains and thirty subfields, with 2,700 manually annotated question-answer pairs ^[5]. Its authors report Gemini 1.5 Pro as the strongest commercial model in their evaluation — a finding specific to this one benchmark’s construction, not a general ranking across incomparable suites — while also concluding that current systems need further improvement in handling longer sequences and multi-modal data jointly, a polite way of saying that performance degrades as duration grows even for the strongest entrant ^[5]. Read together, the two results are not in tension: a large context window is a necessary condition for reasoning over an hour of video, and Video-MME’s own data shows it is not a sufficient one.

Discretizing sound: codecs, quantizers, and two kinds of audio token

Audio has no spatial grid, so the patchification story does not transfer at all; the first job is simply to turn a continuous waveform into a finite alphabet a transformer can share with text.

The mechanism that does this at scale is a learned neural codec built around residual vector quantization: an encoder compresses the waveform into a sequence of continuous frames, and each frame is then quantized in successive stages, each stage encoding what the previous stage’s codebook missed. SoundStream is the reference architecture, described by its authors as an end-to-end neural audio codec built on a convolutional encoder-decoder pair with a residual vector quantizer, trained jointly and shown to outperform prior codecs at comparable bitrates ^[7]. The mechanism gives a concrete, computable token rate. For a signal sampled at $f_s$ hertz, encoded with hop length $h$ samples per frame, and quantized with $n_q$ residual stages of codebook size $K$ each, the bitrate is

Every added quantizer stage buys reconstruction fidelity at a fixed, computable token-rate cost — the audio-codec analogue of the video tokenizer’s frame-versus-resolution trade, and just as unavoidable.

A codec alone only reconstructs audio faithfully; it does not, by itself, capture the kind of long-range structure that makes a spoken sentence cohere or a musical phrase resolve. AudioLM’s authors address this with a hybrid, two-tier scheme: coarse tokens drawn from the discretized internal activations of a masked language model pretrained on raw audio, which the authors show captures long-term structure, paired with fine acoustic tokens from a neural audio codec, which supplies the reconstruction quality the coarse tier lacks on its own ^[6]. The authors report that, trained on speech with no transcript or textual annotation at all, the resulting system generates continuations that are syntactically and semantically plausible while preserving a speaker’s identity and prosody, and that the same approach extends to generating coherent piano-music continuations despite no symbolic representation of music ever entering training ^[6]. The structural point survives even outside generation: joint text-audio processing requires a token vocabulary manufactured by at least one dedicated model trained specifically to discretize sound, layered under whatever text tokenizer the language model already uses — audio tokens are not an incidental byproduct of the same pipeline that tokenizes images.

What “spatial representation” means technically

“Spatial representation” is used loosely enough in multimodal marketing to mean almost anything — a photo of a room, a depth map, a point cloud, a robot’s joint state. Technically, it names a specific and much narrower design choice: how continuous 3D geometry gets converted into something a model can be trained on, and the field has settled on two structurally different answers rather than one.

The first treats a scene as a continuous field rather than a discrete grid at all. Neural radiance fields represent a scene as a fully connected network mapping a continuous 5D coordinate — a 3D position plus a 2D viewing direction — to a volume density and a view-dependent emitted colour, then use classical volume rendering to synthesize the colour a camera ray would see by integrating along it ^[8]. The rendering equation itself is the cleanest statement of what “continuous” buys and costs:

where $\sigma$ is volume density, $\mathbf{c}$ is emitted colour, and $T(t)$ is accumulated transmittance along the ray up to $t$. There is no patch, no voxel grid, no fixed token count anywhere in this formulation — the scene is a function evaluated at query points, and any tokenization of it for a downstream language model has to be imposed afterward, by sampling the field at a chosen resolution, which reintroduces exactly the resolution-versus-cost trade video and audio already face.

The second, more common answer in current multimodal systems discretizes spatial sensing directly, alongside other non-visual sensor channels, into a single joint embedding. ImageBind’s authors report learning one joint embedding space across six modalities — images, text, audio, depth, thermal, and inertial-measurement-unit motion data — and show that image-paired training data alone is sufficient to bind all six together, without needing every pairwise combination of modalities to co-occur in training ^[9]. Depth and IMU data are exactly the sensor channels an embodied system needs to represent “where things are” and “how the body is moving,” which makes this the clearest available evidence that spatial representation in practice usually means treating geometry and motion sensing as additional discretizable input channels bound into the same embedding space as image and text, rather than as a NeRF-style continuous field a language model queries directly.

Embodied interfaces: continuous sensors in, discrete actions out

An embodied system adds a requirement neither video nor audio nor static 3D reconstruction has to meet on its own: it must produce continuous, low-level control outputs, not just consume rich input. That closes a loop the rest of this article has left open, and it forces a second, symmetric tokenization problem on the output side.

Two published systems solve it in structurally different ways, and the difference is the clearest illustration in this article of what “architectural addition” actually means in practice. PaLM-E keeps the state continuous on the way in: its authors describe embodied language models that directly incorporate real-world continuous sensor modalities into a language model, training multi-modal sentences that interleave visual, continuous state-estimation, and textual encodings end-to-end together with a pretrained large language model. Their largest model, PaLM-E-562B at 562 billion parameters, is reported to exhibit positive transfer — benefiting from joint training across internet-scale language, vision, and visual-language data even while also handling robotic manipulation planning — and to reach state-of-the-art performance on the OK-VQA visual question-answering benchmark while retaining general language capability as scale increases ^[10]. Nothing about a robot’s joint angles is discretized here; a continuous state embedding sits in the same sentence as a word embedding.

RT-2 takes the opposite route on the output side: rather than adding a continuous channel, its authors express robot actions as text tokens and incorporate them directly into the model’s existing training data in the same way as natural-language tokens, so a single vision-language-action model can be co-fine-tuned on robot trajectories and ordinary web-scale vision-language data at once ^[11]. Across an evaluation the authors describe running to six thousand trials, they report this recipe yielding emergent capabilities not present in models trained on robot data alone — better generalization to novel objects, the ability to follow commands not present in the robot’s own training data, such as placing an object on a particular printed number, and multi-stage reasoning, such as identifying that a rock is the best available substitute for a hammer or that an energy drink suits someone who says they are tired ^[11]. Gato generalizes the same discretize-everything approach furthest: a single sequence model that, at every step, decides based on context whether to output text, joint torques, button presses, or some other token type at all, trained across hundreds of distinct tasks and embodiments with one shared architecture and vocabulary ^[12]. PaLM-E’s continuous-embedding route and RT-2’s discretize-to-text route are not competing claims about which is correct; they are two different, currently coexisting engineering answers to the same closed-loop requirement, and a specification sheet that says “robotics” without saying which one it uses is not describing a mechanism at all.

The architecture has to grow a slot, not just a bigger encoder

Image-text fusion, as the companion article describes it, comes down to a bridge: a frozen or lightly adapted vision tower, a frozen or lightly adapted language model, and some fixed-capacity connector between them. Extending that pattern to video, audio, and spatial input at once is not a matter of making the same bridge wider. Each modality above needed a dedicated tokenizer built for its own structure — a spacetime-patch scheme for video, a residual-quantized codec for audio, a continuous field or a bound sensor embedding for space — and a fusion architecture has to be able to accept a new one without redesigning the whole stack.

Perceiver IO is the clearest published example of an architecture built for exactly that requirement rather than adapted to it after the fact. Its authors describe a general architecture for structured inputs and outputs that maps arbitrarily sized and structured input and output arrays through a fixed-size latent bottleneck using cross-attention, decoupling the depth and cost of the network’s core processing from the size and modality of what it is fed. The same architecture is reported to handle tasks spanning language, visual understanding, multi-task and multi-modal reasoning, and even structured game-state input from StarCraft II, without modality-specific architectural changes to the core network ^[13]. What this buys, concretely, is a fusion point whose cost does not grow with the number of modalities plugged into it — every new sensor stream adds a modality-specific encoder feeding into the same fixed-size latent array, rather than a new pairwise bridge to every other modality already present. That is the direct architectural answer to the video-plus-audio-plus-space case: not one larger image-text bridge stretched to cover three more inputs, but a bus with a slot cut for each kind of input, so that adding a modality is a local change rather than a full re-fusion.

Evaluating a model that has to attend to time and space

Everything above about the modality-specific tokenization and fusion cost predicts a specific failure mode in evaluation: since each modality’s front end throws away detail at a different point and by a different mechanism, a single aggregate score across video, audio, and spatial tasks measures the union of several different bottlenecks, not one capability.

Video-MME’s own construction is instructive here in a second way, beyond the long-video degradation already discussed. By deliberately spanning clip lengths from 11 seconds to a full hour and by scoring performance across that range rather than at one fixed duration, the benchmark’s authors force a system’s temporal-sampling choices into the open — a model that performs well at short duration and poorly at long duration is not failing at “video understanding” in general; it is failing specifically at whatever compression or sampling scheme it applies once a clip exceeds its effective attention window ^[5]. That is exactly the diagnosis this article’s temporal-tokenization section predicts, and it means a single leaderboard number for “video ability” hides the duration at which the score was earned unless the benchmark reports it separately, which Video-MME does and many narrower video benchmarks do not.

The honest position, consistent with how the companion article treats image-text benchmarks, is to refuse a single cross-vendor ranking built from these numbers. A reported score on Video-MME, an audio-generation quality metric drawn from a codec’s own reconstruction benchmark, and a robot-manipulation success rate from a fixed set of evaluation trials are not commensurable, and no weighted combination of them produces a meaningful “multimodal capability” figure. What each of these benchmarks can support is a narrower, better-grounded claim: this system handles clips up to this duration at this measured accuracy, this codec reconstructs speech at this bitrate with this measured fidelity, this policy succeeds at this rate across this number of physical trials. Anything broader than that is marketing language borrowing the credibility of a specific measurement it did not actually report.

Predictions, with the observations that would falsify them

These are forecasts, separated deliberately from the sourced analysis above. Horizon: August 2028. Assumptions: no architectural discontinuity comparable to the arrival of the transformer, continued commercial pressure toward video and embodied capability specifically, and continued publication of benchmark construction details at roughly the current level of disclosure.

One. Reported video-understanding context length will keep growing while duration-stratified accuracy — performance broken out by clip length, the way Video-MME already reports it — will remain the more informative and more rarely cited number. Indicator: system cards continuing to lead with a token-count or hours-of-video figure. Disconfirmed if duration-stratified scores become the standard headline figure in major system cards by the horizon date.

Two. The discretize-actions-as-text approach exemplified by RT-2 and Gato, rather than PaLM-E’s continuous-embedding approach, will dominate newly published general-purpose embodied systems, because it lets one existing language-model vocabulary and training pipeline absorb a new output modality without a parallel continuous-decoding path. Disconfirmed if a majority of newly published general embodied systems by 2028 use continuous action outputs rather than discretized action tokens.

What to take away

Video, audio, and spatial or embodied input each break the image-text playbook in a different, specific place. Video turns a fixed patch grid into a spacetime volume whose token cost multiplies with duration, and every deployed system answers that with a lossy compression step before the model ever sees a frame. Audio has no spatial grid to cut at all and needs a purpose-built codec, usually layered with a second, coarser tier of tokens to capture structure the codec’s reconstruction objective does not reward. Spatial representation names a real technical fork — a continuous field queried at chosen points, or a discretized sensor channel bound into a shared embedding — and most deployed embodied systems have picked the second. And the fusion architecture that has to hold all of this together cannot simply be a bigger image-text bridge; it needs a fusion point whose capacity grows by adding a slot per modality rather than by widening one bridge to cover everything at once.

None of this is a single unsolved problem waiting on a bigger model. It is four separate, actively developed engineering disciplines that happen to feed into the same downstream transformer, each with its own compression trade-off, its own published benchmark, and its own honest limits. A specification sheet that says “handles video, audio, and 3D” without saying how each was tokenized, sampled, and fused is not yet a claim that can be checked.