Tagged multimodal models · show all articles
A model that sees, hears and reads still returns one fluent answer, and fluency hides the fact that the link between that answer and what was actually there breaks in ten distinct, independently documented ways.
Image-text systems tokenize a grid and align two vectors. Video is a spacetime volume, audio is a waveform, and a 3D scene is a continuous field — none of them arrives as a grid, and each forces its own discretization scheme before any model can read it.
A model that accepts an image is not the same as a model that uses one. A working guide to the decisions that separate the two: API or adapter, what the training pairs must contain, how to test for the unimodal shortcut, and what the tokens actually cost.
Four research lineages solved the same problem — joining vision, audio and text into one system — with four different training recipes. Each recipe holds one cost fixed and lets another float, and the documented trade is an engineering choice, not a hierarchy.
A printed picture exists only when separately produced channels register with one another. Multimodal models are assembled the same way, and the join — not the parts — is where the claim of a single unified system is won or lost.