One word doing four jobs
Every frontier lab now advertises “multimodal” capability, and the word does a lot of quiet work. It bundles at least four separable competencies: understanding a still image, understanding a moving video — a picture track and a soundtrack, together, over time — understanding spoken or ambient audio on its own, and understanding a scanned or born-digital document. A vendor can be genuinely strong on one of these and simply undocumented on another, and marketing copy rarely marks the difference. This article works through what OpenAI, Google and Anthropic each actually document about each of the four, using their own model cards, system cards and API documentation, plus independent academic benchmarks, rather than product pages.
One finding is worth stating plainly before the caveats arrive, because this series has under-covered it until now: Google’s Gemini 3 Pro is the only one of the three current flagship models whose own technical documentation describes native, joint processing of a video’s picture and its soundtrack in a single pass, and native audio understanding built into the same reasoning model rather than parked in a separate product line [1, 5, 6]. That is not a marketing claim pulled from a launch blog; it is what each vendor’s own primary documentation says, or conspicuously does not say. It is also not the whole story. Image understanding has become a converged, commodity capability across all three vendors. Document and PDF understanding has converged in method, if not in accuracy. And the vendor-published tables comparing one company’s model against a competitor’s are, on inspection, comparing systems run under different tools, different effort settings and sometimes different dataset subsets — a methodological problem this article documents directly, with the vendors’ own footnotes as evidence.
Image understanding: the converged layer
Start with the capability that is no longer a differentiator. OpenAI’s current API documentation for images and vision defines a mature, static-image pipeline: images can be supplied as a URL, a base64-encoded payload, or a Files API reference, in PNG, JPEG, WEBP or non-animated GIF format, with payloads up to 512 megabytes and as many as 1,500 individual images per request [7]. Anthropic’s Claude documents a comparably mature pipeline with a different cost model: images are read in 28-by-28-pixel patches, each patch a “visual token,” so an image’s price scales with its pixel count rather than its byte size. Claude 4.7 and later models process images up to 2,576 pixels on the long edge and 4,784 visual tokens, roughly three times the ceiling of the 1,568-pixel, 1,568-token limit that applied to prior Claude generations [10]. Anthropic’s own Claude Opus 4.7 System Card ties a concrete capability gain to that specific resolution increase: on LAB-Bench FigQA, a benchmark of reading complex scientific figures, accuracy without tools rose from 74.0 percent to 78.6 percent purely from processing evaluation images at the new, less-downsampled resolution, holding the model itself constant [4].
Google’s Gemini 3 Pro Model Card reports the same broad picture from the other direction: mature image understanding is now table stakes. On MMMU-Pro, an image-understanding and reasoning benchmark built by embedding exam-style questions directly inside images so a model cannot solve them from text alone, Google’s own comparison table reports Gemini 3 Pro at 81.0 percent, GPT-5.1 at 76.0 percent, and Claude Sonnet 4.5 at 68.0 percent [1]. The independent academic version of that benchmark, MMMU-Pro, was built for exactly this purpose: filtering out questions answerable from text alone, expanding the answer choices to reduce guessing, and testing a vision-only setting where the question itself is rendered as an image. Across the models the paper’s authors tested, that redesign dropped accuracy by 16.8 to 26.9 percentage points relative to the original, easier MMMU benchmark — a reminder that “image understanding” scores are only as meaningful as the benchmark’s resistance to being solved without really looking at the image [12]. All three vendors clear that harder bar at a similar order of magnitude today. Image understanding is where “multimodal” stopped being a claim worth interrogating vendor by vendor.
Video understanding: the one place Gemini’s documentation genuinely stands apart
Video is where the documentation actually diverges, and the difference is architectural rather than cosmetic. Google’s Gemini API documentation describes video as a first-class input: a video file is sampled visually at one frame per second, its audio track is processed jointly with the visual stream, and a caller can query specific moments using MM:SS timestamps. Within a 1-million-token context window, Gemini can process video up to one hour long at default resolution or three hours long at low resolution in a single request [5]. The Gemini 3 Pro Model Card describes the underlying architecture directly: a sparse mixture-of-experts transformer “with native multimodal support for text, vision, and audio inputs,” trained on a pretraining corpus that explicitly included video alongside images and audio, accepting “text strings …, images, audio, and video files” as input [1]. On Video-MMMU, a benchmark of knowledge acquisition from instructional video that Google reports in its own comparison table, Gemini 3 Pro scores 87.6 percent against 80.4 percent for GPT-5.1 and 77.8 percent for Claude Sonnet 4.5 [1].
The documented token economics make the architectural claim concrete rather than rhetorical. Gemini’s video pricing bundles the visual and audio streams into one per-second cost:
where, at default resolution, the sampling rate
OpenAI’s documentation shows a different architecture for the same problem. The current API reference for images and vision defines exactly the four static image formats above and does not mention video as an input type at all [7]. OpenAI’s own developer cookbook is explicit about the workaround: “Although GPT-4.1-mini doesn’t take videos as input directly, we can use vision and the 1M token context window to describe the static frames of a whole video at once.” The worked example extracts frames from an eighteen-minute, 618-frame clip using OpenCV and submits every twenty-fifth frame — a sparse sample of still images, not a video stream — while audio is not understood at all in that pipeline; it is generated afterward, separately, as narration through a text-to-speech model [9]. That is a coherent and often adequate engineering pattern, and it can work well for tasks that tolerate a coarse temporal sample. It is not the same capability as a documented native video-understanding endpoint, and OpenAI’s own materials do not claim that it is.
Anthropic’s documentation does not describe a video input path at all. Claude’s content-block types, as documented for the Messages API, are text, image and document; the image content block accepts JPEG, PNG, GIF and WebP, with an explicit note that “animations are unsupported, and only the first frame is used” [10]. There is no video content-block type in the documentation, and the 193-page Claude Opus 5 System Card — Anthropic’s most current flagship system card, published July 2026 — contains no evaluation of video input anywhere in its text [3]. Independent research corroborates that this is a real, structural gap rather than an omission from one document: the peer-reviewed Video-MME benchmark, which explicitly integrates “video frames plus subtitles and audio” as its multimodal input, reported that Gemini 1.5 Pro was “the best-performing commercial model, significantly outperforming the open-source models” it tested at the time of publication — a finding about an earlier Gemini generation, but consistent with a pattern of Google’s video architecture leading on a benchmark purpose-built to require joint audio-visual reasoning over time [13].
Audio understanding: native, bolted-on, or undocumented
The audio picture largely mirrors video, for a related architectural reason. Gemini’s documentation lists native audio understanding as a core capability of the same model that handles text, image and video: transcription and translation, speaker diarisation, emotion detection in speech and music, free-form description and question-answering over audio content, and recognition of non-speech sounds such as birdsong or sirens. Audio costs 32 tokens per second of input (1,920 tokens per minute), and a single prompt can carry up to 9.5 hours of audio [6].
OpenAI documents real and capable audio understanding, but in a separate model family from the text-and-vision reasoning line. Its audio guide names gpt-audio-1.5 and gpt-realtime-2.1 as the models that are “natively multimodal, meaning they can understand and generate audio and text as input and output,” accessed through the Realtime API over WebRTC, SIP or WebSocket, or through Chat Completions using an input_audio content type [8]. That is a genuine native-audio architecture — but it sits beside GPT-5.5, not inside it. The GPT-5.5 System Card’s dedicated safety section on multimodal input, titled simply “Vision,” evaluates only image-input safety across categories like hate, extremism, self-harm and sexual content; the word “audio” does not appear anywhere in the 45-page document [2]. Whatever GPT-5.5’s reasoning capability is, its own primary safety documentation was not written to cover audio input to that model.
Anthropic’s documentation is the most restrictive of the three on this axis: there is no audio content-block type in the Claude API, and a full-text search of the 193-page Claude Opus 5 System Card turns up zero mentions of “audio” anywhere in the document [3]. The card states plainly that “the model outputs text only,” and its dedicated multimodal evaluation section, 8.12, covers four benchmarks — a chart-reading benchmark, a CAD-from-images benchmark, a computer-use/screenshot benchmark, and a professional-document benchmark — all image- or document-based, none audio-based [3].
None of this means audio understanding is a solved problem for the vendor that documents it best. The independent MMAU benchmark, built from 10,000 curated audio clips spanning speech, environmental sound and music across 27 distinct skills, reported in 2024 that “even the most advanced Gemini Pro v1.5 achieves only 52.97% accuracy, and the state-of-the-art open-source Qwen2-Audio achieves only 52.50%” [14]. Those are older models by two generations now, and the gap between them was negligible regardless of open- or closed-source status — a useful corrective against reading “Gemini documents native audio” as “Gemini has solved audio reasoning.” A parallel independent benchmark, OmnixR, built specifically to test reasoning that requires integrating information across text, vision and audio together, evaluated GPT-4o and Gemini as the state-of-the-art omni-modality systems of its era and found that “all state-of-the-art OLMs struggle with OmnixR questions that require integrating information from multiple modalities to answer” [15]. Native architecture is a documented, verifiable fact about which input paths exist. It is not, by itself, evidence about how reliable the resulting reasoning is — a distinction this article is careful to keep separate.
Document and PDF understanding: converging in method, not in accuracy
Document understanding is the axis where the three vendors have quietly arrived at the same engineering answer. Anthropic’s PDF-support documentation is the most explicit about the mechanism: “the system converts each page of the document into an image,” and “the text from each page is extracted and provided alongside each page’s image,” so that “Claude analyzes both the text and images to better understand the document,” which is why PDF support “relies on Claude’s vision capabilities” and inherits the same limitations [11]. That two-channel method — render the page as an image, extract its text, and give the model both — is the same shape Gemini describes for documents within its general multimodal input, and it is consistent with how OpenAI’s vision pipeline handles any image-bearing content, PDF pages included: as static images processed through the same endpoint as photographs.
Where the three vendors differ is capacity and accuracy, not method. Claude’s documented request limits allow up to 600 pages or 32 megabytes per request, dropping to 100 pages for models with a smaller context window [11]. On OmniDocBench, a document-conversion benchmark scored by edit distance from a ground-truth transcription (lower is better), Google’s own comparison table reports Gemini 3 Pro at 0.115 against Claude Sonnet 4.5 and Gemini 2.5 Pro tied at 0.145 and GPT-5.1 at 0.147 [1] — a real gap, but a much smaller one, proportionally, than the gaps this article documents for video and audio. Anthropic’s own Claude Opus 5 System Card reports a separate, internally consistent document-understanding result on a benchmark called GDP.pdf, built from 100 real-world professional documents spanning finance, healthcare, legal, engineering and insurance workflows: Claude Opus 5 scored a mean criteria pass rate of 83.4 percent without tools and 85.5 percent with tools, against 77.5 percent and 84.8 percent for the prior Claude Opus 4.8 [3]. That comparison is entirely internal to Anthropic’s own model family — it says nothing directly about GPT-5.5 or Gemini on the same benchmark — and it is exactly the kind of number this article’s next section explains how to read correctly.
Why the vendor comparison tables can’t be read as a ranking
The editorial rule against building a cross-vendor ranking from incomparable benchmarks is not an abstract caution here; the vendors’ own published tables demonstrate the problem directly, in their own footnotes. Anthropic’s Claude Opus 5 System Card reports its Chartography chart-reading benchmark for Claude Opus 5 under two conditions — with adaptive thinking at maximum effort, both with and without agentic tools that let the model crop and re-examine the image — scoring 29.6 percent without tools and 83.0 percent with tools. The same figure carries a footnote reporting competitor numbers: “Gemini 3.5 Flash and GPT-5.6 Sol scores as publicly reported by Surge AI, evaluated without tools” [3]. Those competitor scores were not measured by Anthropic’s own harness; they were pulled from a third party’s public report, under a tool-use condition that may or may not match how each vendor would score its own model with its own tools enabled. The same system card’s BenchCAD Vision2Code result carries a parallel footnote: “GPT-5.6 Sol scores as publicly reported by OpenAI, evaluated on the full 17,900 files,” while Anthropic’s own Claude scores on that benchmark come from “a random 1,000 file subset” of the same dataset [3] — different tool conditions and a fourteen-fold difference in evaluated dataset size, sitting in the same comparison chart.
Google’s own comparison table has a structurally similar problem, even without an explicit footnote flagging it: it reports Gemini 3 Pro, Gemini 2.5 Pro, Claude Sonnet 4.5 and GPT-5.1 on the same rows of benchmarks including MMMU-Pro, CharXiv Reasoning, ScreenSpot-Pro and OmniDocBench, without disclosing what reasoning effort, tool access or agentic scaffolding was used for the two competitor models [1]. Compare Anthropic’s own, separately reported CharXiv Reasoning score for a different Claude model, Opus 4.7: 82.1 percent without tools and 91.0 percent with Python tools enabled, both scores far above the 68.5 percent Google’s table attributes to “Claude Sonnet 4.5” on the same benchmark [4, 1]. Both numbers can be true simultaneously — they describe different Claude models under different tool conditions — and that is exactly the point: a benchmark name and a vendor logo are not enough to make two numbers comparable. This is not a claim that any vendor is being dishonest; publishing a competitor’s own publicly reported number, with a footnote saying so, is more transparent than many alternatives. It is a demonstration of why a reader cannot compile these tables into a single ranking without independently controlling for effort level, tool access and evaluated dataset, which is precisely the discipline this publication’s editorial standards require and vendor marketing pages routinely skip.
The gaps that are real, and the ones that are just documentation gaps
Sorting the evidence above into what is architecturally real versus what merely reflects which document a vendor chose to write:
Video is a genuine, documented architectural gap. Gemini’s own API documentation and model card describe native joint audio-visual processing with concrete token costs; OpenAI’s own cookbook describes frame extraction as the workaround because native video input does not exist in its API; Anthropic’s documentation contains no video input path and no video evaluation in its most current system card. This is not a matter of interpretation — it is what each vendor’s own materials state or omit [5, 9, 3].
Audio is a genuine architectural gap, but a split one. OpenAI has real native audio capability, just not inside the model most people mean when they say “GPT-5.5” — it lives in gpt-realtime and gpt-audio instead [8]. Anthropic documents no audio input path at all, in any product [10, 3]. Google is the only vendor of the three whose flagship reasoning model documents audio as a native input alongside text, image and video in one architecture [1].
Image understanding is converged, differentiated mainly by resolution ceiling and token cost rather than by whether the capability exists [10, 7, 1].
Document and PDF understanding is converged in method — render pages as images, extract text, combine both — and differentiated by page-count limits and by accuracy on dense, chart-heavy professional documents, an axis where the gaps are real but proportionally the smallest of the four [11, 1].
Predictions, and what would falsify them
These are forecasts, kept explicitly separate from the sourced analysis above. Horizon: August 2028.
One. OpenAI’s and Anthropic’s flagship reasoning-model system cards will each document a native video-input path — not a frame-extraction workaround, not a separate product line — by the horizon date, given OpenAI’s existing native-audio work in gpt-realtime and Anthropic’s demonstrated pattern of rapid resolution increases for image input. Disconfirmed if by August 2028 either vendor’s flagship system card still routes video exclusively through frame extraction or omits video evaluation entirely.
Two. Vendor-published cross-competitor comparison tables will increasingly disclose the tool-use and effort-level conditions under which competitor numbers were obtained, following scrutiny of exactly the mismatches documented in this article. Disconfirmed if the practice of citing bare “publicly reported” competitor scores without matching harness conditions is unchanged or more common in 2028 system cards than it is in 2026.
Three. Independent, non-vendor audio-visual reasoning benchmarks in the lineage of MMAU, Video-MME and OmnixR will report a narrowing of the gap between Gemini and the other two vendors on joint audio-visual tasks specifically, contingent on prediction one holding. Disconfirmed if the gap on independent multimodal benchmarks remains as wide in 2028 as it is documented here in 2026, despite vendor claims of parity.
What to take away
“Multimodal” is not one capability with a single winner. Image understanding is a converged commodity today, present and broadly comparable across OpenAI, Google and Anthropic. Document and PDF understanding has converged on the same method — render, extract, combine — with the remaining gaps concentrated in page limits and dense-table accuracy. Video and audio are where the real, documented architecture differs: Gemini’s own materials describe a single natively multimodal model handling text, image, audio and video together, with the token economics to prove it is billed as one combined stream; OpenAI’s own materials describe a workaround for video and a separate model family for audio; Anthropic’s own materials describe neither capability at all in its current flagship documentation. None of that licenses a single ranking — the vendors’ own footnotes show competitor numbers gathered under mismatched tools, mismatched effort and mismatched dataset sizes, sitting inside otherwise rigorous system cards. Read any specific claim against its specific source, ask what modality it actually covers, and treat the absence of a capability in a 200-page system card as data in its own right.
