Building a Multimodal AI Application That Actually Uses Its Inputs

Accepting an image is not the same as using one

A team ships an endpoint that takes an image and a question and returns an answer. The demo works. Then someone runs the ablation nobody asked for — remove the image, keep the question — and the answer barely changes. This happens more often than “multimodal” marketing suggests, and it happens for a structural reason: nothing in a standard training objective forces a model to rely on a channel just because that channel is architecturally present. The clearest documented instance is old but has never stopped recurring: building a visual question answering benchmark in which every question is paired against two images with different correct answers exposed how much of the field’s apparent visual competence had actually been language pattern-matching in disguise ^[3]. That result is from 2018. The lesson has to be relearned per project, because nothing about a multimodal architecture prevents it from happening again on a new dataset with a new team.

This article is a practitioner’s route through the decisions that determine which side of that gap a shipped system lands on: whether to call a frontier model’s native multimodal API or fine-tune an open adapter-based model, how to curate the paired data either path depends on, how to build a test suite that catches a unimodal shortcut before a customer does, how to budget the token and dollar cost of image- and video-heavy workloads, and the integration failures that break a working pipeline without raising an error. Every recommendation below traces to a vendor’s own documentation or a published, reproducible method — not to received wisdom about what “should” work.

Frontier API or open adapter: the first decision, made correctly

The decision is not “which model is smarter.” It is a question about where the fitting happens and who controls it, and it has a defensible default: start with a frontier model’s native multimodal API, and move to an open adapter-based model only when a specific, measured limitation of the API forces the move.

The API route means the vision or audio encoder, the fusion mechanism, and the base weights are entirely the vendor’s concern. The integrator’s job shrinks to prompt design, input preprocessing, and orchestration — a genuinely smaller surface to get wrong. It also means the model updates on the vendor’s schedule, sight unseen, which is a cost as well as a benefit; documented mid-generation changes to model behaviour and serving are common enough in this market that a system built only on unpinned defaults is implicitly agreeing to be re-tested by its vendor’s release calendar rather than its own.

Some frontier vendors also expose a narrower form of customization inside the API path itself. OpenAI’s vision fine-tuning guidance describes a workflow for adapting a hosted vision-capable model on labelled image examples — JSONL training files with image content blocks, up to ten images per example, each image capped at ten megabytes, formats restricted to JPEG, PNG, or WEBP in RGB or RGBA mode — aimed specifically at image classification and at correcting instruction-following failures on complex prompts ^[7]. Two facts from that same documentation matter as much as the mechanism: image content is passed through automatic moderation that excludes photographs containing people, faces, children, or CAPTCHAs before training even starts, which quietly removes an entire category of use case from consideration; and, as of this documentation, the vendor states it is winding down the general fine-tuning platform for new users even as it keeps it available to existing ones ^[7]. That second point is a vendor disclosure, not a technical limitation, and it belongs in any build-vs-fine-tune decision precisely because it is a business-continuity fact rather than a capability fact: a fine-tuning dependency on a platform the vendor is deprecating is a different kind of risk than a dependency on a stable one, regardless of how good the fine-tuned result is.

The open adapter-based route is the right default only once one of three conditions holds: the domain shift from the frontier model’s training distribution is large enough that in-context examples cannot close the gap on a held-out set (a specialised imaging modality, a proprietary sensor format, a house style no prompt reliably reproduces); the deployment has a hard constraint the API cannot satisfy (on-premises inference, data residency, a fixed per-request latency budget under active load); or the request volume is high enough that the marginal cost of a self-hosted adapter beats the marginal cost of API tokens once amortised training cost is included — a calculation that has to be done with real numbers, not assumed.

When the adapter route is chosen, the engineering default within it is equally clear: adapt, do not retrain. Low-Rank Adaptation freezes the pretrained weights and injects a pair of small trainable matrices into selected layers, so that a weight update is expressed as a low-rank product rather than a dense matrix the size of the original layer,

with the forward pass computing $h = W_0 x + \Delta W x$ against the frozen base weight $W_0$. The method’s authors report reducing the number of trainable parameters by roughly ten thousand times and GPU memory requirements by roughly three times relative to full fine-tuning of a 175-billion-parameter model, while matching or exceeding full fine-tuning quality and adding no additional inference latency, since the low-rank update can be merged back into the base weight at deployment time ^[1]. QLoRA extends the same idea to quantized base weights, and its authors report finetuning a 65-billion-parameter model on a single 48-gigabyte GPU while preserving full 16-bit finetuning performance, with their best resulting model reaching 99.3 percent of a reference chat model’s quality after twenty-four hours of finetuning on one GPU ^[2]. Hugging Face’s PEFT library packages LoRA and related adapter methods as a standard toolchain specifically so that adapting a large pretrained model no longer requires updating all of its parameters, which the library’s own documentation describes as “prohibitively costly” for most teams, while integrating directly with the standard training and inference stack ^[10].

For multimodal models specifically, the adapter principle extends past the language backbone to the join between modalities. LLaVA connects a frozen vision encoder to a language model through a simple learned projection and trains the whole assembly primarily by generating multimodal instruction-following data with a language-only GPT-4 rather than by hand-labelling images — the first published attempt at that specific data-generation trick — reporting a relative score of 85.1 percent against GPT-4 on a synthetic multimodal benchmark and 92.53 percent accuracy on a science question-answering benchmark after fine-tuning ^[12]. The generalizable lesson is not the specific architecture; it is that an open adapter-based system concentrates almost all of its engineering risk into two places — the quality of the bridge between modalities, and the quality of the data used to train it — and both are addressable with method rather than with more compute.

Curating paired data that survives contact with production

Every adapter-based path and every API-side fine-tuning path depends on the same unglamorous prerequisite: paired multimodal examples that are correct, representative, and — critically — separated cleanly into training and evaluation sets that never touch. Three practices, each backed by a published pipeline, do most of the work.

Filter at scale before curating by hand. DataComp treats dataset construction itself as the object of study rather than an afterthought: starting from a candidate pool of 12.8 billion image-text pairs harvested from Common Crawl, the project’s authors evaluate filtering strategies by training a standardized CLIP model on each candidate dataset and testing it across 38 downstream benchmarks, and they report that a well-filtered subset of the full pool — DataComp-1B — outperforms datasets built by other means at matched scale, including OpenAI’s original CLIP training data, on zero-shot ImageNet accuracy ^[5]. The operational takeaway for a smaller team is not to reproduce the scale, but to reproduce the method: filtering strategy is not a preprocessing footnote, it is the highest-leverage lever available before any model is trained, and it should be evaluated the same way DataComp evaluates it — by training on the candidate set and measuring downstream, not by inspecting the filter’s output for plausibility.

Prefer real, interleaved context over isolated pairs when the target task needs it. Most production multimodal tasks are not “caption this image” — they are “answer this question given this image and the surrounding document,” which is a different training distribution. OBELICS was built specifically to supply that distribution: 141 million web pages processed into interleaved image-and-text documents, yielding 353 million associated images and 115 billion text tokens, with an explicit filtering pipeline described alongside the release ^[4]. A team curating its own data for a document-, chat-, or UI-grounded application should mirror that structural choice — keep images attached to their surrounding text in the order they actually occurred — rather than flattening everything into isolated image-caption pairs, because a model trained only on isolated pairs has never seen the pattern it will be asked to use in production.

Treat the held-out set as inviolable, and prove it, not just declare it. The single most common curation failure is not bad data — it is data that leaked from evaluation into training, or a filtering step applied differently to the two splits. Weak, large-scale supervision compounds the risk: Whisper was trained on 680,000 hours of multilingual and multitask audio collected with comparatively light curation and demonstrated strong zero-shot transfer as a direct consequence of that scale ^[13], which is a genuine and reproducible result — but it also means large web-scraped audio and video corpora are exactly the kind of source where an evaluation clip can already exist verbatim somewhere in the training pool. The practical discipline is boring and non-negotiable: freeze the evaluation set before any filtering decision is finalised on the training set, run a similarity or near-duplicate check between the two, and treat any pipeline change after that freeze as requiring a new evaluation set rather than a note in a changelog.

A modality-ablation test suite, built to fail loudly

The single highest-leverage test a multimodal project can add, and the one most often skipped because it requires deliberately breaking the demo, is a modality-ablation suite: measuring how much performance drops when each input channel is degraded or removed in turn, on the same evaluation set used for the headline number.

The formal version is a simple, reportable metric. For a task with a full-input accuracy $\mathrm{Acc}_{\mathrm{full}}$ and an accuracy $\mathrm{Acc}_{\mathrm{ablate}(m)}$ measured with modality $m$ removed, blanked, or replaced with noise, define

A small $\Delta_m$ for a modality the task specification says should matter is the signature of a unimodal shortcut: the model is scoring well on the full-input evaluation without actually depending on that channel. This is not a hypothetical failure mode. Agrawal and colleagues showed that VQA models trained and evaluated under the field’s original data splits could reach strong scores while relying heavily on the language prior in the question rather than the image content, and that rebuilding the dataset so that identical questions paired with different images required different answers collapsed that shortcut and exposed the true, much lower, vision-grounded accuracy ^[3]. Geirhos and colleagues generalise the underlying failure beyond any one benchmark: a network trained by gradient descent to minimise a loss will find the least-effort decision rule that satisfies the training objective, and if a shortcut feature is available and correlates with the label on the training and evaluation distributions, the network will use it regardless of whether the shortcut is what the task designer intended to teach — a pattern the authors document across vision, language, and other domains, and one that standard held-out test accuracy does not detect if the shortcut happens to generalise as far as the test set does ^[14].

A modality-ablation suite worth shipping with the model has four concrete components, each addressing a way the naive version fails:

Ablate every modality the task claims to use, not just the obvious one. A document-question-answering system that takes an image, an OCR transcript, and a question has three channels, and each one can be silently carrying the task alone.

Ablate with a distribution-matched substitute, not a blank. Replacing an image with pure noise and replacing it with a plausible-but-wrong image test different things; a model that fails only against the second is exploiting fine-grained visual content, while one that fails against both is not using the channel at all. Report both.

Slice the ablation results by task subtype, because an aggregate delta hides subtypes where the shortcut is total. A visual question answering system can show a healthy aggregate $\Delta_{\mathrm{image}}$ while still answering every counting question from the question text alone.

Set a pre-registered threshold before looking at the numbers, and treat crossing it as a release blocker rather than a footnote. A team that decides after seeing a disappointing ablation that “the gap isn’t that bad, actually” has not run a test — it has run a negotiation.

None of this is exotic infrastructure. It is the same evaluation harness used for the headline metric, run several more times with one input systematically degraded, which is precisely why it is so often skipped under deadline pressure and precisely why skipping it is a false economy: the ablation is cheap relative to the cost of discovering the shortcut from a customer’s bug report after launch.

Budgeting tokens and cost for image- and video-heavy workloads

Image and video inputs are priced by tokenization rules that are public, mechanical, and different enough across vendors that a workload’s cost cannot be estimated from a single provider’s numbers and assumed to generalise. The general form, common to every tiling or patch-based image tokenizer, is

where $w$ and $h$ are the image’s pixel dimensions after any provider-side resize and $P$ is that provider’s patch edge length in pixels. The formula is the same shape everywhere; the constant $P$, the resize rule applied before it, and the price per resulting token are what an integrator actually has to look up and budget against.

OpenAI’s newer patch-tokenized models cover an image with 32-pixel-square patches and cap the total patch count per image, resizing proportionally down to fit the cap when an image would otherwise exceed it — for example, resizing an 1800-by-2400-pixel image down to 1056 by 1408 pixels to land at 1,452 patches against a 1,536-patch budget — while a detail: low setting instead fixes the cost at 85 tokens regardless of the original size by resizing to 512 by 512 first, and an original or auto detail setting preserves full resolution for coordinate-sensitive tasks at correspondingly higher token cost ^[6]. Anthropic’s documentation states the rule for Claude directly: an image costs $\lceil w/28 \rceil \times \lceil h/28 \rceil$ visual tokens, with a maximum native resolution — expressed as both a long-edge limit and a total visual-token cap — beyond which the image is downscaled before that formula is applied; the same documentation gives worked cost examples, noting that a 1920-by-1080 image costs on the order of 1,560 tokens on the standard resolution tier once resized, versus roughly 2,691 tokens if processed at full resolution on the high-resolution tier ^[8]. Google’s Gemini documentation prices video by time rather than by frame count directly: video is sampled at one frame per second by default, each sampled frame costs 258 tokens at default media resolution, and the documented aggregate rate is approximately 300 tokens per second of video at default resolution versus approximately 100 tokens per second at a lower media_resolution setting, with audio tracked separately at 32 tokens per second ^[9]. These are vendor-documented mechanisms, not a ranking — the right comparison for any given workload is each vendor’s own formula applied to that workload’s actual image and video dimensions, not a headline number lifted from one vendor’s example and compared against another’s.

Four levers follow directly from these formulas and matter more for image- and video-heavy workloads than for text-only ones, because the cost surface is quadratic in linear resolution rather than linear in content length:

Resize before sending, deliberately, not by accident. Since every documented formula resizes down to a cap before counting, sending a needlessly high-resolution source image does not buy fidelity above that cap — it only risks the image being resized by a rule the integrator does not control instead of one they chose, which can crop, letterbox, or interpolate differently than a purpose-built preprocessing step would.

Use the cheapest detail tier the task can tolerate, and verify that choice against real outputs, not intuition — OpenAI’s low detail and Gemini’s low media_resolution both exist specifically to trade fine-grained legibility for a large, predictable token reduction on tasks that do not need it ^{[6, 9]}.

Sample video, do not stream it whole. At a default of one frame per second, a ten-minute video already costs on the order of tens of thousands of tokens before any audio or text is added; a task that only needs to detect a handful of discrete events benefits far more from an application-level keyframe selector than from raising the sampling rate and asking the model to find the event itself.

Compute cost per workload, not cost per image, since the same nominal image can cost several times more or less depending on its actual pixel dimensions and the detail or resolution tier selected — a distinction the formulas above make unavoidable but that a flat “images cost X tokens” mental model hides.

Integration pitfalls: resolution, format, and normalization

The failures in this section share one property that makes them worse than a crash: none of them raises an error. A resized image, a mismatched colour space, and a missing normalization step all produce a valid tensor of the expected shape and a plausible-looking model output — just a systematically worse one, discovered only by comparing quality against a baseline that did preprocessing correctly.

Preprocessing normalization is part of the model’s contract, not a formatting nicety. CLIP’s reference implementation resizes each image, centre-crops it to the model’s expected input resolution, converts it to a tensor, and then normalizes each channel against a fixed mean and standard deviation baked into the preprocessing code rather than left to a default ^[11]. An integrator who swaps in a different, seemingly equivalent image-loading library, applies a generic ImageNet normalization instead of the encoder’s own, or skips normalization because “the pixels are already zero-to-one” is feeding the encoder inputs it was never trained on. The failure is silent because the encoder still produces an embedding — it is simply an embedding computed on shifted, rescaled input, and everything downstream inherits the shift without any component reporting an error.

Audio has the same failure mode in a different unit. Whisper’s documented training regime standardises on a fixed input representation for all audio regardless of source, which is precisely why it can be applied zero-shot across wildly different recording conditions ^[13]; an integration that instead forwards audio at its native, uninspected sample rate — a phone recording at one rate, a studio recording at another — and skips explicit resampling to the encoder’s expected rate is introducing a systematic distortion that, again, produces a confident, wrong output rather than an error.

Format mismatches compound with resolution mismatches instead of substituting for them. A JPEG re-encoded through several lossy passes, a colour space that has been silently converted from one standard to another, or an image mode that is RGBA when the pipeline downstream assumes RGB will each individually degrade quality; combined with an image also being resized by the provider’s own tiling rule, small preprocessing errors compound rather than average out, which is why format and colour handling deserve the same explicit validation as image dimensions rather than being assumed correct because “the file opened.”

Coordinate outputs do not survive a resize that the integrator forgot to invert. Any workflow that asks a model for bounding boxes, points, or other pixel coordinates is implicitly asking a question about the resized image the model actually saw, not the original file — both OpenAI’s and Anthropic’s documentation flag this explicitly and instruct integrators to remap returned coordinates back to the original image’s dimensions rather than trusting them directly ^{[6, 8]}. Skipping that remap step produces coordinates that are numerically valid and visually wrong, in a way that is easy to miss in a spot check and obvious the first time a bounding box is drawn on the original file.

The common thread across all four is that a preprocessing bug in a multimodal pipeline degrades quality rather than availability. The defence is the same in every case: validate preprocessing output against a known-good reference image or audio clip as part of the test suite, the same way the modality-ablation suite validates that the model is using its inputs at all — because a model correctly using inputs it received incorrectly is not actually a passing system.

Predictions, with the observations that would falsify them

These are forecasts, separated deliberately from the sourced analysis above. Horizon: 15 August 2028. Assumptions: continued commercial pressure toward image- and video-capable APIs, continued publication of vendor tokenization documentation, and no architectural discontinuity comparable to the original shift to transformer-based vision encoders.

One. Modality-ablation reporting will move from an optional practice to an expected line item in serious multimodal system documentation, in the way held-out contamination checks became standard for text benchmarks. Disconfirmed if widely deployed multimodal systems in 2028 still ship with no published ablation or shortcut analysis available even on request.

Two. Per-modality, granular cost controls — separate resolution and sampling settings for image versus video versus audio within a single request — will become more common across vendors rather than converging on one fixed setting, because the documented cost formulas already show the three modalities behave very differently under the same dial. Disconfirmed if major vendors converge on a single undifferentiated multimodal pricing tier with no per-modality control.

Three. Open adapter-based fine-tuning will remain the dominant customization path for narrow, high-volume, or on-premises multimodal deployments even as frontier APIs add more native fine-tuning options, because the cost and control trade favours adapters precisely in the regime where request volume is large enough to amortise training cost. Disconfirmed if published case studies show frontier API fine-tuning displacing open adapter methods at high request volume on cost grounds alone.

What to take away

Building a multimodal application is five separable engineering decisions wearing one label. Choose the frontier API by default and justify the move to an open adapter-based model with a measured limitation, not a preference. Curate paired data the way the field’s own published pipelines do — filter and evaluate at the dataset level, keep interleaved context when the task needs it, and treat the held-out set as something to prove untouched rather than merely declare separate. Build the modality-ablation suite before launch, not after a bug report, and set the threshold before looking at the results. Budget tokens against each vendor’s own documented tokenization formula, applied to real workload dimensions, not against a number borrowed from a different provider’s example. And validate preprocessing — resolution, format, colour space, normalization — with the same seriousness as the model output it feeds, because every failure in that layer is silent by construction.

None of this requires a better model. It requires treating “multimodal” as a property that has to be earned by the whole pipeline, verified the same way any other engineering claim is verified — by trying to break it before a customer does.