What does this equation mean?

The first is measurement-time ablation : take an already-trained system, rerun the exact same benchmark items with one channel removed at inference, and compare. This is what the original VQA paper did with its question-only baseline [ 1 ] , and it is what a 2024 audit of vision-language benchmarks did formally, by defining two paired metrics from three separately measured accuracies. Let $S_v$ be a model’s accuracy on a benchmark with the image present, $S_{wv}$ its accuracy on the same items with the image withheld, and $S_t$ the accuracy of that same model’s underlying text-only language backbone, evaluated on its own before any multimodal training touched it. The audit’s two metrics are then $MG = S_v - S_{wv}, \qquad ML = \max(0,\ S_{wv} - S_t)$. where MG , Multimodal Gain, is how much the image actually added once everything else is held constant, and ML , Multimodal Leakage, is the part of the no-image score that exceeds what the model’s own text backbone could already do unaided — evidence not that the benchmark is answerable from language in general, but that this specific model’s multimodal training pipeline let benchmark answers leak in through some channel other than genuine visual grounding [ 10 ] . The distinction matters because the two problems have different remedies: a benchmark with low MG needs harder, more vision-dependent items; a model with high ML needs a cleaner training pipeline. Collapsing both into a single “the model cheated” headline erases exactly the information a fix would require.