$$\Delta_m = \mathrm{Acc}_{\mathrm{full}} - \mathrm{Acc}_{\mathrm{ablate}(m)}$$

Equation 7 · Foundation Models

Building a Multimodal AI Application That Actually Uses Its Inputs

This equation states an equality: the expressions on both sides have the same value under the article’s assumptions.

The formal version is a simple, reportable metric. For a task with a full-input accuracy $\mathrm{Acc}_{\mathrm{full}}$ and an accuracy $\mathrm{Acc}_{\mathrm{ablate}(m)}$ measured with modality m removed, blanked, or replaced with noise, define $\Delta_m = \mathrm{Acc}_{\mathrm{full}} - \mathrm{Acc}_{\mathrm{ablate}(m)}$. A small $\Delta_m$ for a modality the task specification says should matter is the signature of a unimodal shortcut: the model is scoring well on the full-input evaluation without actually depending on that channel. This is not a hypothetical failure mode. Agrawal and colleagues showed that VQA models trained and evaluated under the field’s original data splits could reach strong scores while relying heavily on the language prior in…

Meanings in this article

• $\Delta_m$: the small.

Equation guide → · Article →