subtraction

The passage around this formula

The formal version is a simple, reportable metric. For a task with a full-input accuracy $\mathrm{Acc}_{\mathrm{full}}$ and an accuracy $\mathrm{Acc}_{\mathrm{ablate}(m)}$ measured with modality m removed, blanked, or replaced with noise, define $\Delta_m = \mathrm{Acc}_{\mathrm{full}} - \mathrm{Acc}_{\mathrm{ablate}(m)}$. A small $\Delta_m$ for a modality the task specification says should matter is the signature of a unimodal shortcut: the model is scoring well on the full-input evaluation without actually depending on that channel. This is not a hypothetical failure mode. Agrawal and colleagues showed that VQA models trained and evaluated under the field’s original data splits could reach strong scores while relying heavily on the language prior in…

Read this part in the article →