$$\mathcal{L} = -\frac{1}{2B}\sum_{i=1}^{B}\left[\log\frac{\exp(\langle u_i, v_i\rangle/\tau)}{\sum_{j=1}^{B}\exp(\langle u_i, v_j\rangle/\tau)} + \log\frac{\exp(\langle u_i, v_i\rangle/\tau)}{\sum_{j=1}^{B}\exp(\langle u_j, v_i\rangle/\tau)}\right]$$

Equation 5 · Foundation Models

A History of Multimodal AI

This equation states an equality: the expressions on both sides have the same value under the article’s assumptions.

Radford and colleagues’ 2021 paper, “Learning Transferable Visual Models From Natural Language Supervision,” is usually remembered simply as CLIP, and the compression loses the specific thing that made it a turning point rather than an incremental improvement. The method itself is not exotic: encode an image, encode its paired caption, and train both encoders so that the true pairing scores higher than every mismatched pairing drawn from the same batch. For a batch of B image-text pairs with normalised image embedding $u_i$ , text embedding $v_i$ , and a learned temperature $\tau$ , the symmetric form of the objective is $\mathcal{L} = -\frac{1}{2B}\sum_{i=1}^{B}\left[\log\frac{\exp(\langle u_i, v_i\rangle/\tau)}{\sum_{j=1}^{B}\exp(\langle u_i, v_j\rangle/\tau)} + \log\frac{\exp(\langle u_i, v_i\rangle/\tau)}{\sum_{j=1}^{B}\exp(\langle u_j, v_i\rangle/\tau)}\right]$. an image-to-text and a text-to-image cross-entropy…

Meanings in this article

• $\mathcal{L}$: the symmetric form of the objective.
• $u_i$: the normalised image embedding.
• $v_i$: the text embedding.
• $\tau$: the learned temperature.

Equation guide → · Article →