$$\mathcal{L} = -\frac{1}{B} \sum_{i=1}^{B} \log \frac{\exp(\langle u_i, v_i \rangle / \tau)}{\sum_{j=1}^{B} \exp(\langle u_i, v_j \rangle / \tau)}$$

Equation 6 · Foundation Models

One Model, Many Modalities: What Multimodal Systems Actually Share

This equation states an equality: the expressions on both sides have the same value under the article’s assumptions.

Alignment is the step that made cross-modal retrieval work, and it has an unusually clean formulation. Contrastive language–image pretraining takes a batch of B image–text pairs, encodes each side separately into a normalised vector, and trains both encoders so that matched pairs score higher than mismatched ones. In its softmax form the objective is $\mathcal{L} = -\frac{1}{B} \sum_{i=1}^{B} \log \frac{\exp(\langle u_i, v_i \rangle / \tau)}{\sum_{j=1}^{B} \exp(\langle u_i, v_j \rangle / \tau)}$. with image embedding $u_i$ , text embedding $v_i$ , and a learned temperature $\tau$ . Radford and colleagues showed that this simple pre-training task, applied to 400 million image–text pairs collected from the internet, matched the accuracy of the original ResNet-50 on ImageNet zero-shot without using any of the 1.28 million…

Meanings in this article

• $\mathcal{L}$: In its softmax form the objective.
• $u_i$: the image embedding.
• $v_i$: the text embedding.
• $\tau$: the learned temperature.

Equation guide → · Article →