What does this equation mean?

Alignment is the step that made cross-modal retrieval work, and it has an unusually clean formulation. Contrastive language–image pretraining takes a batch of B image–text pairs, encodes each side separately into a normalised vector, and trains both encoders so that matched pairs score higher than mismatched ones. In its softmax form the objective is $\mathcal{L} = -\frac{1}{B} \sum_{i=1}^{B} \log \frac{\exp(\langle u_i, v_i \rangle / \tau)}{\sum_{j=1}^{B} \exp(\langle u_i, v_j \rangle / \tau)}$. with image embedding $u_i$ , text embedding $v_i$ , and a learned temperature $\tau$ . Radford and colleagues showed that this simple pre-training task, applied to 400 million image–text pairs collected from the internet, matched the accuracy of the original ResNet-50 on ImageNet zero-shot without using any of the 1.28 million training examples that network was trained on [ 1 ] . A later variant replaces the softmax with a pairwise sigmoid loss that does not require a global view of the batch’s pairwise similarities for normalisation, and the authors report training a model to 84.5% ImageNet zero-shot accuracy in two days on four TPUv4 chips, while also finding that the benefit of larger batches saturates around 32,000 examples [ 6 ] .