=

The passage around this formula

The fourth lineage answers a different question than the first three. Adapters, native pretraining and unified tokenization are all, in the end, recipes for understanding or jointly representing multiple modalities; diffusion is a recipe for generating one, typically conditioned on another, and it does not use next-token prediction at all. A diffusion model learns to reverse a fixed process that gradually adds Gaussian noise to data, training a network to predict the noise component at each step: $\mathcal{L}_{\text{diffusion}} = \mathbb{E}_{x_0,\, \epsilon \sim \mathcal{N}(0, I),\, t} \Big[\, \big\| \epsilon - \epsilon_\theta(x_t, t, c) \big\|^2 \,\Big], \qquad x_t = \sqrt{\bar{\alpha}_t}\, x_0 + \sqrt{1 - \bar{\alpha}_t}\, \epsilon$. where c is a conditioning signal — most often a text embedding — and generation runs the process in reverse, starting from pure noise and repeatedly subtracting a predicted…

Read this part in the article →