What does this equation mean?

Meta’s Chameleon commits to the same principle at a much larger scale and is candid that stability, not capability, was the hard engineering problem. It describes itself as “a family of early-fusion token-based mixed-modal models” that interleaves image and text tokens in one sequence, generating either kind at any position, and its authors state directly that this “requires a stable training approach from inception, an alignment recipe, and an architectural parameterization tailored for the early-fusion, token-based, mixed-modal setting” [ 7 ] . The paper documents the failure mode this addresses concretely: without a query-key normalisation step controlling the growth of attention logits, Chameleon-7B diverges after roughly 20 percent of a single training epoch, a failure the authors attribute to the softmax operation behaving badly across tokens of very different entropy once image and text tokens share one sequence [ 7 ] . The 34-billion-parameter variant needed a second, distinct intervention — reordering where layer normalisation sits within the transformer block — and both model sizes were trained with an added z-loss regularisation term penalising drift in the softmax normaliser [ 7 ] . The reported scale is substantial on its own terms: 4.4 trillion tokens, with the 7-billion-parameter run consuming roughly 856,000 GPU-hours and the 34-billion-parameter run roughly 4.28 million GPU-hours [ 7 ] . Meta’s earlier CM3 model is the more modest ancestor of the same idea, treating structured web documents — text, hyperlinks and VQVAE-encoded image tokens — as one causally-masked token stream so that a single model can perform captioning, zero-shot text-to-image generation and entity-linking without task-specific heads [ 8 ] . The same cross-entropy objective a text-only language model uses is the entire training objective here — the only thing that changed is what $z_t$ is allowed to denote. Whether index $z_t$ names a subword, a 16-by-16 image patch code, or a discretised robot joint angle is invisible to the loss function; the difficulty this section documents is not in the objective but in keeping training numerically stable once the vocabulary spans quantities of such different statistical character.