How Llama's Architecture Actually Works, Generation by Generation

Two different questions called “openness”

A previous piece in this series asked what a Llama release actually licenses — which of five separable artefacts ship, under what restriction, to whom ^[11]. That is a question about paper. This one is about metal and matrices: what is actually inside the checkpoint, what has stayed fixed across three generations of releases, and what Meta’s own papers and model cards say changed at each step. Cover the license terms entirely and the architecture underneath is untouched — which is exactly why the two questions need separate treatment.

The claim this article defends is narrower than “Llama got better.” It is that four specific, documented components have carried the entire family since 2023 — rotary positional encoding, grouped-query attention, a SwiGLU-gated feed-forward block, and RMSNorm — and that almost everything a reader is told changed between generations is a change to how those four components are configured, not a replacement of them. Sebastian Raschka’s running architecture comparison makes the same point about the wider field: from GPT-2 through DeepSeek-V3 to Llama 4, “the models are structurally similar,” with the visible history being an accumulation of engineering refinements rather than a series of redesigns ^[9]. Llama’s own three-generation record is a clean illustration of exactly that pattern, and it is precise enough to state with equations rather than adjectives.

The skeleton every generation shares

Strip a Llama model down and the residual stream is doing what every decoder-only transformer’s residual stream does: a token embedding is added to at each layer by an attention sublayer and a feed-forward sublayer, each wrapped in a normalization step and a skip connection. Llama 2’s paper describes the architecture as the original transformer with specific substitutions applied “mostly” unchanged from Llama 1: pre-normalization using RMSNorm, the SwiGLU activation function, and rotary positional embeddings, with grouped-query attention added for the larger models ^[1]. The Llama 3 herd paper keeps the same four components for models three orders of magnitude apart in deployed cost, from an 8-billion-parameter model to the 405-billion-parameter flagship ^[2]. Llama 4’s model card, describing a mixture-of-experts model with native multimodality, still specifies “auto-regressive language models” built on the same transformer decoder ^[4]. The routing around the feed-forward block changed in the newest generation; the block’s internal gating did not.

That stability is worth pausing on before working through what each piece does, because it reframes the rest of this article. A reader who wants to know “how does Llama 4 differ from Llama 2” is really asking two separable questions: which of the four load-bearing components had a parameter changed, and which additional machinery was bolted around them. The next three sections take the four components in turn, with their formal definitions; the sections after that take the three generations in turn.

Position as rotation, not a label

The first transformers gave each position a fixed or learned vector added to the token embedding. RoPE, introduced by Su and colleagues, does something different: it rotates the query and key vectors by an angle proportional to their position, so that positional information is encoded as a geometric transformation applied at attention time rather than as a value baked into the embedding before the network sees it ^[5].

Split each query or key vector of dimension $d$ into $d/2$ two-dimensional pairs. Pair $i$ is assigned a fixed rotation frequency

and a vector at sequence position $m$ has its $i$-th pair rotated by angle $m\theta_i$:

The property that makes this useful rather than merely elegant is that rotating a query at position $m$ and a key at position $n$ by these matrices before taking their dot product leaves a result that depends only on the distance between them:

Relative position falls out of the mechanism instead of being engineered into it separately, and — as the RoFormer paper’s own experiments emphasise — the same rotation formula applies at any sequence length the rotation frequencies were designed for, which is what makes RoPE the natural place to intervene when a later generation wants a longer context window ^[5].

The one free parameter is base, the constant that sets how quickly the rotation angles grow across the $d/2$ frequency bands. A small base makes low-index pairs rotate almost a full turn per token, encoding fine-grained short-range position; a large base slows that rotation down, which is what lets high-index pairs still carry usable positional signal at sequence positions far beyond what the model saw in training. Llama 1 and Llama 2 shipped with the RoPE default of base = 10,000, matching the value in Su and colleagues’ original formulation. Llama 3 raised it to 500,000 specifically as part of extending context to 128,000 tokens ^{[5, 2]}. Nothing about the rotation mechanism changed between the two generations. One constant did, because that constant is precisely the dial that trades short-range precision for long-range reach.

Attention thinned to groups

Multi-head attention gives every query head its own key and value projections. That is expensive at inference time for an autoregressive model, because every generated token requires reading the cached keys and values for every previous token, once per head, from memory — and memory bandwidth, not arithmetic, is usually the bottleneck in decoding. Multi-query attention collapses all query heads onto a single shared key-value pair, cutting that cache to a fraction of its multi-head size at some cost in quality. Ainslie and colleagues proposed the middle path that all three Llama generations actually use: grouped-query attention, in which $H$ query heads are partitioned into $G$ groups, and every query head within a group shares one key-value head pair ^[8]. The two earlier schemes are the boundary cases of the same construction:

Ainslie and colleagues’ own contribution was as much practical as architectural: they showed an existing multi-head checkpoint can be converted into a grouped-query one by mean-pooling its key and value heads and continuing training with roughly five percent of the original pretraining compute, rather than requiring the smaller attention footprint to be trained in from scratch ^[8].

Llama’s own adoption record shows the setting itself changing rather than the mechanism. Llama 2 applied GQA only to the 34B and 70B models, leaving the 7B and 13B models on full multi-head attention ^[1]. Llama 3 dropped that split: every released size, from the 8B model up through the 405B flagship, uses grouped-query attention with eight key-value heads ^[2]. That is a genuine change in engineering judgment — GQA had gone from an optimisation applied only where the KV-cache was already a proven bottleneck to a default applied everywhere, on the reasoning that the quality cost is small and the cache saving is worth taking even on the smallest deployed model.

Gating and rescaling: SwiGLU and RMSNorm

The other two components sit inside every layer’s feed-forward and normalization sublayers, and both replace a 2017-era default with a variant chosen for a documented empirical reason rather than for architectural novelty on its own.

Shazeer’s paper on gated linear unit variants tested several replacements for the standard ReLU or GELU feed-forward block, in which two linear projections are combined so that one, passed through a nonlinearity, gates the other ^[6]. The variant that stuck across the field, and that Llama adopts in every generation, uses the Swish (SiLU) nonlinearity as the gate:

Two independent linear projections of the residual stream are formed; one is passed through the gate and multiplied element-wise ($\odot$) against the other; the product is projected back down. Shazeer’s own results reported the gated variants outperforming the plain ReLU feed-forward block across the language modelling and fine-tuning tasks tested, at matched parameter and compute budgets ^[6]. Because the gate consumes an extra weight matrix, Llama’s feed-forward hidden dimension is set below the naive $4d$ multiplier used in the original transformer, to hold total parameters roughly constant against a non-gated block of the same width — an accounting detail visible in the hyperparameter tables of Meta’s own papers rather than a claim requiring independent verification here.

RMSNorm, from Zhang and Sennrich, made a narrower and more surgical change to normalization. Standard LayerNorm re-centres a layer’s inputs to zero mean and rescales to unit variance before applying a learned gain. Zhang and Sennrich’s hypothesis, tested empirically, was that the re-centring step is not doing useful work — that rescaling invariance alone accounts for LayerNorm’s benefit — and their RMSNorm drops the mean-subtraction entirely:

Their reported result was performance comparable to LayerNorm at a running-time reduction of roughly seven to sixty-four percent depending on the model, purely from removing the mean and its gradient computation ^[7]. Llama 2’s architecture section lists RMSNorm as the pre-normalization scheme applied before each attention and feed-forward sublayer, and it has not been revisited in any subsequent generation’s public documentation ^[1].

Llama 2: the baseline generation, stated in full

With the four components defined, the three generations can be described as configurations rather than as separate architectures. Llama 2 shipped 7B, 13B, 34B and 70B dense models, each trained on two trillion tokens with a context length of 4,096 — double Llama 1’s — a SentencePiece byte-pair-encoding tokenizer with a 32,000-token vocabulary, RMSNorm, SwiGLU, RoPE at the standard base frequency, and grouped-query attention on the 34B and 70B models only ^[1].

The paper is also unusually explicit about post-training methodology, which the assignment for this series asks to be treated as part of the architecture story rather than a separate concern. Llama 2-Chat’s alignment pipeline ran supervised fine-tuning on a curated set of roughly 27,500 high-quality demonstrations, followed by iterative rejection sampling — sampling many candidate responses per prompt, scoring them with a learned reward model, and keeping the best — and then Proximal Policy Optimization against a reward function combining separate helpfulness and safety models with a KL penalty against the reference policy ^[1]. A second, narrower technique called Ghost Attention (GAtt) addressed a specific failure: the model would follow a system instruction for a turn or two of a multi-turn conversation and then drop it. GAtt trains on synthetically constructed dialogues in which the system instruction is concatenated onto every user turn during data construction, with the loss zeroed out on the concatenated instruction tokens at training time, so the model learns to keep attending to the original instruction without being trained to reproduce it. Meta reports the effect held for twenty or more turns in evaluation ^[1]. Both mechanisms are training-time interventions on top of the fixed architecture described above, not architectural changes in themselves — the distinction the rest of this article depends on.

Llama 3: the same skeleton, extended reach

The Llama 3 herd paper documents a generation that changed the settings, not the parts. The tokenizer was replaced: a 128,000-token vocabulary combining 100,000 tokens from a tiktoken-style byte-level encoder with 28,000 tokens added for non-English coverage, up from Llama 2’s 32,000-token SentencePiece vocabulary ^[2]. Context length was extended from an initial 8,000 tokens to 128,000 through a staged continued-pretraining process — six discrete stages of gradual context expansion, each one validated by confirming the model recovered its short-context benchmark performance and passed needle-in-a-haystack retrieval tests at the new length before the next stage began ^[2]. RoPE’s base frequency was raised from 10,000 to 500,000 to support that extended range, exactly the mechanism described above ^[2]. Grouped-query attention, previously reserved for the two largest Llama 2 models, was applied uniformly with eight key-value heads across every released size including the 405B flagship ^[2].

The flagship model was trained on roughly 15.6 trillion tokens at an estimated 3.8 × 10²⁵ floating-point operations, over a data mixture the paper reports as approximately 50% general knowledge, 25% mathematical and reasoning content, 17% code, and 8% multilingual text, using up to 16,000 H100 GPUs ^[2]. The paper is candid about a deliberate departure from pure training-compute optimality: its own scaling-law fit put the compute-optimal size for the available budget at 402 billion parameters, and the authors chose 405 billion anyway, and separately trained their smaller models — the 8B and 70B — far past the point a compute-optimal recipe would stop, because “the resulting models perform better than compute-optimal models at the same inference budget” ^[2]. Training cost is paid once; inference cost recurs for the life of the deployment, and the paper treats that asymmetry as a design input rather than an afterthought.

Post-training changed algorithm, not just data. Where Llama 2 used Proximal Policy Optimization, the Llama 3 paper reports the team explored PPO and moved away from it: “we also explored on-policy algorithms such as PPO, but found that DPO required less compute for large-scale models and performed better,” settling on a pipeline of supervised fine-tuning, rejection sampling, and Direct Preference Optimization ^[2]. DPO removes the separate reward model and the online rollout loop, optimising directly against preference pairs — a genuine post-training architecture change, distinct from every parameter discussed so far, which is why this article keeps it in a separate category from RoPE, GQA, SwiGLU, and RMSNorm rather than folding it into “the architecture.”

Multimodality arrived in stages under the Llama 3 name rather than inside the 405B flagship itself. The initial herd paper describes image, video, and speech integration through a compositional approach — separately trained encoders and adapters attached to the frozen language model — explicitly noting those capabilities were not yet broadly released at publication ^[2]. Llama 3.2, released two months later, shipped the vision half of that plan: 11B and 90B models gained image understanding through a set of cross-attention adapter layers trained to feed image-encoder representations into the language model, while the core language-model weights were kept frozen throughout adapter training, preserving the existing text capabilities untouched ^[10]. The same release added 1B and 3B text-only models built by structured pruning of the 3.1 8B model combined with knowledge distillation from the 8B and 70B models’ output logits, aimed at on-device deployment with a 128,000-token context length carried over unchanged ^[10]. Every one of these is an addition bolted onto the existing four-component skeleton, not a modification of it — vision arrived as a satellite the text model does not depend on and can be swapped without retraining the base weights.

Llama 4: where the skeleton itself changed

Llama 4 is the first generation in which the architecture, not just its settings, moved. Meta’s announcement and model card describe Scout and Maverick as the company’s first mixture-of-experts Llama models: Scout has 109 billion total parameters routed across 16 experts with 17 billion active per token and a documented context length of 10 million tokens; Maverick has 400 billion total parameters across 128 experts, also with 17 billion active per token, alternating mixture-of-experts and dense feed-forward blocks across its layers, with a 1-million-token context length ^{[3, 4]}. A larger, unreleased Behemoth model, described as a teacher used in distilling Scout and Maverick, is reported at roughly two trillion total parameters with 288 billion active ^[3]. Scout was pretrained on roughly 40 trillion tokens and Maverick on roughly 22 trillion, both markedly larger than Llama 3’s 15.6 trillion, using FP8 training precision reported at 390 teraflops per GPU ^{[3, 4]}.

Mixture-of-experts changes what “the feed-forward block” means without changing the feed-forward block itself: instead of one SwiGLU-gated block processing every token, a router selects a small subset of expert blocks per token, and only the selected experts’ weights are read and computed against. Total parameter count and active parameter count separate for the first time in the family’s history, which is the entire economic argument for the design — Maverick carries roughly the parameter budget of a much larger dense model while paying the inference cost of a 17-billion-parameter one.

Long context in Llama 4 comes from a second architectural change layered on top of RoPE rather than a further adjustment of RoPE’s base frequency. Hugging Face’s technical write-up of the release describes an interleaved scheme — Meta calls it iRoPE — in which every fourth transformer layer drops positional encoding entirely and applies full, unchunked causal attention over the whole context, while the other three layers in each group of four keep standard RoPE but restrict attention to fixed 8,000-token chunks for memory efficiency ^[13]. The reasoning is that RoPE-based layers, however their base frequency is tuned, are still extrapolating a rotation trained at one length to a much longer one, while a layer with no positional encoding at all has no such extrapolation to fail — it processes tokens as an unordered set and lets attention patterns learned during training carry whatever positional structure remains useful. Meta’s own reporting adds an inference-time attention temperature adjustment and, in Scout, RMS normalization applied directly to query and key states without a learnable gain, both described as further length-generalisation aids layered onto the interleaved scheme ^[13].

Multimodality is where the architecture, rather than just the training scheme, diverges most clearly from Llama 3.2. Meta describes Llama 4 as using “early fusion” — text and image tokens enter and are processed by a single unified backbone from the start of pretraining, rather than a vision encoder producing representations that a separately trained cross-attention adapter feeds into an otherwise-frozen language model after the fact ^{[3, 4]}. That is a difference in when the two modalities meet the shared weights, not merely in how much data supports them: Llama 3.2’s vision adapter is bolted onto a base model that already exists and already works as a text model; Llama 4’s image handling is trained into the same weights that also learn to model text, from the beginning.

What changed, what didn’t, and what that costs to verify

Laid end to end, the pattern is more disciplined than “newer is more complex.” RoPE, grouped-query attention, SwiGLU, and RMSNorm are present, by name, in Meta’s own description of every generation covered here. What moved were: the RoPE base frequency and the staged context-extension recipe built around it; how uniformly GQA was applied across model sizes; the post-training optimisation algorithm, from PPO to DPO; the route to multimodality, from a frozen-backbone adapter to early fusion; and, only in the newest generation, whether the feed-forward block is dense or sparsely routed across experts. Raschka’s broader survey frames this as the field’s general shape, not something specific to Meta: “engineering refinements” accumulating onto a stable transformer skeleton across every major open-weight family it covers, Llama included ^[9].

The one place this generational record itself changed is how completely it was written down. Llama 2 and Llama 3 shipped as full papers with named contributor lists, hyperparameter tables, and dedicated sections walking through architecture, data, training infrastructure and post-training method ^{[1, 2]}. Llama 4’s technical description, as of this writing, is a blog post and a GitHub model card ^{[3, 4]} — both genuinely informative, and both short of the earlier generations’ level of documented detail. The iRoPE layer pattern, the attention-temperature adjustment, and the QK-normalization detail used in this article were confirmed through Hugging Face’s independent technical write-up of the release rather than through a citable passage in Meta’s own materials, because Meta’s own materials do not describe the mechanism at that level of detail in public form ^[13]. Nathan Lambert’s review of the release raises a related, separate concern about the evaluation claims built on top of that architecture: Meta’s headline chat-arena results were reported for “an experimental chat version” that was never itself released, which Lambert argues makes those specific comparisons unverifiable by outside parties even though the released Scout and Maverick checkpoints can be tested directly ^[12]. Both observations point the same direction. The architecture is real and the components named above are genuinely what the released weights implement — but the standard of documentation that let earlier generations be checked against a citable primary description is not uniform across the family, and a reader who wants Llama-4-level detail with Llama-3-level sourcing currently has to combine Meta’s release materials with independent reconstruction to get there.

Predictions, with the observations that would falsify them

These are forecasts, separated from the sourced architecture record above. Horizon: five years from this article’s publication date.

One. RoPE, in some variant, remains the positional scheme in Meta’s next dense or expert-routed flagship, because every generation covered here has extended it rather than replaced it, and the interleaved RoPE/no-positional-encoding split in Llama 4 is itself an extension rather than an abandonment. Disconfirmed if a subsequent Llama flagship ships with a positional scheme that is not describable as RoPE, an interleaving of RoPE with other schemes, or a direct successor sharing RoPE’s rotation-based relative-position mechanism.

Two. Grouped-query attention’s group count keeps moving toward more aggressive key-value sharing rather than back toward multi-head attention, following the same direction visible from Llama 2’s partial adoption to Llama 3’s universal one. Disconfirmed if a subsequent Llama generation increases the number of key-value heads relative to query heads in its largest model compared with Llama 3 and Llama 4.

Three. Meta’s documentation of future Llama releases will not return to the full-paper standard set by Llama 2 and Llama 3 for every release; blog-post-and-model-card documentation, supplemented by third-party technical reconstruction, becomes the durable pattern rather than a one-generation gap. Disconfirmed if Meta publishes a comprehensive architecture and training paper, with named authors and a full hyperparameter table, for its next flagship Llama release.

The skeleton that keeps standing

Four components, precisely defined, account for what has not moved across three Llama generations: a rotation that encodes position, a sharing scheme that thins how many key-value heads attention actually reads, a gate that decides how much of a signal a feed-forward block passes through, and a normalization step that rescales without re-centring. Everything documented as new — longer context, a bigger vocabulary, a different post-training optimiser, sparse expert routing, multimodal fusion at two different points in the pipeline — is a change to how those four components are configured, extended, or surrounded, not a departure from them. That is not a smaller story than “a new model came out.” It is the more checkable one, because it can be read directly off Meta’s own equations and hyperparameter tables rather than off a benchmark chart — right up to the generation where the equations stopped being published in full, and checking had to start pulling from more than one source.