#grouped-query attention

Tagged grouped-query attention · show all articles

Three labelled checkpoint trays for Llama 2, Llama 3 and Llama 4 sit in a row on a bright inspection bench, the middle tray caught mid-slide into a comparison rig with its ribbon cable not yet seated

How Llama's Architecture Actually Works, Generation by Generation

Meta's own papers specify exactly how Llama encodes position, shares attention across heads, gates its activations and normalizes its signals — and exactly what changed, and did not change, from Llama 2 through Llama 3 to Llama 4.

By Brecht Corbeel · August 12, 2026 · 18 min read

Browse by Category

Surprise me →

All AI Hardware & Semiconductors 100 AI Agents & Systems 33 Foundation Models 27 Model Evaluation 15 AI Industry 9 Edge AI & Electronics 7 AI Safety 7 AI Research 7 Open Models 6 AI Infrastructure 5 Industry Analysis 2 Datacenters & Infrastructure 2 Writing & Information 1 Structural Biology 1 Small & Edge Models 1 Semiconductor Markets 1 Security & Technology 1 Security 1 Scientific Revolution 1 Robotics & Embodied AI 1 Quantum Information 1 Planetary Science 1 Physics 1 Particle Physics 1 Origins & Astrobiology 1 Open-Weight Models 1 Neuroscience 1 Metrology 1 Mathematics & Computation 1 Materials Science 1 Labour & Technology 1 Interpretability 1 Institutions & State Capacity 1 Institutions & Economy 1 Inference Economics 1 Industrial Revolutions 1 Humanity & Technology 1 History of Technology 1 History of Computing 1 Governance & Policy 1 Evolution & Ecology 1 Ethics of Technology 1 Energy & Civilization 1 Earth & Climate 1 Demography & Urbanization 1 Data & Training 1 Cumulative Culture 1 Cosmology 1 Condensed Matter 1 Chemistry & Catalysis 1 Alignment & Safety 1 Agriculture & Cities 1 Agent Evaluation 1 AI Security 1 AI & The Web 1