What does this equation mean?

Multi-head attention gives every query head its own key and value projections. That is expensive at inference time for an autoregressive model, because every generated token requires reading the cached keys and values for every previous token, once per head, from memory — and memory bandwidth, not arithmetic, is usually the bottleneck in decoding. Multi-query attention collapses all query heads onto a single shared key-value pair, cutting that cache to a fraction of its multi-head size at some cost in quality. Ainslie and colleagues proposed the middle path that all three Llama generations actually use: grouped-query attention, in which H query heads are partitioned into G groups, and every query head within a group shares one key-value head pair [ 8 ] . The two earlier schemes are the boundary cases of the same construction: $G \in \{1, 2, \dots, H\}, \qquad G = H \Rightarrow \text{multi-head}, \quad G = 1 \Rightarrow \text{multi-query}, \quad 1 < G < H \Rightarrow \text{grouped-query}$. Ainslie and colleagues’ own contribution was as much practical as architectural: they showed an existing multi-head checkpoint can be converted into a grouped-query one by mean-pooling its key and value heads and continuing training with roughly five percent of the original pretraining compute, rather than requiring the smaller attention footprint to be trained in from scratch [ 8 ] .