$$N \times N$$

Equation 10 · AI Hardware & Semiconductors

Locality Is the Whole Game: The Memory Hierarchy and What a Kernel Does Not Read

This mathematical expression combines the displayed quantities; its precise role follows from the surrounding article text.

Matrix multiplication is the canonical demonstration because it has enormous latent reuse and a naive schedule that throws almost all of it away. Multiplying two N $\times$ N matrices performs about 2$N^3$ operations on 3$N^2$ words — an intensity proportional to N , which for large N is spectacularly compute-bound in principle. Executed as three nested loops in the obvious order, though, each element of one input is re-read from main memory on every pass, and the realised intensity collapses toward a small constant. The available reuse and the exploited reuse are different quantities, and only the second appears in the roofline.

Meanings in this article

• $N$: spectacularly compute-bound in principle.

Equation guide → · Article →

Equation 27 · AI Hardware & Semiconductors

Locality Is the Whole Game: The Memory Hierarchy and What a Kernel Does Not Read

This mathematical expression combines the displayed quantities; its precise role follows from the surrounding article text.

Standard attention forms the score matrix $\mathbf{S}$ = $\mathbf{Q}$$\mathbf{K}^{\top}$ , applies a row-wise softmax to obtain $\mathbf{P}$ , and multiplies by $\mathbf{V}$ . Both $\mathbf{S}$ and $\mathbf{P}$ are N $\times$ N , and standard implementations materialise them in main memory, which is quadratic in sequence length [ 1 ] . That materialisation is the problem. It is a large intermediate, it is written and read at least once, and the operations applied to it — masking, softmax, dropout — are exactly the memory-bound kind.

Equation guide → · Article →