multiplication

The passage around this formula

Standard attention forms the score matrix $\mathbf{S}$ = $\mathbf{Q}$$\mathbf{K}^{\top}$ , applies a row-wise softmax to obtain $\mathbf{P}$ , and multiplies by $\mathbf{V}$ . Both $\mathbf{S}$ and $\mathbf{P}$ are N $\times$ N , and standard implementations materialise them in main memory, which is quadratic in sequence length [ 1 ] . That materialisation is the problem. It is a large intermediate, it is written and read at least once, and the operations applied to it — masking, softmax, dropout — are exactly the memory-bound kind.

Read this part in the article →