addition

The passage around this formula

FlashAttention converted that observation into wall-clock speed by combining tiling with recomputation. Blocks of the inputs are loaded into on-chip SRAM, the softmax reduction is performed incrementally across blocks using the online normaliser, and the output is written back without the score matrix ever reaching main memory; for the backward pass, the softmax normalisation factors are kept from the forward pass so that attention can be recomputed on-chip, which the authors state is faster than reading the stored intermediate back from HBM. Their analysis gives O($N^2$ $d^2$ $M^{-1}$) HBM accesses against $\Omega(Nd + N^2)$ for standard attention, with a matching lower bound showing no exact…

Read this part in the article →