Symbol O

The passage around this formula

…the softmax normalisation factors are kept from the forward pass so that attention can be recomputed on-chip, which the authors state is faster than reading the stored intermediate back from HBM. Their analysis gives O($N^2$ $d^2$ $M^{-1}$) HBM accesses against $\Omega(Nd + N^2)$ for standard attention, with a matching lower bound showing no exact algorithm can do asymptotically better across all SRAM sizes [ 1 ] . The reported results were a 7.6 times speedup on the attention computation itself for GPT-2, 15 per cent…

Read this part in the article →