Symbol Omega

The passage around this formula

…kept from the forward pass so that attention can be recomputed on-chip, which the authors state is faster than reading the stored intermediate back from HBM. Their analysis gives O($N^2$ $d^2$ $M^{-1}$) HBM accesses against $\Omega(Nd + N^2)$ for standard attention, with a matching lower bound showing no exact algorithm can do asymptotically better across all SRAM sizes [ 1 ] . The reported results were a 7.6 times speedup on the attention computation itself for GPT-2, 15 per cent faster end-to-end BERT-large training…

Read this part in the article →