← All parts of this equation

Equation 29 · Part 5 · Locality Is the Whole Game: The Memory Hierarchy and What a Kernel Does Not Read

addition

Ω(Nd+N2)\Omega(Nd + N^2)
addition

What this part means

Add the term after the plus sign to the term or group before it.

Its job in the formula

Add the term after the plus sign to the term or group before it.

The passage around this formula

FlashAttention converted that observation into wall-clock speed by combining tiling with recomputation. Blocks of the inputs are loaded into on-chip SRAM, the softmax reduction is performed incrementally across blocks using the online normaliser, and the output is written back without the score matrix ever reaching main memory; for the backward pass, the softmax normalisation factors are kept from the forward pass so that attention can be recomputed on-chip, which the authors state is faster than reading the stored intermediate back from HBM. Their analysis gives O(N2N^2 d2d^2 M−1M^{-1}) HBM accesses against Ω(Nd+N2)\Omega(Nd + N^2) for standard attention, with a matching lower bound showing no exact…

Read this part in the article →

Learn the underlying idea

Addition combines quantities; subtraction measures the signed difference between them. Parentheses show what is combined before the rest of the expression is evaluated.

Open the illustrated addition and subtraction in an equation guide →

Sources cited in the surrounding passage

These citations provide research context; check each source for the exact claim it supports.