Equation 35 · Part 1 · What an AI Accelerator Actually Is: Silicon, Packaging, and the Memory It Can Reach
Symbol p
What this part means
the latency term grows linearly in.
Its job in the formula
p is a part of this expression. Its role is fixed by the surrounding article and by the operations shown in the formula.
Full expression→Symbol p→Article meaning
Where the article explains it
The bandwidth term approaches 2N/ and stops growing with p ; the latency term grows linearly in p .
The passage around this formula
with the per-step latency and the per-link bandwidth. The bandwidth term approaches 2N/ and stops growing with p ; the latency term grows linearly in p . Small, frequent collectives are therefore latency bound and scale badly, while large ones are bandwidth bound and scale well — which is precisely why gradient bucketing and overlapping communication with backward computation are standard practice rather than optimisations.
Learn the underlying idea
A variable is a named place for a value. Its letter is a local label: x can mean position in one formula and a data point in another.
Open the illustrated variables: a letter stands for a value guide →
See this notation across published equations →
Sources cited in the article section
- [11] NVIDIA Collective Communication Library (NCCL) Documentation ↗
- [9] NVIDIA Hopper Architecture In-Depth ↗
- [3] TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings ↗
These citations provide research context; check each source for the exact claim it supports.