subscript

The passage around this formula

for a dense model with $N_{\mathrm{dense}}$ parameters compared against an MoE model activating $N_{\mathrm{active}}$ of its $N_{\mathrm{total}}$ parameters per token. What this ratio hides is exactly what a compute-only comparison always hides: memory. Serving an MoE model requires holding all $N_{\mathrm{total}}$ parameters resident — on one device or, more often, sharded across several with communication between them whenever a batch’s tokens route to different experts — even though only $N_{\mathrm{active}}$ of them do arithmetic on any given token. A dense model of the same activated size carries no such requirement. The compute saving is real and is the reason MoE models can be commercially…

Read this part in the article →