≈

The passage around this formula

For inference cost specifically, what matters is that per-token compute tracks activated parameters, not total parameters. Approximating compute per token as proportional to the parameter count actually touched [ 13 , 11 ] , the ratio of dense to MoE compute at equal activated size is approximately $\frac{C_{\mathrm{dense}}}{C_{\mathrm{MoE}}} \approx \frac{N_{\mathrm{dense}}}{N_{\mathrm{active}}}$. for a dense model with $N_{\mathrm{dense}}$ parameters compared against an MoE model activating $N_{\mathrm{active}}$ of its $N_{\mathrm{total}}$ parameters per token. What this ratio hides is exactly what a compute-only comparison always hides: memory. Serving an MoE model requires holding all $N_{\mathrm{total}}$ parameters resident — on one device or, more often, sharded across…

Read this part in the article →