addition

The passage around this formula

For a mixture-of-experts layer routing each token to one shared expert plus a fixed number of routed experts, the active parameter count per token is approximately $P_{\mathrm{active}} \approx P_{\mathrm{shared}} + k \cdot P_{\mathrm{expert}}$. where $P_{\mathrm{shared}}$ is the always-active shared-expert capacity, $P_{\mathrm{expert}}$ is the parameter count of one routed expert, and k is the number of routed experts activated per token. Total capacity scales with the number of experts E available to the router, roughly $P_{\mathrm{total}}$ $\approx$ $P_{\mathrm{shared}}$ + E $\cdot$ $P_{\mathrm{expert}}$ , while inference compute scales with $P_{\mathrm{active}}$ , not $P_{\mathrm{total}}$ . This is the exact mechanism behind Meta’s headline figures: Maverick’s 400…

Read this part in the article →