$$P_{\mathrm{active}} \approx P_{\mathrm{shared}} + k \cdot P_{\mathrm{expert}}$$

Equation 1 · Open Models

A History of Llama and the Open-Weight AI Movement

This equation gives an approximation: it relates the quantities while allowing an approximation.

For a mixture-of-experts layer routing each token to one shared expert plus a fixed number of routed experts, the active parameter count per token is approximately $P_{\mathrm{active}} \approx P_{\mathrm{shared}} + k \cdot P_{\mathrm{expert}}$. where $P_{\mathrm{shared}}$ is the always-active shared-expert capacity, $P_{\mathrm{expert}}$ is the parameter count of one routed expert, and k is the number of routed experts activated per token. Total capacity scales with the number of experts E available to the router, roughly $P_{\mathrm{total}}$ $\approx$ $P_{\mathrm{shared}}$ + E $\cdot$ $P_{\mathrm{expert}}$ , while inference compute scales with $P_{\mathrm{active}}$ , not $P_{\mathrm{total}}$ . This is the exact mechanism behind Meta’s headline figures: Maverick’s 400…

Meanings in this article

• $P_{\mathrm{shared}}$: the always-active shared-expert capacity.
• $k$: the number of routed experts activated per token.
• $P_{\mathrm{expert}}$: the parameter count of one routed expert.

Equation guide → · Article →