$$P_{\mathrm{total}} \approx P_{\mathrm{shared}} + E \cdot P_{\mathrm{expert}}$$

Equation 6 · Open Models

A History of Llama and the Open-Weight AI Movement

This equation gives an approximation: it relates the quantities while allowing an approximation.

where $P_{\mathrm{shared}}$ is the always-active shared-expert capacity, $P_{\mathrm{expert}}$ is the parameter count of one routed expert, and k is the number of routed experts activated per token. Total capacity scales with the number of experts E available to the router, roughly $P_{\mathrm{total}}$ $\approx$ $P_{\mathrm{shared}}$ + E $\cdot$ $P_{\mathrm{expert}}$ , while inference compute scales with $P_{\mathrm{active}}$ , not $P_{\mathrm{total}}$ . This is the exact mechanism behind Meta’s headline figures: Maverick’s 400 billion total parameters and 17 billion active parameters are not two different measurements of the same quantity, they are $P_{\mathrm{total}}$ and $P_{\mathrm{active}}$ under a router…

Meanings in this article

• $P_{\mathrm{total}}$: the not.
• $P_{\mathrm{shared}}$: the always-active shared-expert capacity.
• $P_{\mathrm{expert}}$: the parameter count of one routed expert.

Equation guide → · Article →