$$C_{\text{lifetime}} = \underbrace{C_{\text{reuse}}}_{\text{compress}} \;+\; \underbrace{C_{\text{curate}} + C_{\text{train}}}_{\text{train small}} \;+\; \underbrace{C_{\text{search}}}_{\text{NAS}} \;+\; Q \cdot \bar c_{\text{tok}}, \qquad \text{with} \quad M_{\text{resident}} \le B_{\text{mem}}$$

Equation 33 · Edge AI & Electronics

Shrink It, Train It Small, or Search for It: The Main Strategies for Small Models, Compared

This equation states a bound: one expression must stay on the indicated side of the other under the article’s assumptions.

Set side by side, the four strategies do not compete on a single scale, and a comparison that reduces them to one leaderboard number is not describing what any of them actually trades off. Each holds a different quantity fixed as “already spent” and treats a different quantity as the one still to be paid: $C_{\text{lifetime}} = \underbrace{C_{\text{reuse}}}_{\text{compress}} \;+\; \underbrace{C_{\text{curate}} + C_{\text{train}}}_{\text{train small}} \;+\; \underbrace{C_{\text{search}}}_{\text{NAS}} \;+\; Q \cdot \bar c_{\text{tok}}, \qquad \text{with} \quad M_{\text{resident}} \le B_{\text{mem}}$. Compression pays mostly $C_{\text{reuse}}$ , a documented small fraction of a from-scratch training cost [ 3 ] , but only if a suitable large model is available to reuse in the first place. Training small on purpose pays $C_{\text{curate}}$ and $C_{\text{train}}$ in full, with no discount, in exchange for a model that inherits nothing it was not deliberately given.…

Meanings in this article

• $C_{\text{reuse}}$: the compression pays mostly.
• $C_{\text{curate}}$: the training small on purpose pays.
• $C_{\text{search}}$: the architecture search pays.

Equation guide → · Article →