$$M_{\text{resident}} \le B_{\text{mem}}$$

Equation 39 · Edge AI & Electronics

Shrink It, Train It Small, or Search for It: The Main Strategies for Small Models, Compared

This equation states a bound: one expression must stay on the indicated side of the other under the article’s assumptions.

Compression pays mostly $C_{\text{reuse}}$ , a documented small fraction of a from-scratch training cost [ 3 ] , but only if a suitable large model is available to reuse in the first place. Training small on purpose pays $C_{\text{curate}}$ and $C_{\text{train}}$ in full, with no discount, in exchange for a model that inherits nothing it was not deliberately given. Architecture search pays $C_{\text{search}}$ , which can be enormous when paid fresh per target [ 7 ] or amortized across many targets when paid once as a supernet [ 10 ] , on top of whatever strategy trains the architecture it discovers. Sparse mixture-of-experts is the odd one out in this accounting: its saving shows up only in \bar…

Equation guide → · Article →