≤

The passage around this formula

is the condition that actually governs whether an allocation succeeds, and it can fail even when $S_{\mathrm{request}}$ $\le$ $\sum_i$ $\mathrm{free}_i$ holds comfortably — the requested size fits inside total free memory but not inside any one free block. This is precisely the problem the smartphone inference system PowerInfer-2 was built to manage: it introduces a segmented neuron cache with a least-recently-used eviction policy specifically to keep the working set of “hot” and “cold” neuron clusters from fragmenting a mobile device’s limited DRAM, allowing models “whose sizes exceed the device’s memory capacity” to run without the allocator failures that naive loading produces, while cutting…

Read this part in the article →