≤

The passage around this formula

SparseGPT poses pruning as a per-layer reconstruction problem. For a layer with weight matrix W and a small calibration set of activations X , it looks for a sparse replacement W' that keeps that layer’s output as close as possible to the original: $W' = \operatorname*{arg\,min}_{\|W'\|_0 \le k} \|WX - W'X\|_2^2$. Rather than solving this by retraining, Frantar and Alistarh adapt a closed-form update derived from the layer’s second-order (Hessian) information, in the spirit of the older Optimal Brain Surgeon method, so that whenever a weight is removed the remaining weights in that row are analytically nudged to compensate for its absence. The result, reported for the GPT-family models tested, is that “large-scale generative pretrained…

Read this part in the article →