superscript

The passage around this formula

When the adapter route is chosen, the engineering default within it is equally clear: adapt, do not retrain. Low-Rank Adaptation freezes the pretrained weights and injects a pair of small trainable matrices into selected layers, so that a weight update is expressed as a low-rank product rather than a dense matrix the size of the original layer, $\Delta W = BA, \qquad B \in \mathbb{R}^{d \times r},\ A \in \mathbb{R}^{r \times k},\ r \ll \min(d, k)$. with the forward pass computing h = $W_0$ x + $\Delta$ W x against the frozen base weight $W_0$ . The method’s authors report reducing the number of trainable parameters by roughly ten thousand times and GPU memory requirements by roughly three times relative to full fine-tuning of a 175-billion-parameter model, while matching or…

Read this part in the article →