multiplication

The passage around this formula

Low-Rank Adaptation, LoRA, takes a different approach: it freezes the pretrained weight matrix entirely and represents the update as the product of two much smaller matrices. For a frozen weight matrix $W_0$ $\in$ $\mathbb{R}^{d \times k}$ , LoRA represents the adapted weight as $W = W_0 + \Delta W = W_0 + BA, \qquad B \in \mathbb{R}^{d \times r},\ A \in \mathbb{R}^{r \times k},\ r \ll \min(d, k)$. Only B and A are trained; $W_0$ never moves during fine-tuning [ 5 ] . The effect on trainable parameter count for that one matrix is to fall from dk to r(d + k) , which is small whenever the chosen rank r is small relative to d and k — and because BA can be merged back into $W_0$ after training, LoRA adds no extra inference latency once deployed. Hu and colleagues report, for their comparison against full…

Read this part in the article →