Symbol r

The passage around this formula

…k} , LoRA represents the adapted weight as $W = W_0 + \Delta W = W_0 + BA, \qquad B \in \mathbb{R}^{d \times r},\ A \in \mathbb{R}^{r \times k},\ r \ll \min(d, k)$. Only B and A are trained; $W_0$ never moves during fine-tuning [ 5 ] . The effect on trainable parameter count for that one matrix is to fall from dk to r(d + k) , which is small whenever the chosen rank r is small relative to d and k — and because BA can be merged back into $W_0$ after training, LoRA adds no extra inference latency once deployed. Hu and colleagues report, for their comparison against full fine-tuning of GPT-3 175B, up to a…

Read this part in the article →