superscript

The passage around this formula

One standard mitigation is regularization: penalize the optimizer for moving parameters that mattered to earlier tasks. The best-known form estimates a per-parameter importance weight — commonly the diagonal of the Fisher information, $F_i$ — from the old task, and adds it to the new loss: $\mathcal{L}(\theta) = \mathcal{L}_{\mathrm{new}}(\theta) + \lambda \sum_{i} F_i\left(\theta_i - \theta_i^{*}\right)^2$. where $\theta_i^{*}$ is the old optimum for parameter i and $\lambda$ sets how strongly the old task is protected. This equation exposes the actual trade rather than resolving it: raising $\lambda$ protects old knowledge at the direct expense of how much the new update is allowed to change the model, and there is no value of $\lambda$ that removes the trade — only one that relocates it. It also…

Read this part in the article →