superscript

The passage around this formula

where $\theta_i^{*}$ is the old optimum for parameter i and $\lambda$ sets how strongly the old task is protected. This equation exposes the actual trade rather than resolving it: raising $\lambda$ protects old knowledge at the direct expense of how much the new update is allowed to change the model, and there is no value of $\lambda$ that removes the trade — only one that relocates it. It also exposes a cost specific to on-device deployment: computing and storing $F_i$ for every parameter, and doing so repeatedly as the device keeps learning, is itself memory and compute that a phone-class budget has to find room for, on top of whatever the update itself costs.

Read this part in the article →