subscript

The passage around this formula

then optimizes the policy against that fitted reward under a penalty that keeps it near its starting point, $\max_\theta \; \mathbb{E}_{x,\, y\sim\pi_\theta}\big[r_\phi(x,y)\big] \;-\; \beta \, D_{\mathrm{KL}}\big(\pi_\theta \,\Vert\, \pi_{\mathrm{ref}}\big)$. Read the two equations together and the oversight limit is visible in the structure itself, not only in later critiques of it. The reward model $r_\phi$ is trained on whatever a human rater could see and judge in the time given; nothing forces it to track truth or usefulness beyond what raters could distinguish. The KL term in the second equation is a standing admission of that limit — it constrains how far the policy may travel from the region where $r_\phi$ was actually calibrated, because past that region the fitted proxy is not trusted to mean anything. Casper and…

Read this part in the article →