Symbol r_phi

The passage around this formula

…under a penalty that keeps it near its starting point, $\max_\theta \; \mathbb{E}_{x,\, y\sim\pi_\theta}\big[r_\phi(x,y)\big] \;-\; \beta \, D_{\mathrm{KL}}\big(\pi_\theta \,\Vert\, \pi_{\mathrm{ref}}\big)$. Read the two equations together and the oversight limit is visible in the structure itself, not only in later critiques of it. The reward model $r_\phi$ is trained on whatever a human rater could see and judge in the time given; nothing forces it to track truth or usefulness beyond what raters could distinguish. The KL term in the second equation is a standing admission of that limit — it constrains how far the policy may travel from the…

Read this part in the article →