What does this equation mean?

then optimizes the policy against that fitted reward under a penalty that keeps it near its starting point, $\max_\theta \; \mathbb{E}_{x,\, y\sim\pi_\theta}\big[r_\phi(x,y)\big] \;-\; \beta \, D_{\mathrm{KL}}\big(\pi_\theta \,\Vert\, \pi_{\mathrm{ref}}\big)$. Read the two equations together and the oversight limit is visible in the structure itself, not only in later critiques of it. The reward model $r_\phi$ is trained on whatever a human rater could see and judge in the time given; nothing forces it to track truth or usefulness beyond what raters could distinguish. The KL term in the second equation is a standing admission of that limit — it constrains how far the policy may travel from the region where $r_\phi$ was actually calibrated, because past that region the fitted proxy is not trusted to mean anything. Casper and colleagues’ survey states the underlying identification directly: “reward models are trained to reflect human approval instead of human benefit,” and separately, that “a single reward function cannot represent a diverse society of humans,” concluding that condensing feedback from many people into one scalar is “a fundamentally misspecified problem” rather than a bug fixable within the RLHF framework [ 5 ] . Their paper is explicit that this is a distinct claim from RLHF’s more tractable engineering problems: a limitation counts as fundamental, in their terms, only when overcoming it “would require a method that is no longer a form of RLHF” [ 5 ] . RLHF’s scalable-oversight ceiling, in other words, is set by rater competence and by what a single scalar can represent — which is precisely the ceiling the other five techniques each try to raise or route around in a different way.