Symbol y

The passage around this formula

The reward term r(x,y) pulls the policy toward whatever the fitted preference model scores highly; the Kullback–Leibler penalty, weighted by $\beta$ , pulls it back toward the reference distribution. That second term is not a minor regulariser. It is the load-bearing assumption behind the entire method: remove it, and a policy optimised hard enough against an imperfect reward model will find outputs the reward model over-scores without those outputs actually being better, a failure usually called reward hacking. Casper and thirty-one co-authors, surveying RLHF across the field rather than defending any one lab’s implementation, catalogue this and related problems as fundamental rather than…

Read this part in the article →