subtraction

The passage around this formula

Gao, Schulman and Hilton measured this cleanly by building a synthetic setting in which a large fixed “gold” reward model plays the role of the human, and a smaller “proxy” reward model is trained on its labels [ 8 ] . Optimising the proxy raises the gold score at first and then lowers it. They fit empirical functional forms in d , the square root of the KL divergence from the initial policy, finding for best-of- n sampling $R_{\mathrm{BoN}}(d)$ = d($\alpha_{\mathrm{BoN}}$ - $\beta_{\mathrm{BoN}}$ d) and for reinforcement learning

Read this part in the article →