What does this equation mean?

Gao, Schulman and Hilton measured this cleanly by building a synthetic setting in which a large fixed “gold” reward model plays the role of the human, and a smaller “proxy” reward model is trained on its labels [ 8 ] . Optimising the proxy raises the gold score at first and then lowers it. They fit empirical functional forms in d , the square root of the KL divergence from the initial policy, finding for best-of- n sampling $R_{\mathrm{BoN}}(d)$ = d($\alpha_{\mathrm{BoN}}$ - $\beta_{\mathrm{BoN}}$ d) and for reinforcement learning $R_{\mathrm{RL}}(d) = d\left(\alpha_{\mathrm{RL}} - \beta_{\mathrm{RL}} \log d\right)$. with coefficients that vary smoothly and roughly logarithmically with the number of proxy reward model parameters [ 8 ] . Two structural facts are embedded there. The curve has an interior maximum, so there exists an amount of optimisation beyond which more is worse. And the location of that maximum is a property of the reward model, not of the policy: larger policies benefited less from optimisation but overoptimised by a similar amount [ 8 ] . They also report a minimum data threshold, below roughly two thousand comparisons, under which reward models barely improved on chance [ 8 ] .