Symbol R_BoN
oN is part of the quantity the equation computes from the expression on the right.
Read this term in its guide →Published equation contexts
Why this formula appears here
Gao, Schulman and Hilton measured this cleanly by building a synthetic setting in which a large fixed “gold” reward model plays the role of the human, and a smaller “proxy” reward model is trained on its labels [ 8 ] . Optimising the proxy raises the gold score at first and then lowers it. They fit empirical functional forms in d , the square root of the KL divergence from the initial policy, finding for best-of- n sampling = d( - d) and for reinforcement learning
Read the representative guide
Symbol d
d is an argument of the function-like quantity on the left; its role is set by that function’s stated inputs.
Read this term in its guide →Symbol alpha_BoN
alphoN is one of the signed contributions combined to compute the quantity on the left.
Read this term in its guide →Symbol beta_BoN
betoN is one of the signed contributions combined to compute the quantity on the left.
Read this term in its guide →How to interpret it
Read it with the definitions, units, and assumptions supplied by the article.
Research cited beside this formula
Published contexts (1)
A symbol can carry a different meaning in another article. Each occurrence keeps its own guide and term definitions.
Equation 9 · Alignment & Safety
What RLHF Actually Optimises: Rated Agreeableness, and Where It Parts from Helpfulness
This equation states an equality: the expressions on both sides have the same value under the article’s assumptions.
Gao, Schulman and Hilton measured this cleanly by building a synthetic setting in which a large fixed “gold” reward model plays the role of the human, and a smaller “proxy” reward model is trained on its labels [ 8 ] . Optimising the proxy raises the gold score at first and then lowers it. They fit empirical functional forms in d , the square root of the KL divergence from the initial policy, finding for best-of- n sampling = d( - d) and for reinforcement learning
Equation guide → · Article →