$$\mathbb{E}\big[r_\theta(x,y)\big] - \beta\,\mathbb{E}\big[D_{\mathrm{KL}}\!\left(\pi^{\mathrm{RL}} \,\|\, \pi^{\mathrm{SFT}}\right)\big] + \gamma\,\mathbb{E}\big[\log \pi^{\mathrm{RL}}(x)\big]$$

Equation 3 · Alignment & Safety

What RLHF Actually Optimises: Rated Agreeableness, and Where It Parts from Helpfulness

This mathematical expression combines the displayed quantities; its precise role follows from the surrounding article text.

Policy optimisation. The fine-tuned model is then optimised, typically with PPO [ 6 ] , to maximise the reward model’s score, with a penalty on divergence from the starting policy. InstructGPT’s published objective adds a per-token KL penalty from the supervised model and, in the PPO-ptx variant, a term mixing in pretraining gradients: $\mathbb{E}\big[r_\theta(x,y)\big] - \beta\,\mathbb{E}\big[D_{\mathrm{KL}}\!\left(\pi^{\mathrm{RL}} \,\|\, \pi^{\mathrm{SFT}}\right)\big] + \gamma\,\mathbb{E}\big[\log \pi^{\mathrm{RL}}(x)\big]$. where $\beta$ sets the strength of the KL penalty and $\gamma$ the pretraining mixture, with $\gamma$ set to zero for the plain PPO models [ 4 ] . Direct preference optimisation later showed that the explicit reward model can be dispensed with entirely — the optimal policy under this objective has a closed form, so the same problem can be solved…

Meanings in this article

• $\mathbb{E}$: The expected value operator: the probability-weighted average of the quantity inside its brackets.
• $\gamma$: the pretraining mixture.

Equation guide → · Article →