superscript

The passage around this formula

Policy optimisation. The fine-tuned model is then optimised, typically with PPO [ 6 ] , to maximise the reward model’s score, with a penalty on divergence from the starting policy. InstructGPT’s published objective adds a per-token KL penalty from the supervised model and, in the PPO-ptx variant, a term mixing in pretraining gradients: $\mathbb{E}\big[r_\theta(x,y)\big] - \beta\,\mathbb{E}\big[D_{\mathrm{KL}}\!\left(\pi^{\mathrm{RL}} \,\|\, \pi^{\mathrm{SFT}}\right)\big] + \gamma\,\mathbb{E}\big[\log \pi^{\mathrm{RL}}(x)\big]$. where $\beta$ sets the strength of the KL penalty and $\gamma$ the pretraining mixture, with $\gamma$ set to zero for the plain PPO models [ 4 ] . Direct preference optimisation later showed that the explicit reward model can be dispensed with entirely — the optimal policy under this objective has a closed form, so the same problem can be solved…

Read this part in the article →