Equation 4 · Part 1 · The Main Technical Approaches to AI Alignment, Compared
Symbol r_phi
What this part means
hi is a part of this expression. Its role is fixed by the surrounding article and by the operations shown in the formula.
Its job in the formula
hi is a part of this expression. Its role is fixed by the surrounding article and by the operations shown in the formula.
Full expression→Symbol r_phi→Article meaning
The passage around this formula
Read the two equations together and the oversight limit is visible in the structure itself, not only in later critiques of it. The reward model is trained on whatever a human rater could see and judge in the time given; nothing forces it to track truth or usefulness beyond what raters could distinguish. The KL term in the second equation is a standing admission of that limit — it constrains how far the policy may travel from the…
Learn the underlying idea
A subscript is a label attached below a symbol. It often selects a time step, component, category, or member of a sequence.
Open the illustrated subscripts: which member of a family? guide →
See this notation across published equations →
Sources cited in the surrounding passage
These citations provide research context; check each source for the exact claim it supports.