Equation 7 · Part 2 · Claude, From First Principles: Training, Constitutional Methods, and What Actually Shapes a Response
Symbol x
What this part means
x is a part of this expression. Its role is fixed by the surrounding article and by the operations shown in the formula.
Its job in the formula
x is a part of this expression. Its role is fixed by the surrounding article and by the operations shown in the formula.
Full expression→Symbol x→Article meaning
The passage around this formula
The reward term r(x,y) pulls the policy toward whatever the fitted preference model scores highly; the Kullback–Leibler penalty, weighted by , pulls it back toward the reference distribution. That second term is not a minor regulariser. It is the load-bearing assumption behind the entire method: remove it, and a policy optimised hard enough against an imperfect reward model will find outputs the reward model over-scores without those outputs actually being better, a failure usually called reward hacking. Casper and thirty-one co-authors, surveying RLHF across the field rather than defending any one lab’s implementation, catalogue this and related problems as fundamental rather than…
Learn the underlying idea
A variable is a named place for a value. Its letter is a local label: x can mean position in one formula and a data point in another.
Open the illustrated variables: a letter stands for a value guide →
See this notation across published equations →
Sources cited in the surrounding passage
These citations provide research context; check each source for the exact claim it supports.