What does this equation mean?

Constitutional AI is Anthropic’s best-documented departure from plain RLHF, and it is worth being precise about exactly what it changes, because the method is easy either to overstate or to wave at vaguely. Bai and fifty co-authors describe it as a two-phase procedure layered onto the pipeline above [ 1 ] . In the first phase, the model is prompted to critique its own draft response against one principle drawn from a written list, then to revise the response in light of that critique; those revised responses become supervised fine-tuning data, so the model that comes out the other side has been trained partly on its own corrected output rather than solely on external demonstrations. In the second phase, the same reinforcement-learning-against-a-reward-model structure from the previous section is reused, but for the harmlessness comparisons specifically, the preference labels ordinarily supplied by human raters are instead generated by a separate instance of the model, prompted with a constitutional principle and asked which of two candidate responses better satisfies it — a substitution the paper’s authors call reinforcement learning from AI feedback, RLAIF, to distinguish it from RLHF [ 1 ] . Put in terms of the objective in the previous section, Constitutional AI’s second phase is the same J($\theta$) , with r(x,y) for the harmlessness component sourced from a principle-conditioned AI judgment rather than an additional round of human labelling; the original paper’s helpfulness comparisons continued to draw on human preference data [ 1 ] .