What does this equation mean?

What the article says around this equation

Nothing in that loss distinguishes where a pair came from; a human-labeled winner and an AI-labeled winner are interchangeable once written down as (x, $y_w$, $y_l$) . That is the precise, narrow sense in which Constitutional AI “differs mechanically from plain RLHF”: it changes the labeling function for one half of one dataset, not the loss, not the optimizer, not the use of a KL penalty against the supervised policy. For the soft-labeled AI comparisons specifically, where the feedback model outputs a probability p of preferring response $y_A$ over $y_B$ rather than a hard choice, the corresponding cross-entropy term is

Read the equation in its article →