← All parts of this equation

Equation 11 · Part 13 · How Constitutional AI Actually Constrains a Model's Behavior

superscript

LPMAI(θ)=− E(x, yA, yB, p) ∼ DAI[p log⁡σ(rθ(x,yA)−rθ(x,yB))+(1−p) log⁡σ(rθ(x,yB)−rθ(x,yA))],\mathcal{L}^{AI}_{PM}(\theta) = -\,\mathbb{E}_{(x,\,y_A,\,y_B,\,p)\,\sim\, D_{AI}}\Big[p\,\log \sigma\big(r_\theta(x,y_A)-r_\theta(x,y_B)\big) + (1-p)\,\log \sigma\big(r_\theta(x,y_B)-r_\theta(x,y_A)\big)\Big],
superscript

What this part means

A raised number can be a power. When it is a label or bound, it selects a case or the upper limit of a sum; the formula’s structure distinguishes these uses.

Its job in the formula

A raised mark can be a power or an index. Its position and the surrounding notation determine which.

The passage around this formula

Nothing in that loss distinguishes where a pair came from; a human-labeled winner and an AI-labeled winner are interchangeable once written down as (x, ywy_w, yly_l) . That is the precise, narrow sense in which Constitutional AI “differs mechanically from plain RLHF”: it changes the labeling function for one half of one dataset, not the loss, not the optimizer, not the use of a KL penalty against the supervised policy. For the soft-labeled AI comparisons specifically, where the feedback model outputs a probability p of preferring response yAy_A over yBy_B rather than a hard choice, the corresponding cross-entropy term is LPMAI(θ)=− E(x, yA, yB, p) ∼ DAI[p log⁡σ(rθ(x,yA)−rθ(x,yB))+(1−p) log⁡σ(rθ(x,yB)−rθ(x,yA))]\mathcal{L}^{AI}_{PM}(\theta) = -\,\mathbb{E}_{(x,\,y_A,\,y_B,\,p)\,\sim\, D_{AI}}\Big[p\,\log \sigma\big(r_\theta(x,y_A)-r_\theta(x,y_B)\big) + (1-p)\,\log \sigma\big(r_\theta(x,y_B)-r_\theta(x,y_A)\big)\Big]. which exposes the other genuine mechanical difference:…

Read this part in the article →

Learn the underlying idea

An exponent tells how a base is used in multiplication. In x³, x is the base and 3 is the exponent: x³ = x × x × x.

Open the illustrated exponents: repeated multiplication and powers guide →

Sources cited in the article section

These citations provide research context; check each source for the exact claim it supports.