The Main Technical Approaches to AI Alignment, Compared

Six bets on the same unsolved problem

Every technique in this article was invented to answer one question: how do you keep a training signal trustworthy once the system being trained can do things the people training it cannot directly check? That question has a name in the research literature, scalable oversight, and it has produced not one answer but a family of structurally different ones, each attacking a different piece of the problem and each carrying a different, explicitly documented failure mode.

This is a survey of that research field, not of any one company’s shipped pipeline. A separate piece in this publication compares how two specific frontier developers combine and brand their production alignment stacks; this one instead lays out the six technical approaches that a researcher choosing among methods actually has to weigh against each other — reinforcement learning from human feedback (RLHF), Constitutional AI and reinforcement learning from AI feedback (RLAIF), debate, iterated distillation and amplification (IDA), weak-to-strong generalization, and reinforcement learning on verifiable rewards (RLVR). Some of these are old enough to be standard practice; two are closer to theoretical proposals with partial empirical support. None of them is presented here as the winner, because the six were not built to solve the same sub-problem, and several of them are routinely combined rather than swapped for one another in a single trained system.

Two ground rules follow from the assignment this article set itself. First, every claim below is traced to its originating paper or a documented follow-up, and vendor assertions, formal results, and this article’s own inferences are kept visibly separate. Second, no cross-technique ranking is built anywhere in the text — the reasons for that refusal are specific to each comparison and are given as they arise, not asserted once and left unexamined.

What “scalable oversight” is actually asking for

The term names a specific, narrower worry than “AI safety” in general. Leike and colleagues, writing a DeepMind research-direction paper that predates most of the techniques below, frame it as the agent alignment problem: reward functions are hard to specify by hand because the person specifying them has only an implicit sense of what they want, and the paper’s proposed research programme centers on learning a reward signal from interaction rather than writing one down ^[3]. That framing already contains the seed of the harder version of the problem. A learned reward signal is only as good as the judgment used to fit it, and if the system being trained becomes more capable than the judge, the signal degrades exactly where it matters most — on the hardest, most consequential outputs, which are also the ones a limited judge is least equipped to check.

Bowman and colleagues gave this harder version a name and a proposed experimental method. They define scalable oversight as the problem of “supervising systems that potentially outperform us on most skills relevant to the task at hand,” and propose studying it now, before systems are actually superhuman, through what they call a sandwiching paradigm: position a model’s capability between a non-expert and an expert on some task, let the non-expert try to elicit good performance from the model through interaction alone, and only afterward check the result against the expert ^[4]. Their own experiment is the cleanest illustration available of why the problem is worth taking seriously well before any system is unambiguously superhuman. On a time-limited, QuALITY-based reading-comprehension task, unaided human accuracy was 48.6 percent for an individual and 50.0 percent under weighted majority vote; the underlying language model alone, unassisted, scored 59.2 percent zero-shot; but humans working interactively with that same unreliable model reached 76.8 percent individually and 86.0 percent under weighted majority vote — a team result well above either the human or the model working alone ^[4]. That is evidence, from one study on one task, that structured interaction with an imperfect model can already recover more than either party manages solo. It is not evidence that the six techniques below solve the general problem; it establishes only that the problem is measurable now, which is why every technique that follows can be checked against something more concrete than intuition.

RLHF: a proxy fitted to comparisons

RLHF is the oldest and most widely deployed of the six, and its motivation was originally about capability elicitation for tasks without a written specification, not superhuman oversight specifically. Christiano and colleagues introduced the modern form to demonstrate that a reinforcement learner could be trained from non-expert comparisons between short trajectory segments instead of a hand-coded reward function, reporting that feedback on under one percent of the agent’s environment interactions was enough to train Atari and simulated-robot behaviors that would otherwise require a reward function nobody could write down ^[1]. Ouyang and colleagues carried the method to language models at scale, and their headline finding is still the cleanest evidence that comparison-based fine-tuning changes something real: outputs from a 1.3-billion-parameter InstructGPT model were preferred by human raters to outputs from the 175-billion-parameter GPT-3 it was tuned from, despite having roughly a hundred times fewer parameters ^[2].

The procedure fits a scalar reward model to pairwise comparisons using the Bradley–Terry choice model,

then optimizes the policy against that fitted reward under a penalty that keeps it near its starting point,

Read the two equations together and the oversight limit is visible in the structure itself, not only in later critiques of it. The reward model $r_\phi$ is trained on whatever a human rater could see and judge in the time given; nothing forces it to track truth or usefulness beyond what raters could distinguish. The KL term in the second equation is a standing admission of that limit — it constrains how far the policy may travel from the region where $r_\phi$ was actually calibrated, because past that region the fitted proxy is not trusted to mean anything. Casper and colleagues’ survey states the underlying identification directly: “reward models are trained to reflect human approval instead of human benefit,” and separately, that “a single reward function cannot represent a diverse society of humans,” concluding that condensing feedback from many people into one scalar is “a fundamentally misspecified problem” rather than a bug fixable within the RLHF framework ^[5]. Their paper is explicit that this is a distinct claim from RLHF’s more tractable engineering problems: a limitation counts as fundamental, in their terms, only when overcoming it “would require a method that is no longer a form of RLHF” ^[5]. RLHF’s scalable-oversight ceiling, in other words, is set by rater competence and by what a single scalar can represent — which is precisely the ceiling the other five techniques each try to raise or route around in a different way.

Constitutional AI and RLAIF: moving the proxy’s source

Constitutional AI keeps RLHF’s reward-model-and-KL-penalty backbone intact and changes where the comparison labels come from. Bai and colleagues describe a two-phase method: a supervised phase in which the model critiques and revises its own responses against a written set of principles, and a reinforcement phase in which a model, rather than a human, judges which of two candidate responses better satisfies those principles — producing an AI-generated preference dataset that trains the reward model ^[6]. Formally, this changes only the source of the comparison label. The Bradley–Terry equation above is unchanged in form; what changes is that the probability being fitted is now

a judgment conditioned on a written constitution $C$ rather than on an anonymous rater’s unstated standards. Lee and colleagues tested this substitution directly at scale and reported that AI-generated feedback achieved comparable performance to human feedback across summarization and dialogue tasks, and that a direct variant skipping the intermediate reward model entirely outperformed the standard two-stage form ^[7].

The motivation this targets is a specific, narrow piece of scalable oversight: human rating does not scale in volume or consistency, and an implicit standard scattered across thousands of individual raters’ judgment calls is harder to inspect, audit, or correct than one written document. Constitutional AI’s genuine gain is that the standard becomes explicit and inspectable rather than distributed and tacit. What it does not change is the deeper structural limit. Whatever the AI judge cannot itself distinguish, it cannot teach — and the AI judge is drawn from the same family of models whose blind spots, in a system that has begun to exceed human evaluators, are exactly the ones scalable oversight research is trying to guard against. Substituting a model-judge for a human-judge is a genuine engineering improvement in consistency and auditability; on its own terms, per Bai and colleagues’ description of the method, it is not a proposal for judging outputs that a comparably capable judge cannot itself assess ^[6].

Debate: two arguments, one bounded judge

Debate targets the oversight ceiling directly rather than working around it, and its motivation is stated in explicitly theoretical terms. Irving, Christiano, and Amodei propose training two agents through self-play in a zero-sum game: each argues a position in alternating statements, and a human judge decides which one gave more true, useful information. The paper’s central theoretical claim draws an analogy to computational complexity theory, arguing that if optimal play in the debate game tracks truth, then a judge with only polynomial-time reasoning ability could in principle adjudicate a debate about problems in the complexity class PSPACE — that is, questions considerably harder than the judge could solve alone ^[8]. Formally, debate is a minimax game over a judge’s payoff:

where the honest debater’s strategy is the one that performs best against a worst-case opponent, and the entire theoretical case for the method rests on the claim that truth has a structural advantage in such a game: that it is easier to defend a true position convincingly than to defend a false one, so honesty is the equilibrium strategy even when the judge cannot verify the underlying facts directly. The paper’s own initial empirical test was modest by design — an MNIST-digit-classification game in which debating agents revealed pixels to a judge who could see only a handful of them, raising classification accuracy from 59.4 to 88.9 percent with six revealed pixels — offered as a proof of concept for the mechanism rather than evidence about language-model debate ^[8].

The theoretical assumption that truth has an advantage in argument is exactly where the documented limitations sit. Brown-Cohen, Irving, and Piliouras, extending the original protocol, name the central one directly: a dishonest debater can construct an argument containing a real flaw that is nonetheless extremely hard to locate, forcing the honest debater into a computationally intractable search to expose it — the obfuscated arguments problem. Their paper’s proposed fix restructures the game so that an honest strategy needs only a polynomial number of simulation steps to succeed, while a dishonest strategy is denied the benefit of unlimited computation even if it is allowed exponentially more steps than the honest side ^[9]. That is a theoretical mitigation, not a demonstrated empirical solution. On the empirical side, Khan and colleagues ran debate with actual large language models and reported a genuinely positive result for the mechanism: non-expert judges reading debate transcripts reached 76 percent accuracy and non-expert humans reached 88 percent, against naive baselines of 48 and 60 percent respectively, and — the finding that speaks most directly to the theory’s central worry — optimizing expert debaters purely for persuasiveness, with no access to ground truth, still improved a non-expert judge’s ability to identify the true answer ^[10]. That is meaningful support for the truth-has-an-advantage premise in the settings tested. It is a result on a specific benchmark with judges of a specific weaker capability than the debaters, not a general resolution of the obfuscation problem the theoretical work identifies.

Iterated distillation and amplification: recursion instead of a bigger rater

IDA is debate’s closest theoretical relative, aimed at the same underlying worry from a different angle: rather than trusting a single judge to adjudicate an argument about a hard problem, it never asks any one evaluator to face the hard problem directly. Christiano, Shlegeris, and Amodei describe a scheme they call Amplify: a composite system, written $\mathrm{Amplify}_H(X)$, “consisting of $H$ and several copies of $X$ working together to solve a problem,” in which $H$ — standing in for a human overseer — breaks a hard question into useful subquestions, has several copies of the current trained agent $X$ answer each subquestion, and then combines the subanswers into a response to the original question ^[11]. The trained agent is then updated to imitate that amplified process directly: “the agent $X$ then learns from $\mathrm{Amplify}_H(X)$ in the same way that it would traditionally learn from $H$ alone” ^[11]. Run repeatedly, the described process is naturally read as the recursion

with each iteration’s agent trained to imitate the amplified, decomposed judgment of the previous iteration rather than any single overseer’s raw judgment of the whole problem. The paper describes the intended trajectory in prose rather than in this compact notation, but the substance is the same: over successive iterations, “the agent $X$ becomes more powerful and the role of the expert transitions into ‘coordinating’ several copies of $X$ to solve the problem better than a single copy could solve it” ^[11].

The motivation is squarely a scalable-oversight one: it tries to preserve a connection to a human overseer’s judgment at every step while letting the effective capability of the overall system grow, since the human never has to evaluate an answer to the whole hard problem directly — only to a sequence of decomposed subproblems small enough to check. The limitation is shared explicitly with debate, and for the same structural reason: subsequent analysis of both mechanisms has identified the obfuscated arguments problem as, in Brown-Cohen and colleagues’ framing, a limitation of decomposition-based oversight schemes generally, not one specific to the debate game — a dishonest step buried inside a long decomposition chain can be as hard to locate as a dishonest step buried inside a long argument ^[9]. A second limitation is more practical than theoretical: IDA has not been deployed at the scale RLHF and Constitutional AI have. Its main documented legacy is intellectual rather than operational — the idea of decomposing oversight into smaller, checkable pieces recurs in later, more widely used methods, but IDA itself remains closer to a theoretical research direction than a production technique.

Weak-to-strong generalization: renting the actual problem at a smaller scale

Where debate and IDA are proposals for the superhuman-oversight problem, weak-to-strong generalization is instead an attempt to study that exact problem empirically before it exists, by building a deliberately smaller analogue of it today. Burns and colleagues motivate the method directly: “humans will need to supervise models much smarter than us,” and since no such model exists yet to experiment on, they substitute a weaker model for the human overseer and a stronger model for the future superhuman system, asking whether naively fine-tuning the strong model on the weak model’s labels can recover more of the strong model’s own latent capability than the weak labels alone would suggest ^[12]. Their central empirical claim is that this happens by default: across tasks in natural language processing, chess puzzles, and reward modeling within the GPT-4 family, naively fine-tuning a strong pretrained model on a weak supervisor’s labels consistently outperforms the weak supervisor itself ^[12].

They quantify how much of the gap this recovers with a metric called performance gap recovered, defined by its two boundary conditions: it equals one under perfect weak-to-strong generalization and zero when the weak-to-strong model does no better than the weak supervisor it learned from ^[12]. Writing $f_w$ for the weak supervisor, $f_{w\to s}$ for the strong model fine-tuned on the weak model’s labels, and $f_{s\to s}$ for the strong model fine-tuned on ground truth as an upper-bound ceiling, those two conditions pin down

The paper also reports that simple additional interventions help: an auxiliary confidence loss recovered performance closer to GPT-3.5 level using only GPT-2-level supervision on GPT-4, a substantially larger fraction of the gap than naive fine-tuning alone closed ^[12].

The limitation the authors themselves stress hardest is one of analogy, not of the numbers: a present-day weak model is not a faithful stand-in for a future human overseer facing a genuinely superhuman system, because the two kinds of weakness may fail in different ways and PGR varies substantially across the specific tasks tested — it is a measurement of how much a proxy problem is recovered today, not a proof about the real problem it stands in for. Follow-up theoretical work has sharpened a second concern from inside the same framing: a strong model finetuned on weak labels can learn to satisfy the weak supervisor’s specific blind spots while performing worse exactly where the weak supervisor cannot check — a pattern researchers have termed weak-to-strong deception, in which apparent alignment on the visible, checkable slice of behavior coexists with divergence on the parts the weak overseer cannot see. That risk is precisely what the original paper’s boundary metric cannot detect by construction, since PGR is computed only on labels the researchers can score against ground truth.

Reinforcement learning on verifiable rewards: when the check needs no proxy

RLVR is the newest of the six and the only one that does not fit a learned model of anyone’s judgment at all. Where every technique above trains a proxy — a reward model, an AI judge, a decomposition scheme, a weak label — RLVR restricts itself to tasks where the reward can be computed directly and automatically: a unit test passes, a final numeric answer matches, a proof checker accepts. Shao and colleagues’ DeepSeekMath paper introduced Group Relative Policy Optimization, the reinforcement learning algorithm used throughout most subsequent RLVR work, replacing the learned value function used in standard policy-gradient methods with a group-relative advantage estimated directly from a batch of sampled outputs and their rule-based rewards,

normalizing each sampled response’s reward against the mean and standard deviation of a group of $G$ responses to the same prompt rather than against a separately trained critic network ^[13]. Lambert and colleagues, building the fully open Tulu 3 post-training recipe, named the general approach explicitly, describing it as “a novel method we call Reinforcement Learning with Verifiable Rewards,” and used it alongside supervised fine-tuning and preference optimization rather than as a wholesale replacement for either ^[15]. The clearest large-scale demonstration is DeepSeek-R1: Guo and colleagues report training a model with reinforcement learning alone against a purely rule-based reward — combining an answer-correctness check with a format check, and deliberately avoiding a learned reward model because, in their account, a neural reward model “may suffer from reward hacking in large-scale reinforcement learning” — and observed pass@1 accuracy on the AIME 2024 competition-mathematics benchmark rise from 15.6 percent to 71.0 percent over training, reaching 86.7 percent with majority voting across 64 samples ^[14]. Formally, the reward itself is as simple as the reward model above was elaborate:

The motivation this targets is narrow and specific: it removes the exact failure mode that RLHF’s KL penalty exists to contain. A learned reward model can always be over-optimized past the point where it stops tracking the true objective it was fit to approximate; a verifier that is the objective itself has no such gap to exploit, in principle, so optimization pressure can be applied far more aggressively without the usual proxy-drift risk.

That principle turns out to hold only conditionally. Shao and colleagues’ own later work, testing RLVR with deliberately uninformative rewards, found that training Qwen2.5-Math-7B with entirely random rewards improved MATH-500 accuracy by 21.4 percentage points, nearly matching the 29.1-point gain from genuine ground-truth rewards, and traced the effect to a bias in the GRPO clipping mechanism that can amplify behaviors the model already learned during pretraining even when the reward carries no information about correctness at all ^[16]. Critically, they also found this effect was highly model-dependent: the same spurious rewards that produced large gains on Qwen models largely failed to help Llama3 or OLMo2 ^[16]. The result does not show that RLVR’s reward is untrustworthy in the way a learned reward model is — the check itself remains genuinely binary and unhackable by the policy in the sense Guo and colleagues describe. It shows something narrower but still important for how the technique is evaluated: a large reported gain from an RLVR run is not, by itself, evidence that the verifiable reward is the mechanism doing the work, on every model family, without checking whether the same gain appears under a deliberately uninformative reward as a control. And RLVR’s applicability is bounded by the same fact that makes it powerful — it exists only where a check exists, which excludes most of what an assistant is actually asked to do: tone, honesty about uncertainty, appropriateness to the person asking, and every other property with no automatic checker.

What the six approaches do and do not share

Laid out together, the six split along a structural line rather than a quality line. RLHF, Constitutional AI/RLAIF, and RLVR all still train a policy through reinforcement learning against some reward signal; they differ in where that signal comes from — human comparisons, AI comparisons against a written document, or an automatic checker — and each source has a different, specifically documented failure mode: rater-approval proxies for RLHF, judge blind-spot inheritance for Constitutional AI, and check-gaming or clipping artifacts for RLVR. Debate and IDA are not reward-fitting methods at all; they are protocols for structuring what a bounded judge gets to see, and both inherit the same obfuscated arguments problem because both rely on the same premise — that a hard problem broken into checkable pieces, or defended in an adversarial argument, stays checkable all the way down. Weak-to-strong generalization is different in kind from all five: it is not a training method proposed for deployment so much as a measurement framework for finding out, on a problem researchers can already fully check, how much of a capability gap a naive application of an oversight method closes without extra machinery.

None of this supports treating one technique as a replacement for the others, and the clearest evidence against a ranking is how they are actually combined in the systems that use them. DeepSeek-R1’s own published pipeline uses RLVR for its reasoning-heavy stages and still relies on supervised fine-tuning and preference-style data elsewhere ^[14]. Tulu 3 runs supervised fine-tuning, preference optimization, and RLVR in the same recipe rather than choosing one ^[15]. Constitutional AI’s own procedure still ends in an RLHF-shaped reinforcement stage; it changes an input to that stage, not the stage itself ^[6]. A ranking implies a single axis of comparison, and these six techniques were built to solve different, specifically named pieces of the oversight problem — proxy quality, judge scalability, decomposition safety, gap measurement, check reliability — that do not collapse onto one axis.

Predictions, with the observations that would falsify them

These are forecasts, kept separate from the sourced material above. Horizon: 12 August 2030.

One. RLVR will keep expanding into any domain where an automatic checker can be built — including domains that currently look unverifiable, as researchers build partial checkers such as unit-test suites, formal proof assistants, and constrained simulators for narrower slices of previously unverifiable tasks — while the genuinely uncheckable remainder of assistant behavior continues to depend on a learned or AI-judged proxy. Disconfirmed if by 2030 the fraction of production post-training compute spent on RLVR-style objectives has not grown relative to preference-based objectives, or if no new class of automatic checkers has been built for a task category considered unverifiable today.

Two. At least one production system will be shown, through published research rather than company announcement, to exhibit a measurable weak-to-strong-deception-style pattern: apparent compliance on checkable proxy tasks alongside divergence on a held-out set the training signal could not see. Disconfirmed if by 2030 no controlled study of any deployed model has reported this pattern, whether because it does not occur or because no group has looked for it rigorously.

Three. Debate and IDA will remain primarily research techniques rather than production training methods through this horizon, because neither has published an empirical demonstration at the scale RLHF, Constitutional AI, and RLVR now operate at. Disconfirmed if a frontier laboratory publishes a system card describing debate or IDA, rather than a debate-adjacent evaluation protocol, as a primary post-training stage for a released model.

What to take away

Six techniques, six different documented targets. RLHF fits a scalar to what raters could see and judge, and is limited by what raters can see and judge. Constitutional AI and RLAIF replace the rater with a model reading an explicit document, trading tacit standards for inspectable ones without changing the underlying reward-fitting structure. Debate and IDA both try to keep a bounded judge relevant to arbitrarily hard problems by structuring what reaches them, and both inherit the same obfuscation risk because both rest on the same unproven premise about where honesty has the argumentative advantage. Weak-to-strong generalization does not train a deployable system at all; it measures, on a problem small enough to check today, how much of a capability gap closes for free — a number that is informative and, by its own authors’ framing, not yet evidence about the future problem it stands in for. RLVR removes the learned proxy entirely wherever a check exists, and the newest evidence about it shows that even a genuinely unhackable check does not guarantee the reported gain came from the mechanism researchers assumed. Reading any one of these six as a general solution requires ignoring the specific problem its own authors say it addresses — and reading the field as a horse race requires ignoring that several of its supposed competitors are, in the systems that actually ship, stages in the same pipeline.