A field, not a technique
“Alignment” is often used as a synonym for whatever a specific company did to make its most recent model behave. That usage erases a longer and more interesting history. As a research field, alignment is a lineage of documents that cite one another: philosophical papers written before there was a machine-learning community to publish them in, institutes founded around a specific, then-unfashionable worry, a paper that translated that worry into problems an ordinary machine-learning lab could run experiments on, a training technique that jumped from a research paper to a consumer product in the space of months, a documented alternative to that technique’s central assumption, and a still-open cluster of proposals for supervising systems that may eventually know more than the humans checking their work.
This article is a history of that field, not a survey of current products and not a company’s timeline. It stays close to dated primary sources — the papers themselves, the founding and closing of specific organizations, government institute announcements — and it separates, throughout, what a source claims from what has actually been measured.
Before deep learning: warnings written as philosophy
The concern that a machine given a fixed objective might pursue it in ways its designer never intended predates artificial neural networks by decades. Norbert Wiener stated the problem in Science in 1960, in an article whose central claim is worth reading in something close to its original form: a learning machine that develops its own strategies to satisfy a programmed purpose “may be a genie which we cannot second-guess, and of which we cannot in any real sense arrest the action” once it has been set running, because “the penalty for errors of foresight” in such a machine “is of an entirely different order from that which we generally encounter” [1]. Wiener was writing about game-playing and industrial-control programs of his own era, not about anything resembling a modern language model, and the article should be read as exactly that: a warning about the general logical structure of delegating a fixed purpose to a system that can find strategies its designer did not anticipate, not a prediction about any particular technology.
Five years later, the mathematician I. J. Good pushed the same underlying concern toward a specific, stronger claim in an essay for Advances in Computers. Good defined an “ultraintelligent machine” as one that could “far surpass all the intellectual activities of any man however clever,” argued that such a machine could design still better machines, and concluded that “there would then unquestionably be an ‘intelligence explosion,’” making the ultraintelligent machine “the last invention that man need ever make” — with a caveat the essay states directly: this holds “provided that the machine is docile enough to tell us how to keep it under control” [2]. That single conditional clause is, in a real sense, the seed of everything that follows in this article: Good identified control of a system smarter than its makers as the load-bearing assumption behind the rest of the argument, decades before anyone had a research program aimed at securing it.
Both pieces sat mostly outside computer science for the following half-century, cited more often in philosophy and futures studies than in machine-learning venues. Nick Bostrom’s 2014 book Superintelligence: Paths, Dangers, Strategies, published by Oxford University Press, is the work that carried Wiener’s and Good’s line of concern into a form the wider technology industry engaged with directly, arguing at book length that a sufficiently capable optimizing system pursuing a fixed goal poses control problems that do not resolve themselves as capability increases [3]. The book is philosophy, not an experimental paper, and this article treats it accordingly: as the document that made value alignment a topic technology leaders and machine-learning researchers felt obliged to have a position on, arriving in the same decade that deep learning was beginning to produce results the earlier authors could not have had in mind, but before any of the technical machinery described later in this article existed.
Alignment becomes a distinct research area, with its own institutions
A worry sustained by three papers across fifty years is not yet a research field. A field needs institutions that fund people to work on it full time, and a body of technical work with problems other researchers can pick up and extend. Both arrived in stages.
The earliest dedicated institution in this history predates the term “AI alignment” itself. The Singularity Institute for Artificial Intelligence was incorporated on 27 July 2000 by Brian Atkins, Sabine Atkins, and Eliezer Yudkowsky, originally to accelerate AI development; by 2005 its focus had shifted specifically to identifying and managing the risks Good’s essay had named, and in January 2013 the organization renamed itself the Machine Intelligence Research Institute, a change that marked a sharpened focus on technical research over broader futurist advocacy [4]. Nick Bostrom founded a second institution, the Future of Humanity Institute, at the University of Oxford in November 2005, several years before writing the book described above [5]. FHI ran for nearly nineteen years and is itself now part of this history rather than only a live participant in it: the University of Oxford closed the institute on 16 April 2024, following what reporting on the closure describes as administrative friction with its host faculty rather than any single event [5]. A field whose founding-generation institute has already opened and closed within a quarter-century is a genuinely young field, and that fact is itself part of the record rather than incidental to it.
The paper that did the most to move this concern from philosophy into machine-learning practice appeared in 2016. Amodei, Olah, Steinhardt, Christiano, Schulman, and Mané’s “Concrete Problems in AI Safety” deliberately reframed the field’s central worries as five specific, empirically tractable research problems rather than long-run philosophical arguments: avoiding negative side effects, avoiding reward hacking, scalable oversight, safe exploration, and robustness to distributional shift [6]. The paper’s stated organizing move is to sort these by whether they arise from a poorly specified objective function, from an objective that is too expensive to evaluate often, or from undesirable behavior during the learning process itself [6]. What made the paper historically pivotal was not any single result — it is a research-agenda paper, not an experimental one — but that it gave a graduate student or a small lab a menu of specific problems to run experiments against, where Wiener, Good, and Bostrom had given readers an argument to agree or disagree with.
A concrete technique to work on arrived the following year. Christiano, Leike, Brown, Martic, Legg, and Amodei’s “Deep Reinforcement Learning from Human Preferences” demonstrated that a reinforcement-learning agent could be trained from comparisons between pairs of behavior — a human simply indicating which of two short video clips of the agent’s behavior was better — rather than from a hand-written reward function, and reported that novel behaviors could be trained from feedback on well under one percent of the agent’s interactions with its environment [7]. That paper is the direct technical ancestor of the preference-based training pipeline described in the next section, and its historical significance here is narrower than its technical content: it was the first widely cited demonstration that “alignment,” previously an argument about what could go wrong, could also be a trainable machine-learning method with its own benchmarks and ablations.
The field’s institutional base kept differentiating after this point. Paul Christiano, who had co-authored both the concrete-problems agenda and the preference-learning paper, left a position at OpenAI to found the Alignment Research Center in April 2021, a nonprofit built specifically around theoretical alignment research conducted outside a product-shipping lab [8]. By this stage the field had split into a recognizable set of roles that did not exist in Wiener’s or Good’s time: philosophical argument, research-agenda-setting, applied technique development inside product labs, and independent theoretical and evaluation work, each with its own institutions.
The breakthrough that made an alignment technique commercially real
Every stage described so far produced arguments, agendas, or research results. The next stage produced a shipped training procedure, and the distance it closed between “alignment research” and “what a deployed product actually runs” is the reason this section marks a genuine turning point in the field’s history rather than merely the next paper in a sequence.
Ouyang and colleagues’ 2022 paper, generally known by its resulting model family InstructGPT, combined supervised fine-tuning on human demonstrations with the preference-comparison method from the 2017 paper described above, fitting a learned reward model to human rankings of candidate outputs and then optimizing a language model against that reward model [11]. The paper’s headline empirical result is worth restating for what it establishes historically rather than only technically: human raters preferred outputs from a 1.3-billion-parameter InstructGPT model to outputs from the 175-billion-parameter GPT-3 model it was built from, despite the smaller model having more than a hundred times fewer parameters [11]. That finding reframed what “alignment” bought a product team in terms a business could act on directly — not a safety property alone, but a large improvement in rated output quality obtainable without training a larger, more expensive model.
The historically important fact is what happened to this specific technique in the months after the paper appeared, not the paper’s internal mechanics, which later work in this field has documented and revised extensively. The combination of demonstration-based fine-tuning and preference-model reinforcement learning that this paper describes became, within the same calendar year it was published, the standard post-training step attached to essentially every major conversational language model shipped to a general audience. No prior item in this history — not Wiener’s article, not the founding of MIRI or FHI, not the concrete-problems agenda — had moved from a research finding to a training step running inside a mass-deployed product on anything like that timescale. That compression is the InstructGPT paper’s real historical significance for a field whose founding documents had spent decades as pure argument: it is the moment a specific, named alignment technique became something a product organization could put on a roadmap.
A documented, separately published alternative
Alignment as a research field did not converge on the preference-learning pipeline as its only answer, and the clearest evidence of that is dated to the same year the InstructGPT paper appeared. In December 2022, Bai and colleagues published “Constitutional AI: Harmlessness from AI Feedback,” describing a two-stage method in which a model critiques and revises its own responses against a written set of principles during supervised fine-tuning, and then a model — rather than a human — judges which of two sampled responses better satisfies those principles, with that AI-generated preference data used to train the reward signal for reinforcement learning [12]. The paper’s authors describe the aim plainly: training “a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs” for that specific objective [12].
What matters for a history of the field, as distinct from an account of the method’s mechanics, is the timing and the target of the disagreement it represents. The paper appeared within weeks of the InstructGPT-style pipeline becoming the industry’s default, and it targeted that pipeline’s specific dependency on scarce, expensive, and population-specific human-labeled comparisons for the harmlessness objective — one of the open questions the 2016 concrete-problems agenda had already flagged as expensive to evaluate at scale. Constitutional AI did not replace preference learning; it is still, structurally, a reward model trained on comparisons and a policy optimized against it. What it changed was who, or what, supplies the comparisons. That a fully documented, publicly reasoned alternative to the field’s newly dominant technique existed within the same year that technique reached the market is itself a historical fact worth recording: even as one method became the default, the field’s own literature was already testing whether its central assumption was necessary.
The current frontier: supervising work you cannot fully check
Every stage above eventually runs into a version of the same limit, one the 2016 concrete-problems paper had already named as “scalable supervision”: human comparison and human correction work as a training signal only as long as a human can tell which of two outputs is actually better. As models are trained to attempt tasks a rater cannot easily verify — a long mathematical proof, a large codebase, a technical claim outside the rater’s expertise — that assumption stops holding, and the field’s current, unresolved frontier is a cluster of proposals for what replaces it.
Irving, Christiano, and Amodei’s 2018 “AI Safety via Debate” proposed one answer: train two copies of a model to argue opposing sides of a question in front of a human judge, under a length limit, with the judge deciding which debater was more helpful and honest [9]. The paper’s own framing draws an explicit analogy to computational complexity theory, arguing that a judge relying on a single unverified report is limited to adjudicating claims a bounded verifier can check directly, while a judge equipped with a structured, adversarial debate between two competing reporters can in principle adjudicate a substantially larger class of claims, since each debater is incentivized to expose flaws in the other’s argument that the judge alone would not have found:
That relation is the paper’s real theoretical claim, not decoration: it is the specific argument for why debate should, in principle, extend how complex a claim a time-limited human judge can reliably oversee, and the paper reports early supporting evidence from a simplified image-classification setting where a debate-trained judge’s accuracy rose from 59.4% to 88.9% relative to a judge working from limited direct evidence alone [9].
The same year, a separate team at DeepMind proposed a structurally different answer. Leike, Krueger, Everitt, Martic, Maini, and Legg’s “Scalable Agent Alignment via Reward Modeling” argued for learning a reward function from ongoing interaction with a human user and optimizing that learned reward with reinforcement learning, with the reward model itself recursively assisted by earlier, already-trained agents rather than adjudicated through adversarial debate [10]. Debate and recursive reward modeling were proposed the same year, by researchers who had previously collaborated on other papers in this history, as two different, non-converged technical bets on the same open problem — a genuine disagreement within the field about mechanism, not a settled sequence of one superseding the other.
Both proposals remained largely conceptual for several years, because testing them properly requires a system whose abilities exceed the overseer’s in some checkable way, and such systems were not yet available to test against. Bowman and colleagues’ 2022 “Measuring Progress on Scalable Oversight for Large Language Models” is explicit about this constraint, stating that empirical work on the problem is difficult “since we do not yet have systems that broadly exceed our abilities” [13]. Their response was a substitute experimental design: give a human access to an unreliable large-language-model assistant on tasks — drawn from MMLU and a reading-comprehension benchmark called QuALITY — that unaided humans and the base model alone both perform poorly on, and measure whether the human-plus-model combination outperforms either alone; the paper reports that it does, treating this as a first proof of concept that the general question is empirically tractable now rather than only in the future [13].
Burns and colleagues’ 2023 “Weak-to-Strong Generalization,” from OpenAI, reframed the same underlying problem from the opposite direction: rather than asking whether a human aided by a model can oversee a stronger model, it asks whether a weak model’s supervision alone can elicit a stronger model’s latent capability, testing this across natural-language, chess, and reward-modeling tasks using GPT-4-family models supervised by much smaller ones [14]. The paper reports that naively fine-tuning a strong model on a weak model’s labels consistently recovers a useful fraction of the strong model’s capability, and that an added confidence-based loss term recovers substantially more — but its authors state their own result’s limit directly: “we are still far from recovering the full capabilities of strong models with naive fine-tuning alone, suggesting that techniques like RLHF may scale poorly to superhuman models without further work” [14]. Read together, the two 2022–2023 papers mark the point at which scalable oversight stopped being only a proposal a paper could argue for and became something research teams could put a number on, while both teams’ own stated caveats keep that number from being read as a solved problem.
This same period is when the field’s open research questions became, for the first time, the explicit subject of institutions outside universities and product labs. In May 2023, the Center for AI Safety published a one-sentence public statement — “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war” — signed by a large number of senior researchers and industry figures [15]. Six months later, the United Kingdom announced the AI Safety Institute at the first global AI Safety Summit at Bletchley Park, describing it as the first state-backed organization focused specifically on advanced AI safety for the public interest, built on a £100 million initial commitment and tasked with evaluating frontier models directly rather than funding external research alone [16]. In February 2024, the United States’ National Institute of Standards and Technology announced the U.S. AI Safety Institute Consortium, bringing together more than two hundred member organizations to build what NIST’s director described as “the foundations for a new measurement science in AI safety” [17]. Where MIRI and FHI had been small, independently funded nonprofits and Concrete Problems had been a paper from within a single lab, this wave of institutions is government-convened and explicitly aimed at evaluating models across multiple labs — a different institutional form than anything earlier in this history.
Where the field disagrees
Two disagreements run through the material above, and this article states them rather than resolving them, because the sourcing gathered here does not support resolving them.
The first is mechanism. Debate and recursive reward modeling are both live research directions rather than a settled sequence; neither paper’s authors have published a claim that one has superseded the other, and both approaches, along with newer variants, continue to appear in current scalable-oversight papers side by side [9, 10]. The second is about how much current empirical evidence actually tells us. The authors of the two papers that first tried to measure scalable oversight empirically are each explicit that their own experimental settings are imperfect stand-ins for the eventual problem: one states directly that no system broadly exceeding human ability yet exists to test against, and the other states directly that its own best method still falls well short of full capability recovery [13, 14]. Neither caveat is a reason to dismiss the results. Both are reasons not to read either paper as having closed the question its own authors say it opened.
Predictions, with the observations that would falsify them
These are forecasts, kept separate from the sourced history above. Horizon: 12 August 2031.
One. Scalable-oversight results in the style of the 2022 and 2023 papers described above will become a named, recurring category in frontier system cards, comparable to how capability and red-teaming evaluations are reported today. Disconfirmed if system cards from the leading labs in 2031 still report no scalable-oversight-specific evaluation at all, five years after this article’s publication.
Two. Government AI safety institutes will move from bilateral, lab-by-lab evaluation arrangements toward at least one shared, publicly documented evaluation protocol used across more than one national institute and more than one frontier lab. Disconfirmed if, by 2031, the UK and U.S. institutes named above are still operating exclusively through separate, undisclosed bilateral arrangements with individual labs, with no published shared methodology.
Three. The number of well-funded alignment-focused research organizations operating independently of a product-shipping lab will be higher in 2031 than it was at the Future of Humanity Institute’s closure in April 2024, despite that closure. Disconfirmed if the total count of such organizations with more than twenty researchers has fallen, not risen, over that period — which would suggest FHI’s closure was the start of a contraction rather than an isolated event.
Four. Constitutional-style, self-critique-based alternatives to purely human-labeled preference data will be in routine use at more than one major lab, rather than remaining associated with a single organization’s published method. Disconfirmed if, by 2031, published post-training methods at the major labs still rely exclusively on human-labeled preference comparisons with no documented AI-feedback or self-critique component anywhere in the pipeline.
None of these predictions requires a capability breakthrough. They follow from the pattern already visible across sixty-six years of this specific history: philosophical argument, then dedicated institutions, then concrete technical problems, then a shipped technique, then a documented alternative, then an open and actively contested frontier — each stage building directly on a dated document the previous stage produced.
What the history actually shows
Read as a sequence of dates rather than a single technology’s marketing narrative, AI alignment’s history is longer than any one company’s, and it did not begin with a chatbot. A mathematician stated the core problem in 1960, before there was a field to publish it in. A statistician gave that problem its sharpest form in 1965, with a caveat about control that the rest of this history is, in a real sense, still trying to satisfy. Institutions built specifically to work on that problem opened in 2000 and 2005, and one of them has already closed. A 2016 paper turned the problem into a menu of concrete, checkable research questions. A 2022 paper turned one answer to those questions into the training step behind a product used by a mass audience within months. A separately published 2022 method showed that answer was not the field’s only one. And a cluster of 2018-through-2023 proposals for overseeing systems that may eventually exceed their supervisors remains exactly what its own authors say it is: measured, partially validated, and unresolved. That last description is not a weakness in the reporting. It is the honest state of a field still being written.
