Training on Your Own Output: Synthetic Data and What It Does to a Distribution

The question as usually asked is malformed

Ask whether it is safe to train a model on model-generated text and you will get two confident and opposite answers, both supported by published work in good venues. The disagreement is not really about evidence. It is that the phrase “synthetic data” names an input by its origin, and origin turns out to be almost the least informative property it has. What matters is the process the data passed through between being generated and being trained on, and whether anything outside the model’s own distribution had a say in which samples survived.

A bakery’s culture room makes the distinction physical. A sourdough starter is a population propagated from itself: today’s jar is seeded from yesterday’s jar, and the lineage may be decades old. No baker regards this as a hazard. What keeps the culture alive is not novelty of ancestry but management — fresh flour and water at every feed, a discard step that throws away most of the previous generation, a temperature the baker holds, and a crock and a nose that report whether the population is still behaving. Remove the management and feed the jar nothing but its own drawn-off discard, and it will thin, sour, separate, and eventually stop lifting a dough. Both jars are, in the loose sense, “trained on their own output”. Only one of them is a closed loop.

Everything below is an attempt to say precisely which of those two conditions a given pipeline is in, why the published degeneration results describe the second and not the first, and what has to remain true of a corpus for the distinction to stay checkable from the inside.

Four different reasons the field reaches for generated data

Synthetic data is not one practice with one motivation. There are at least four, and they have different acceptance criteria, different failure modes, and no reason to be argued about together.

Coverage of rare cases. The events that matter most in a deployed system are frequently the ones the natural corpus contains least of: the unusual failure mode, the minority dialect, the drug interaction that occurs in a thousandth of patients. Real data are distributed by how often things happen, not by how much they cost when they happen. Generation is a way of buying density where the world declines to supply it. The risk here is specific and bounded: if the generator’s notion of the rare case is wrong, you have manufactured confident coverage of a fiction.

Licensing. The legal availability of web text has moved sharply and in one direction. A large longitudinal audit of the consent protocols underlying corpora such as C4 found that, over a single year to April 2024, restrictions rose steeply — the authors report roughly five per cent or more of all tokens in C4, and more than 28 per cent of the most actively maintained critical sources in C4, fully restricted by robots.txt, with a far larger fraction restricted by terms of service ^[7]. Generated text carries no such encumbrance, which makes it attractive for reasons that have nothing to do with quality.

Privacy. Where the underlying records are sensitive, synthetic surrogates are a way to move utility without moving the records. Work on differentially private synthetic text has shown that a sensitive dataset can be used to generate synthetic data carrying a formal privacy guarantee, on which downstream classifiers can then be trained non-privately, with results competitive against directly training those classifiers under differential privacy — and with the practical advantage that the synthetic corpus can be reused, retained, and shared ^[8]. This is the cleanest of the four motivations, because the acceptance test is a formal guarantee rather than a judgement.

Exhaustion. The most discussed motivation is the weakest as an argument and the strongest as a pressure. Forecasting demand against an estimate of the total stock of public human text, one widely cited analysis concludes that on current trends models will be trained on datasets roughly equal in size to that stock somewhere between 2026 and 2032, earlier if models are deliberately overtrained ^[6]. That is a projection with wide error bars, not a wall with a date on it. But it does mean the marginal high-quality human token is getting more expensive, and that generated tokens are increasingly the cheapest thing on the shelf.

Note that only the first and third of these want the synthetic data to be unlike what the model already produces well. The second and fourth are content for it to be exactly like it — which is the condition under which the trouble starts.

What recursive self-training actually does

The degeneration is real, it is documented, and it has a mechanism worth stating carefully.

Shumailov and colleagues, publishing in Nature, fine-tuned OPT-125m on the wikitext2 dataset, then generated a new training set from that model and fitted the next generation on it, repeating the cycle. They report two phases. Early model collapse is the loss of information about the tails of the distribution — low-probability events stop being represented. Late model collapse is convergence toward a distribution with substantially reduced variance, tending in the limit toward something close to a point mass, with little resemblance to the original. In their language-model experiments the original model reached a mean perplexity of about 34; under the condition where no original data were preserved, performance degraded by roughly 20 to 28 perplexity points across generations. Their histograms show generated samples piling up around high-probability sequences while also developing long tails of errors the original model never produced ^[1].

That last detail is the one most often dropped in summary, and it is the most informative. The failure is not simply narrowing. It is narrowing in the region the data actually occupied, plus the invention of a new and spurious tail of artefacts. The culture does not only get weaker; it acquires a smell that was not in the ancestor.

The same shape appears in images. An analysis of what the authors call autophagous loops — three families differing in how much fresh real data is available at each generation, and in whether samples are biased to trade quality against diversity — concludes that without enough fresh real data in each generation, quality (precision) or diversity (recall) progressively decreases ^[4]. The phrase to hold onto is enough fresh real data in each generation, because it names the variable rather than the outcome.

A third line of work reframes the phenomenon in the language of scaling. Rather than asking whether models fail, it asks what happens to the scaling law itself when synthesised data enter the corpus, and finds a family of decay phenomena: loss of scaling, scaling shifted by the number of generations, the “un-learning” of skills, and grokking behaviour when human and synthesised data are mixed, validated on a transformer arithmetic task and on text generation with Llama 2 ^[3]. This is a more useful framing for practitioners than a binary, because it says the damage shows up as a change in the return on further scale, which is not something a single benchmark run can see.

Two compounding mechanisms produce all of this, and neither is exotic. The first is finite sampling: any finite sample from a distribution under-represents its tail, and a model fitted to that sample inherits the deficit and passes it on amplified. The second is truncation at generation time. The reported protocol used five-way beam search over 64-token continuations ^[1]; beam search is a mode-seeking procedure, and so are the temperature and nucleus settings used in essentially every production sampler. This is analysis rather than a finding: decoding strategies chosen to make output look good are, by construction, strategies that discard low-probability mass. If the output of such a sampler becomes the next training set, the discarding is not a display choice any more. It is an edit to the distribution.

The assumptions doing the work

Now the part that is easy to overstate. The collapse results are correct about the regime they study. The regime has a specific structure, and it is worth writing down what it assumes.

Let $\hat{p}_k$ be the model fitted at generation $k$, and $\mathcal{F}$ the fitting procedure. The classical collapse setting is

in which generation $k+1$ sees $n$ samples drawn from its immediate predecessor and nothing else. The original data are gone. No filter selects among the samples. The lineage is single. Under those conditions, degradation compounds because there is no channel by which an error introduced at generation $k$ can ever be corrected.

Change one term and the analysis changes with it. Consider instead

where $D_0$ is the original real corpus and $S_j$ the synthetic output of generation $j$. Gerstgrasser and colleagues make exactly this substitution and report that while replacing real data with each generation’s synthetic data does tend toward collapse, accumulating successive generations alongside the original real data avoids it — across transformers, diffusion models and variational autoencoders — and they prove that under accumulation the test error has a finite upper bound independent of the number of iterations ^[2]. The accumulating case is also the more accurate description of the actual web, which does not delete last year’s pages when this year’s are published.

The original Nature work contains its own version of this result and it is rarely quoted: preserving ten per cent of the original training data across generations produced only minor degradation of performance ^[1]. A tenth. That is the discard schedule, and it is the whole difference between a starter that runs for a decade and one that dies in a fortnight.

There is a live disagreement here, and it should be characterised rather than resolved by preference. A position paper by Schaeffer and colleagues argues that the public narrative has misread the science: they identify eight distinct and sometimes conflicting definitions of model collapse across the literature, grouped loosely into the behaviour of test loss on real data across fitting iterations, the deformation of the data distribution across iterations, and the scaling behaviour of test loss; and they contend that several prominent collapse scenarios rest on assumptions that poorly match real-world conditions and are readily avoidable ^[5]. The disagreement is not about the mathematics of the replacement setting, which nobody disputes. It is about which idealisation is the better model of how corpora are actually assembled, and about whether “collapse” should name a mechanism or a forecast. Both camps would agree, I think, on the operational statement: unmanaged full replacement is dangerous, and it is also not what anyone deliberately does.

Filtering makes it a different object

The decisive variable is whether anything selects. A filter is an information channel from outside the loop, and its criterion — a unit test, a solver, a human preference, a deduplication threshold, a verifier model — carries information the generator did not have. This is why the same generated tokens can be poison or nutrition depending on what happened between generation and training.

The argument has been made formally. Feng and colleagues start from the observation that it is easier, for humans and machines alike, to tell good examples from bad ones than to generate high-quality samples in the first place, and study verification as the intervention. On two tasks that both exhibit collapse when trained naively on generated data — computing matrix eigenvalues with transformers, and news summarisation with language models — they show that verifiers, even imperfect ones, can be harnessed to prevent collapse, and that a measurable proxy for verifier quality correlates strongly with the resulting performance ^[9]. The theoretical part derives conditions under which a verifier is good enough to be worth using, which is the question a practitioner actually faces.

The practice long predates the theory. Self-Instruct bootstrapped instruction-following from a model’s own generations while filtering invalid and overly similar samples before fine-tuning, and reported a 33 per cent absolute improvement over vanilla GPT-3 on Super-NaturalInstructions, on par with a model trained on private user data and human annotation ^[10]. The filter is not incidental to that pipeline; it is the pipeline. Similarly, the phi-1 result trained a 1.3-billion-parameter code model on six billion tokens of “textbook quality” data selected from the web plus one billion tokens synthesised with GPT-3.5, reaching 50.6 per cent pass@1 on HumanEval and 55.5 per cent on MBPP at a scale where that was surprising ^[11]. Read carefully, that is at least as much a result about selection as about generation.

Synthetic data also has its own scaling behaviour, and it is finite. A systematic study of synthetic-data scaling reports that generated corpora can be made to follow a predictable rectified scaling law across model sizes, but that performance improvements plateau near 300 billion tokens, with larger models reaching their optimum on fewer tokens — an 8-billion-parameter model peaking around one trillion tokens where a 3-billion-parameter model needed four ^[12]. Synthetic data is therefore best understood as a resource with diminishing returns like any other, rather than as either a contaminant or a perpetual-motion machine.

Distillation is the case that plainly works

The most successful use of model-generated supervision is also the oldest, and it is instructive because it breaks the recursion at the structural level. Knowledge distillation trains a student to match the output distribution of a teacher rather than the hard labels, on the reasoning that the teacher’s full distribution over classes carries information — the relative probabilities of the wrong answers — that a one-hot label discards ^[13]. Every token the student learns from is synthetic. Nobody calls it collapse.

Three structural features explain why. The teacher is fixed: nothing the student produces flows back into it, so there is no feedback loop to compound. The chain has length one, not $k$. And the objective is transfer of a known-good distribution, not improvement beyond it, so the success criterion is agreement rather than novelty. The relevant failure mode is entirely different from collapse: a student inherits the teacher’s errors and cannot exceed the teacher’s ceiling on the distilled behaviour.

The pattern remains current. The Nature paper on DeepSeek-R1 states that the reasoning patterns of the large RL-trained model can be systematically used to guide and enhance smaller models, and that the resulting distilled models exhibit strong reasoning capabilities, surpassing their original instruction-tuned counterparts ^[14]. That is a vendor-adjacent claim published under peer review, and it should be read as what it is: evidence that a single deliberate generation of synthetic supervision, aimed at transfer, is a productive engineering technique. It is not evidence about repeated self-training.

Where a checker exists, the argument is over

There is a class of domains in which synthetic data is unambiguously safe, and it is defined by a single property: an external procedure can decide whether a generated example is correct, without consulting the generator.

AlphaGeometry is the cleanest demonstration. Faced with a severe scarcity of machine-verifiable geometry proofs, its authors sidestepped human demonstrations entirely by synthesising 100 million unique theorems and proofs across levels of complexity, training a language model from scratch on that data to guide a symbolic deduction engine. On a test set of 30 recent olympiad-level problems it solved 25, against ten for the previous best method, approaching the performance of an average International Mathematical Olympiad gold medallist ^[15]. The training corpus was entirely generated. It was also entirely checked, because a symbolic deduction engine is not a model and does not share the generator’s blind spots.

AlphaProof extends the same logic into general mathematics through the Lean proof assistant. It was trained by reinforcement learning on approximately 80 million formal problems auto-formalised from around one million natural-language problems, and at the 2024 IMO it solved three of the five non-geometry problems, including the hardest; combined with AlphaGeometry 2 the system reached 28 out of 42 points, within the silver-medal range ^[16]. Lean’s kernel is the referee. A proof is accepted or it is not, and no amount of fluent plausibility substitutes.

Two boundaries need stating, because this case is frequently over-generalised. First, the guarantee applies precisely to what the checker checks. In AlphaProof the auto-formalisation step used a fine-tuned language model refined with human and synthetic data ^[16]; Lean can certify that the formal statement was proved, not that the formal statement faithfully renders the informal problem. The verified region ends at the translation boundary. Second, most valuable text has no checker. There is no kernel that accepts or rejects a paragraph of legal analysis. In those domains the best available substitute is a fallible verifier, which the theory says still helps, provided you can estimate how fallible it is ^[9].

The measurement problem is the hard part

Suppose the tails are quietly going. How would you know?

The uncomfortable answer is that the standard instruments are close to blind to it. An aggregate benchmark score is a mean over a test set whose mass sits where the data’s mass sits — in the head. Early collapse, in the Nature framing, is precisely the loss of low-probability events ^[1], which are by construction the events a head-weighted test set contains few of. A model can shed a great deal of its tail while its headline numbers move within noise. The scaling-law framing sharpens this: the reported symptoms include a shift in scaling with the number of generations and the un-learning of specific skills ^[3], neither of which is a level effect that one run at one scale can reveal.

This is the crock. The culture rises past the band exactly as it should; the measurement you habitually take says healthy. The information that something has changed is in the bubble structure, which nobody wrote down, and in the first thin bead of separated liquid at the glass.

Four instruments follow, and they are the practical content of this section.

Freeze a real anchor and never let it into training. A corpus of human data, fixed on a date, held out permanently, and re-evaluated after every data-generation round. Perplexity on that anchor is the closest available analogue to the starter’s rise time — a stable, boring number whose drift is the signal.

Measure per slice, and define the slices before training. Aggregate accuracy hides rare-class recall by design. Slices defined after seeing results are a way of not finding anything.

Measure diversity, not only quality. The autophagy work is explicit that the loop can degrade either precision or recall depending on how sampling is biased ^[4], and a quality-only dashboard will therefore miss half the failure modes. Distinct-n-gram rates, embedding dispersion, and output-length distributions are cheap and they move first.

Never regenerate an evaluation set with the pipeline that generates training data. If the same generator writes both, a filter’s blind spot becomes invisible in exactly the measurement designed to catch it. This is the one hard rule in this article.

Provenance, and what it can realistically deliver

If the difference between managed and unmanaged propagation is decisive, the obvious question is whether provenance metadata can tell you which one a corpus went through. The honest answer is: not from outside, and not soon.

The leading content-provenance standard is worth reading closely for what it does not claim. The C2PA technical specification defines assertions as data structures representing statements made by a signer or gathered at claim-generation time, combined into a cryptographically signed claim describing an asset’s history ^[17]. It is a chain-of-custody format for individual assets. Version 2.2 removed the previous Training and Data Mining assertion, retaining only a digitalSourceType value for algorithmically trained data as metadata, and the specification is oriented toward recording what happened to content — actions, edits, ingredients — rather than declaring how content was generated ^[17]. It is a strong standard for the problem it addresses. It is not, and does not present itself as, a labelling system for pretraining corpora.

The other external lever is site-level consent, and the audit cited earlier shows it being applied rapidly, inconsistently, and with acute divergence between robots.txt and terms of service ^[7]. Consent signals tell you whether you were permitted to crawl a page. They tell you nothing about whether a model wrote it.

So the control that actually works is internal, unglamorous, and entirely within a team’s own gift: corpus lineage as a first-class artefact. For every shard in the mix, record where it came from; whether a model produced it and which one; what filter admitted it and what that filter’s criterion was; and what fraction of the final token budget it represents. Keep the synthetic fraction a declared parameter rather than an emergent property of whatever the crawler happened to find. Keep the frozen real anchor outside the whole apparatus. None of this requires a standards body, and all of it is the difference between being able to answer “what changed between these two checkpoints” and not.

Vendor statements deserve their usual treatment. Several providers assert that their pipelines use heavily filtered and verified synthetic data. Those are claims; the pipelines are generally not disclosed in enough detail to check, and the published evidence above establishes that filtering can work, not that any particular unspecified filter did.

Predictions, with the observations that would falsify them

These are forecasts and clearly separated from the sourced analysis. Horizon: 8 August 2029. They assume no change in the basic economics — that generated tokens remain far cheaper than commissioned human ones, and that no external provenance standard becomes mandatory for training corpora.

One. Unfiltered recursive self-training will remain a laboratory construct rather than a deployed practice, and the degradations actually observed in production systems will be traced to mix-ratio errors, evaluation contamination, and filter blind spots rather than to collapse in the classical sense. Disconfirmed if a major provider publicly attributes a capability regression to recursive contamination of its pretraining corpus.

Two. The unit of discussion will shift from “how much synthetic data” to “what verified it”, and technical reports will describe pipelines by their verifier rather than by their synthetic fraction. Disconfirmed if leading technical reports in 2029 still quote synthetic-data proportions with no account of the verification step.

Three. Per-slice and distributional metrics will appear alongside aggregate benchmarks in model documentation, because tail loss is the failure aggregate scores cannot see. Disconfirmed if system cards in 2029 continue to report only aggregate means with no diversity or rare-slice reporting.

Four. Content-provenance standards will not become the filtering mechanism for pretraining corpora; internal lineage tracking will do that work instead. Disconfirmed if a major pretraining corpus is publicly documented as having been filtered at scale using cryptographic content credentials.

None of these requires anything novel to happen. They follow from what is already visible: a mechanism that needs a closed loop to operate, a set of interventions that reliably open the loop, and a measurement apparatus that has not yet caught up with either.

What to take away

Synthetic data is not a substance with a fixed effect. It is an input whose consequences are determined almost entirely by what selected it. Recursive fitting on unfiltered self-output erases the tails first and the variance afterwards, and the published demonstrations of this are sound within their assumptions — full replacement, no accumulation, no filter, one lineage. Relax any one of those and the picture changes: accumulate rather than replace and the error is bounded; keep a tenth of the real data and the degradation is minor; put a verifier in the loop, even an imperfect one, and generated data becomes a resource with an ordinary scaling curve. Where a genuine checker exists, the argument is simply over.

The baker’s discipline is the right one to import, and it is three rules long. Feed something from outside the lineage at every generation. Discard most of the previous generation rather than compounding it. And keep one measurement that has nothing to do with how good the last batch looked — a frozen anchor, taken on a schedule, whose drift you are willing to believe over your own impression of the dough.