How Training Data and Synthetic Data Actually Work

A frontier language model is often described as an architecture plus a training run, as if the data were a fixed, given thing sitting inertly underneath both. It is not. By the time a training corpus reaches the point where gradients start flowing, it has already been filtered, deduplicated, checked against known evaluation sets, and — increasingly — supplemented with data the lab generated itself. Every one of those steps is a mechanical, auditable process with its own published methodology, its own failure modes, and, in the case of copyright provenance, its own active litigation. This article walks through how each step actually works, using the dataset papers and legal filings that document them, rather than the folk version where “more data” is treated as a single undifferentiated input.

Filtering: deciding what counts as good text before training begins

The starting point for a modern pretraining corpus is usually a Common Crawl snapshot — a periodic scrape of a large fraction of the public web — and the first fact worth sitting with is how little of that raw scrape survives filtering. The FineWeb project at Hugging Face processed ninety-six Common Crawl snapshots spanning 2013 to 2024 and, after deduplication and quality filtering, retained a fifteen-trillion-token dataset; the paper documents each filtering stage as an ablation, showing what each rule removes and what it costs or buys in downstream model performance ^[7]. Allen Institute for AI’s Dolma corpus, built for the OLMo model family, takes a comparable approach across three trillion tokens and a mix of web text, code, academic papers, books and encyclopedic material, and — importantly for anyone trying to reason about training data rather than take it on faith — documents its own construction pipeline in enough detail that outside researchers can inspect and reproduce each filtering decision ^[6]. This is the first thing worth correcting in how training data gets discussed publicly: “the training set” is not a single object procured once. It is the output of a long, documented pipeline of quality classifiers, heuristic filters (removing boilerplate, adult content, and machine-generated spam), and language identification, run repeatedly as labs iterate on what “quality” should even mean for a given model.

Fact, clearly marked as such: both FineWeb and Dolma publish their filtering pipelines and ablation results, which is why they can be cited as evidence rather than described secondhand ^{[7, 6]}. Vendor assertion, clearly marked as such: claims from closed-source labs about “extensively filtered, high-quality” training data for their flagship models are not independently verifiable in the same way, because those pipelines are not published; the reader should treat such claims as asserted quality, not demonstrated quality, until a lab discloses the pipeline the way Hugging Face and AI2 did.

Deduplication: the un-glamorous fix with an outsized effect

Filtering decides what stays in; deduplication decides how many times a given passage of text gets to count. The canonical study here, from Google and collaborators, found that large web-scraped corpora contain enormous amounts of near-duplicate content — one sixty-one-word sequence, for example, was found repeated more than sixty thousand times in the C4 dataset used for early T5 and other models ^[2]. That kind of repetition matters for two separable reasons. First, it wastes training compute: the model spends gradient steps re-learning the same passage instead of encountering new material. Second, and more consequentially, repeated exact or near-exact passages are exactly the material a model is most likely to memorize verbatim and later regurgitate. The same research group’s follow-up study on training-data extraction showed that adversarial querying of GPT-2 could recover hundreds of verbatim training sequences — including names, contact information, and unique identifiers — and that larger models were more susceptible to this kind of extraction than smaller ones, not less ^[3]. Deduplication is the first and cheapest lever against this: the Lee et al. study found that deduplicating training data before a run reduced the rate at which trained models emitted memorized text by roughly a factor of ten, while training to the same or better accuracy in fewer steps ^[2].

It is worth being precise about what deduplication mechanically does, because “duplicate” is doing more work than it sounds like. Exact deduplication (matching identical documents or exact substrings) is cheap and catches mirrored pages, syndicated wire copy, and repeated boilerplate. Near-duplicate deduplication — typically implemented with locality-sensitive hashing (MinHash) over shingled n-grams — catches paraphrased mirrors, template pages with only a few fields changed, and partial reposts, which are far more common on the open web than exact copies. FineWeb’s published ablations show measurable downstream benchmark differences attributable specifically to the choice of deduplication granularity (per-snapshot versus across the entire corpus), which is itself evidence that this is not a housekeeping detail but a design decision with real consequences for what the model learns to treat as common versus rare ^[7].

Provenance and the copyright question sitting underneath all of it

Filtering and deduplication answer “is this passage good and non-redundant.” They do not answer “did the corpus have the right to use this passage at all,” and that second question is now the subject of active federal litigation rather than settled law. The New York Times filed suit against Microsoft and OpenAI in the Southern District of New York in December 2023, alleging that millions of the paper’s copyrighted articles were used in the training of GPT-based systems without authorization, along with claims of trademark dilution and DMCA violations tied to the removal of copyright management information ^[5]. This is a primary legal document, not a research paper, and it is important to characterize it accurately: it is a set of allegations, tested in court, not an adjudicated finding of infringement. As of the court’s ruling on the defendants’ motion to dismiss in early 2025, the presiding judge allowed the core copyright infringement claims to proceed to further litigation while dismissing the unfair-competition claim with prejudice and most of the DMCA claims without prejudice ^[5]. That procedural posture matters: it means a court found the copyright claims sufficiently plausible to proceed to discovery and further argument, not that infringement has been established, and not that it has been rejected.

Analysis, clearly marked as such: the provenance question is structurally different from the filtering and deduplication questions above, because it cannot be resolved by better engineering. A perfectly deduplicated, perfectly filtered corpus can still be built entirely from unlicensed copyrighted text, and a corpus built from properly licensed sources can still contain duplicates. Provenance tracking — recording where a given training document came from and under what terms — is consequently a separate discipline from quality filtering, closer to editorial sourcing and legal clearance than to data engineering, even though in practice the two pipelines run side by side over the same documents. Readers should not conflate “this corpus was carefully filtered for quality” with “this corpus’s contents were properly licensed”; dataset papers like FineWeb and Dolma are explicit that they filter and document Common Crawl web text under research and fair-use framings specific to their jurisdiction and use case, and that framing is precisely the question being tested in the New York Times litigation and similar suits ^{[7, 6, 5]}.

Contamination: when the test leaks into the training set

A related but distinct failure is evaluation contamination: when material from a benchmark’s test set — or a close paraphrase of it — ends up inside the training corpus, inflating a model’s apparent performance on that benchmark without reflecting a genuine capability gain. This is hard to detect by inspection at scale, because a benchmark question can leak in paraphrased form, in a different language, or embedded inside a longer document, none of which exact-match deduplication would catch. Golchin and Surdeanu’s “Time Travel in LLMs” method addresses this directly: it prompts a model with a “guided instruction” containing a benchmark’s dataset name, its partition label, and a random initial segment of a reference instance, then checks whether the model’s completion matches the held-out remainder closely enough to indicate the model has actually seen that exact instance during training, as opposed to having merely learned the task in general. Their method reported detection accuracy between ninety-two and one hundred percent across seven benchmark datasets tested ^[8]. The mechanism is worth stating plainly because it is often described loosely as “checking for overlap”: it is not a string-match sweep over the training set (which is frequently unavailable to auditors for closed models) but a behavioral probe of the model itself, inferring contamination from how precisely it can complete a benchmark instance it should never have needed to memorize.

Fact: the guided-instruction method is a published, peer-reviewed technique with reported accuracy figures on specific benchmarks, and it targets closed models where the training corpus itself is not inspectable ^[8]. Scenario, explicitly marked: a lab that reports a benchmark score without a corresponding contamination audit is not necessarily reporting a contaminated result — but a reader has no way to distinguish that case from one where contamination occurred, absent a disclosed audit, which is why contamination reporting has become a routine request in peer review of new model releases rather than an optional add-on.

Synthetic curricula: manufacturing training material on purpose

Where the sections above describe curating text the world already produced, synthetic data generation is the deliberate manufacture of new training material by another model, aimed at teaching a specific capability more directly than incidental web text does. The clearest published case is Microsoft Research’s “Textbooks Are All You Need” project, which trained a 1.3-billion parameter code model, phi-1, on a mixture of about six billion tokens of filtered “textbook quality” web text and roughly one billion tokens of synthetic textbooks and exercises generated by GPT-3.5 specifically to teach reasoning and coding concepts in a structured, pedagogical style ^[4]. The result the paper reports is that phi-1, despite its small size and short training run, reached 50.6% pass@1 on the HumanEval coding benchmark — competitive with or exceeding much larger models trained on far more web-scraped text — which the authors attribute to the synthetic data’s density of clearly-explained reasoning steps rather than to scale ^[4].

The mechanical process behind a curriculum like this has a validation step that is easy to skip in summary but is not optional in the actual pipeline: generated exercises and their solutions are filtered again, checking for correctness (does the code actually run and pass its own stated test), for redundancy against material already in the corpus, and for proximity to any benchmark the model will later be evaluated on — the same contamination concern from the previous section, now applied to material the lab manufactured itself rather than material it scraped. A synthetic curriculum that is not checked against its own eventual evaluation set risks manufacturing the exact contamination problem dataset filtering exists to prevent, just by a different route.

If the curriculum has a rough per-item cost of generation and validation, a useful way to reason about scale is a simple accounting identity rather than a law of nature:

where $N$ is the number of exercises targeted, $c_{\text{generate}}$ is the compute cost of producing one candidate item, $p_{\text{fail}}$ is the fraction that fail correctness or contamination filtering and must be regenerated, and $c_{\text{regenerate}}$ is the cost of another attempt. This is not drawn from a specific paper’s disclosed cost model — none of the sources here publish exact figures — it is included only because it makes explicit an assumption that is easy to gloss over in prose: a synthetic curriculum’s real cost is dominated by $p_{\text{fail}}$, the rejection rate, not by the sticker price of one generation call, which is why validation pipelines rather than generation prompts are where most of the engineering effort in these projects actually goes.

Model collapse: what happens when a model trains on its own output, repeatedly

The concern that gets the most public attention, and the one most often overstated in either direction, is “model collapse” — the idea that models trained on data generated by earlier models, recursively across generations, degrade. The founding study, from Shumailov, Shumaylov, Zhao, Gal, Papernot and Anderson, is precise about the mechanism in a way that is worth reproducing carefully, because the informal version in circulation (“AI trained on AI gets dumber”) loses the actual claim. The paper shows that when a generative model is trained on data sampled from a previous generation of the same or a similar model, and that process repeats across generations, the tails of the original data distribution — rare events, unusual phrasings, minority patterns — are progressively underrepresented and eventually vanish, because each generation’s sampling and fitting process introduces small statistical errors that compound rather than cancel. The paper demonstrates the effect across several model classes, including Gaussian mixture models, variational autoencoders, and large language models, and describes the resulting defects as “irreversible” once enough generations have passed without an anchor to real data ^[1].

The word “irreversible” is precise and load-bearing: it does not mean the process is fast, or that any synthetic data at all causes it, or that mixing synthetic data with real data necessarily triggers it. It means that once the tails of the distribution are lost through repeated recursive resampling without fresh real data anchoring each generation, that specific information cannot be recovered from later generations of the same recursive process, because it never encoded a copy of those tails anywhere in the chain. This is the load-bearing distinction between the phi-model result and the collapse result, and conflating them is the most common misreading of both papers. Phi-1’s synthetic textbooks are generated once, by a stronger external teacher model (GPT-3.5), and then mixed with real filtered web text as a curriculum for a smaller student model — a single generation of teacher-to-student distillation, not a recursive chain of a model consuming its own prior outputs across many generations ^[4]. Model collapse, as demonstrated in the Shumailov paper, is specifically about the recursive case: generation $n+1$ trained substantially on the sampled output of generation $n$, repeated across many generations, with the real data anchor diluted or absent ^[1]. Curated, teacher-generated synthetic data used once as a supplement to real data is a different mechanism from an uncurated corpus that increasingly consists of unlabeled prior-model output scraped back off the open web across successive training cycles — and the second describes a real and growing risk as more of the public web itself is populated by earlier models’ output, which is the actual mechanism by which recursive contamination could occur without any lab intending it.

Prediction, with horizon, assumptions and a disconfirmation condition, clearly marked as such: as a larger fraction of newly-crawled web text is itself model-generated, labs that do not distinguish real from synthetic provenance in their web crawls will see measurable narrowing of rare-pattern coverage in models trained on 2027-and-later web snapshots, relative to models trained on 2023-and-earlier snapshots of comparable size — this assumes crawl-dating metadata remains available to make the comparison, and it is assumed that no widescale provenance-labeling standard for AI-generated web content is adopted in the interim. It would be disconfirmed by dataset papers from that period showing stable or improved rare-pattern coverage despite an undiminished (or increased) fraction of detected synthetic content in their crawls, which would indicate that generation-aware filtering had absorbed the risk rather than the risk failing to materialize.

Privacy-preserving synthetic data: a provable trade, not a promise

The final mechanism worth separating out is synthetic data generation aimed specifically at privacy rather than capability — producing a dataset that preserves the statistical properties of a sensitive real dataset (medical records, private communications) without preserving any individual record well enough to be re-identified. The standard mechanical tool here is differential privacy, formalized by Abadi and colleagues’ DP-SGD algorithm, which modifies ordinary stochastic gradient descent by clipping each individual training example’s gradient contribution to a bounded norm and then adding calibrated random noise before the aggregated update is applied ^[9]. The guarantee this produces is a specific, quantifiable one, expressed as a privacy budget:

where $\mathcal{M}$ is the training mechanism, $D$ and $D'$ are two datasets differing by exactly one record, and $S$ is any set of possible outcomes. In plain terms: the probability of any particular trained model (or any synthetic dataset it produces) coming out of the process is bounded so that it cannot depend too strongly, by a factor set by $\varepsilon$, on whether any one individual’s record was included or excluded. This is the load-bearing distinction between “privacy-preserving” as a marketing description and as an engineering guarantee: $\varepsilon$ is a number a data controller chooses and can disclose, and a smaller $\varepsilon$ buys a stronger guarantee at the cost of more injected noise and correspondingly lower utility in the resulting synthetic data or model ^[9].

Vendor assertion, clearly marked as such, versus the underlying fact: a product description that says a synthetic dataset is “privacy-preserving” is not by itself verifiable; a disclosed $\varepsilon$ value, together with the mechanism used to enforce it, is the artifact that makes the claim checkable. Extraction attacks of the kind Carlini and colleagues demonstrated against GPT-2 are precisely the attack differential privacy is designed to bound the success rate of — but a system without a disclosed, enforced privacy budget offers no comparable guarantee at all, whatever language is used to describe it ^{[3, 9]}.

What this adds up to

None of these five mechanisms — filtering, deduplication, provenance and copyright clearance, contamination screening, and synthetic generation with or without a formal privacy guarantee — is a single solved problem with a settled best practice across the field. What the published record does support is that each is a distinct, mechanically different process with its own failure mode: filtering can under- or over-remove; deduplication choices measurably change downstream benchmarks; provenance is a legal question current litigation has not resolved; contamination requires behavioral probing rather than simple string matching to detect reliably; and recursive synthetic training has a specific, demonstrated tail-loss mechanism that is not the same thing as “a model learned from another model’s output” in the singular, curated, teacher-distillation sense. Treating “training data” as one undifferentiated input, rather than the output of this specific pipeline of choices, is the single most common oversimplification in how these systems get discussed — and it is the one this piece has tried to undo, mechanism by mechanism, source by source.