From n-Grams to Reasoning Models: A Technical History of the Language Model

The curve that was not a curve

The standard telling of language model history is a single exponential: someone noticed that bigger models trained on more data got better, and then everyone made bigger models. It is a story with one idea in it, and it is wrong in a way that matters, because it makes the present look like an extrapolation rather than a construction.

The accurate telling is a chain. At each link, a specific and nameable thing was preventing progress; someone identified it; and the fix was structurally different from the fix before it. Scale mattered enormously — but scale was repeatedly unlocked by an architectural change, not the other way round. Between 1948 and 2025 the field removed at least ten distinct constraints, and the ones removed early cannot be re-derived by making the later models larger.

A ropewalk is the useful picture. Hemp fibre is a few feet long at best. Rope is arbitrarily long. The whole craft consists of a sequence of operations — combing, spinning, twisting, laying — each of which converts a limited local material into a longer continuous one, and none of which is simply “more of” the operation before it. The history below has the same shape: a fixed local window of a few words became, by stages, an effectively unbounded one, and each stage used a different mechanism.

Shannon: defining the object before building it

The first move was not a model but a definition. Shannon’s 1948 treatment established that a source of English text can be treated as a stochastic process, that its statistical structure can be measured in bits, and that successive approximations to it — zeroth-order letters, first-order letters weighted by frequency, letter digrams and trigrams, then word-level approximations — produce text of visibly increasing plausibility ^[1]. In 1951 he sharpened the measurement, using human subjects guessing the next letter of concealed text to bound the entropy of printed English from above and below ^[2].

Two consequences follow, and both are load-bearing for everything after.

First, the object of study became prediction. A model of language is a probability assignment over what comes next. The chain rule makes this exact for any sequence of tokens:

Second, quality became measurable without a task. Cross-entropy on held-out text is a number, and a lower number is unambiguously better. That gave the field a scalar to descend for the next seventy years, long before anyone knew what descending it would buy. Brown and colleagues later made the benchmark concrete, estimating an upper bound of 1.75 bits per character for English from a word trigram model measured against a balanced sample, and proposing a common corpus as a standard against which to measure progress ^[3].

The bottleneck Shannon removed: there was no formal object to improve. That is a real bottleneck, and no amount of compute removes it.

n-grams and the sparsity wall

The obvious implementation of the chain rule is to condition on everything. That is impossible, so the Markov approximation truncates the history to a window of fixed width:

Estimate each conditional by counting. This is the n-gram model, and it dominated applied language modelling for roughly three decades because it is cheap, transparent, and surprisingly hard to beat on enough data.

It has one fatal structural property. The number of possible contexts grows exponentially in $n$ while the corpus does not, so almost every context of any useful length has been seen zero times. Maximum-likelihood counting assigns those contexts probability zero, and a single zero anywhere in a sentence sends the whole sequence probability to zero. The model is not merely uncertain about unseen material; it is confidently, catastrophically wrong about it.

The entire smoothing literature exists to patch this. Discounting moves probability mass off observed counts; backoff and interpolation borrow from shorter, denser contexts when the long context is unseen. Chen and Goodman’s empirical study is the standard reference point, comparing the major techniques systematically across training set sizes and n-gram orders and introducing variants that outperformed the established methods ^[4]. Their work is worth reading now for a reason unrelated to smoothing: it is a demonstration that careful, controlled empirical comparison of methods was possible in this field long before it became fashionable.

But smoothing is redistribution, not generalisation. Backing off from a four-gram to a trigram uses less context; it does not use different but similar context. To the model, “the cat sat on the” and “the dog sat on the” are unrelated symbols. Nothing learned about one transfers to the other. That is the wall.

The bottleneck: discrete symbol identity. Every context is atomic, similarity between contexts is invisible, and data requirements therefore grow exponentially with the amount of context you want to use.

Bengio: representation instead of redistribution

The neural probabilistic language model attacked the wall directly. Bengio and colleagues proposed learning a distributed real-valued feature vector for each word jointly with the probability function over sequences expressed in terms of those vectors, explicitly framing the target as the curse of dimensionality in language modelling ^[5].

The mechanism is worth stating plainly because it is the single most under-credited idea in the lineage. If “cat” and “dog” occupy nearby points in a continuous space, then a sentence containing “cat” contributes gradient signal that improves the model’s predictions for sentences containing “dog” — sentences it has never seen. Generalisation now happens through interpolation in a learned space rather than through backoff to a shorter window. Data requirements stop growing exponentially in context length, because the model no longer stores contexts; it stores a function of them.

This is 2003. It is not a scaling result and it did not immediately displace n-grams in production, where count-based models with good smoothing remained competitive for years on the corpora and hardware then available. Its importance is that it changed what the parameters are. Everything downstream — word vectors, contextual embeddings, attention, transformers — is a refinement of the claim that meaning should live in a continuous vector rather than in a symbol’s identity.

The bottleneck removed: no generalisation across similar contexts.

Distributed representations as infrastructure

The next step made representations cheap enough to be used by everything. Mikolov and colleagues introduced architectures that could learn high-quality word vectors from a 1.6-billion-word dataset in under a day, at accuracies that outperformed prior neural approaches on syntactic and semantic similarity tasks ^[6]. The engineering point is the training cost, not the vectors themselves: a representation you can compute overnight becomes a component other people build on, and a representation that takes weeks does not.

A second, quieter infrastructure fix came from machine translation. Sennrich and colleagues showed that encoding rare and unknown words as sequences of subword units — byte pair encoding being the practical choice — made open-vocabulary generation possible without a back-off dictionary, improving translation over baselines that handled unknown words by substitution ^[10]. Every modern tokenizer descends from this. It closed the out-of-vocabulary problem that had haunted every fixed-vocabulary model since the n-gram era, and it did so by changing the unit rather than the model.

The bottleneck removed: representations were expensive, and the vocabulary was closed.

Sequence to sequence, and a bottleneck named in the literature

Word vectors are static. A model that maps one sequence to another needs to consume variable-length input and emit variable-length output, which fixed-window models cannot do. Sutskever, Vinyals and Le supplied the general recipe: a multilayered LSTM encodes the input into a fixed-dimensional vector, and another LSTM decodes the output from it, reporting 34.8 BLEU on the WMT’14 English-to-French test set against 33.3 for the phrase-based baseline, and 36.5 when used to rerank that system’s hypotheses ^[7]. They also reported that reversing the word order of source sentences improved results markedly by shortening the dependencies the model had to carry.

That last detail is a symptom, and Cho and colleagues diagnosed the disease in the same year: encoder-decoder neural translation performed well on short sentences but its performance “degrades rapidly as the length of the sentence and the number of unknown words increase” ^[8]. The cause is architectural and unavoidable. The entire source sentence, however long, must be compressed into one fixed-dimensional vector. Information beyond that vector’s capacity is simply lost.

This is the only stage in the whole lineage where the field’s own name for the problem — the encoder-decoder bottleneck — is literally the word bottleneck, and it was identified in print before the fix was published.

The bottleneck: a fixed-capacity intermediate representation between two variable-length sequences.

Attention as the fix, not as a feature

Bahdanau, Cho and Bengio proposed letting the decoder search the source for the parts relevant to each output word, rather than reading from a single compressed vector ^[9]. The decoder computes, at each output step $i$, a context vector as a weighted sum of all encoder states:

The capacity of the intermediate representation now grows with the input rather than being fixed in advance. They further reported that the learned alignments corresponded well with human linguistic intuition — an interpretability result that arrived free with a performance fix, which is rare.

It is worth being precise about what attention is here, because the retrospective framing distorts it. Attention was proposed as a targeted repair for a specific measured failure mode. It was not proposed as a general-purpose architecture, and for three years it was a module bolted onto recurrent networks.

The bottleneck removed: fixed intermediate capacity.

The transformer removed the last sequential constraint

With attention available, recurrence was doing one remaining job: carrying state along the sequence. It was also imposing one remaining cost: positions had to be processed in order, which prevented parallelisation within a training example and made long sequences slow to train.

Vaswani and colleagues removed recurrence entirely, proposing an architecture “based solely on attention mechanisms, dispensing with recurrence and convolutions entirely”, and reported 28.4 BLEU on WMT 2014 English-to-German and 41.8 on English-to-French with substantially less training time ^[11]. The core operation is a single scaled dot-product:

The analytical point is that this is primarily a hardware result dressed as an architectural one. Every position attends to every other position in one parallel matrix multiplication, which maps precisely onto accelerator hardware in a way that a sequential recurrence never can. The transformer is what made it economically possible to spend the training budgets that the scaling era then spent. Interpretation, clearly labelled as such: without this step the scaling laws would still have been true and would still have been unaffordable.

The bottleneck removed: sequential computation within a training example.

The split: masked and autoregressive

Once pretraining on unlabelled text became the standard first stage, the field forked over a question that sounds narrow and was not: which side of the token should the model be allowed to see?

Peters and colleagues had already shown that representations taken from a deep bidirectional language model, varying with context rather than fixed per word type, improved a broad range of tasks ^[12]. Devlin and colleagues took the bidirectional idea into the pretraining objective itself, masking tokens and predicting them from both directions at every layer, and reported large gains across GLUE, MultiNLI and SQuAD ^[13]. Masked models are excellent encoders. They cannot generate, because they are never trained to continue text.

The autoregressive branch kept the left-to-right objective and got generation for free. OpenAI’s GPT-2 release accompanied the claim that a 1.5-billion-parameter transformer trained on web text performed well across language modelling datasets in a zero-shot setting ^[14]. That work is a preprint and a code release rather than a peer-reviewed paper, and it should be read as such.

For roughly two years the masked branch looked like the winner: it topped the benchmarks that existed. The autoregressive branch won because the objective it optimised turned out to be the one that supports arbitrary tasks stated in text. This is an important corrective to any purely-scaling account. Both branches scaled. They did not converge to the same capability, because the pretraining objective determines what the scaled model can be asked to do.

The bottleneck at issue: task-specific architectures and labelled data for every task. Masked pretraining reduced it to a fine-tuning head; autoregressive pretraining eventually eliminated it.

Few-shot prompting: the fine-tuning step disappears

Brown and colleagues reported that a 175-billion-parameter autoregressive model performed strongly across translation, question-answering and reasoning tasks specified purely by a natural-language instruction and a handful of demonstrations in the prompt, with no gradient updates, while also noting datasets where it remained weak ^[15].

Read carefully, this is a claim about interface, not about intelligence. The capability was in the pretrained distribution; what changed was that no dataset, no training run and no deployment step stood between a user and it. The economic consequence was immediate and is the reason the field’s shape changed in 2020: the marginal cost of supporting a new task fell from weeks of data collection to a paragraph of text.

The bottleneck removed: per-task fine-tuning as a prerequisite for use.

Scaling laws and the correction that followed

By this point compute had become the binding constraint, and the field did not know how to spend it. Kaplan and colleagues characterised the relationship empirically, finding that cross-entropy loss scales as a power law in model size, dataset size and compute, with trends spanning more than seven orders of magnitude, and concluding that compute-efficient training meant very large models on relatively modest data, stopped well short of convergence ^[16]. The functional form is the important part:

Hoffmann and colleagues then trained over four hundred models and reached a different allocation: model size and training tokens should scale in roughly equal proportion, and the large models of the era were significantly undertrained. Chinchilla, at 70 billion parameters trained on roughly four times more data than the 280-billion-parameter Gopher at the same compute budget, outperformed it and reached 67.5% on MMLU ^[17].

Two things deserve separating here. The established fact is that the compute-optimal frontier lies at a much higher token-to-parameter ratio than the earlier work indicated. The interpretation — widely held, and mine — is that this was the most expensive single error in the field’s history, because it caused years of budgets to be spent on parameters that would have bought more capability as data. It is also the clearest evidence against the one-smooth-curve story: two careful groups, using the same methodology on the same object, disagreed about the shape of the curve by a factor that changed industry practice.

The bottleneck removed: not knowing the exchange rate between parameters and data.

Instruction tuning and RLHF: fixing the objective

A model trained to predict the next token is optimising likelihood over a corpus. A user wants a helpful, truthful, non-harmful answer. These objectives are correlated and are not the same, and no amount of pretraining closes the gap because the gap is in the loss function.

Two fixes arrived close together. Wei and colleagues showed that finetuning on a collection of tasks phrased as natural-language instructions substantially improved zero-shot performance on unseen task types, with their 137-billion-parameter FLAN surpassing zero-shot GPT-3 on 20 of 25 evaluated tasks ^[18]. Ouyang and colleagues added human preference: supervised finetuning on demonstrations, a reward model fitted to human comparisons, then reinforcement learning against that reward model. Their headline result is the one worth memorising — outputs from the 1.3-billion-parameter InstructGPT model were preferred by human raters to outputs from the 175-billion-parameter GPT-3, a model with over a hundred times more parameters ^[19].

That single comparison falsifies the pure-scaling narrative more cleanly than any argument. A hundredfold parameter advantage was overturned by changing what the model was optimised for.

The bottleneck removed: objective mismatch between likelihood and usefulness.

Verifiable rewards and trained deliberation

The last link is the newest and the least settled. Wei and colleagues had shown that prompting a model with worked exemplars containing intermediate steps improved multi-step reasoning, with a 540-billion-parameter model reaching state-of-the-art accuracy on GSM8K from eight exemplars ^[20]. Cobbe and colleagues had already established the complementary half: sampling many candidate solutions and training a verifier to rank them scaled better with data than a finetuning baseline on the same benchmark ^[21].

The synthesis was to stop prompting for deliberation and start training for it, using the fact that some answers can be checked automatically. Lambert and colleagues named the technique in the open literature as Reinforcement Learning with Verifiable Rewards, applied within an otherwise conventional post-training pipeline ^[22]. The objective is unusually simple:

where $r$ is not a learned preference model but a program: a unit test that passes, a numerical answer that matches, a proof that checks. Because $r$ is exact, it cannot be gamed the way a learned reward model can, though it is only available where correctness is mechanically decidable.

The strongest peer-reviewed evidence that this produces genuine behavioural change is the DeepSeek-R1 work published in Nature, which reported that reasoning capability can be incentivised through reinforcement learning on verifiable tasks without human-annotated reasoning traces, with self-verification, reflection and dynamic strategy exploration emerging during training rather than being demonstrated ^[23]. Two caveats belong with that: it describes one laboratory’s method and must not be read as a description of any closed frontier system, and “emergent” here means “not explicitly supervised”, not “unexplained”.

The bottleneck removed: deliberation was elicited by the prompt and therefore fragile; it is now a trained policy.

What the chain actually shows

Set the links side by side and the pattern is unmistakable. Shannon supplied the object. n-grams supplied a computable estimator and hit sparsity. Neural language models replaced symbol identity with a learned space. Word vectors and subword units made representation cheap and the vocabulary open. Sequence-to-sequence handled variable length and hit a fixed-capacity bottleneck that its own authors’ colleagues measured. Attention removed that ceiling. The transformer removed sequential computation and made large budgets spendable. The masked and autoregressive branches diverged on the pretraining objective, and the objective, not the scale, decided which one generalised to arbitrary tasks. Few-shot prompting removed the fine-tuning step. Scaling laws priced compute and were then corrected. Instruction tuning and RLHF replaced the objective. Verifiable-reward RL made deliberation a trained behaviour.

Not one of those is “more of the previous thing”. Several were corrections against the prevailing scaling intuition of their moment — Chinchilla against Kaplan, InstructGPT against parameter count. The honest summary is that scale was necessary and never sufficient, and that every era’s bottleneck was invisible until the era before it had been cleared.

Predictions, with horizons and disconfirmation conditions

These are forecasts, separated from everything sourced above. Horizon: 8 August 2029.

One. The next bottleneck to be named and broken will be the reward surface for domains where correctness is not mechanically decidable — long-form analysis, design, judgement under ambiguity — and the fix will be structural rather than a larger verifier. Disconfirmed if by the horizon the dominant published method for such domains is still a learned preference model trained on pairwise human comparisons.

Two. Context length will stop being a headline specification, because the binding constraint will have moved to retrieval and attention quality over the available context rather than its nominal size. Disconfirmed if nominal context windows remain a primary axis of vendor comparison in 2029 model announcements.

Three. At least one widely accepted scaling relationship of the 2024–2026 period will be publicly corrected by a factor large enough to change training practice, as Chinchilla corrected Kaplan. Disconfirmed if no such correction is published and adopted by the horizon.

None of these requires a discontinuity. They are the same pattern the previous seventy years produced: a constraint becomes visible only once the one in front of it is gone.

What to take away

Treat the phrase “just scaling” as a claim requiring evidence rather than a summary. The material fact is that a chain of distinct engineering problems was identified and solved, in an order, by people who mostly knew exactly which problem they were attacking and said so in their abstracts.

The practical use of knowing this is diagnostic. When a system underperforms, the question worth asking is which historical bottleneck it has regressed to — is it sparse in the data sense, mismatched in the objective sense, capacity-limited in the encoder-decoder sense, or simply small? Those have different fixes, and only one of them is more compute. Short fibres do not become rope by being piled higher. They become rope by being combed, spun, twisted and laid, in that order, each operation doing something the previous one could not.