← Back to article

Equation 2 · From n-Grams to Reasoning Models: A Technical History of the Language Model

What does this equation mean?

P(wt∣w1,…,wt−1)≈P(wt∣wt−n+1,…,wt−1).P(w_t \mid w_1, \ldots, w_{t-1}) \approx P(w_t \mid w_{t-n+1}, \ldots, w_{t-1}).

Read the formula alongside the article passage below. Each part has a deeper page with its role in the equation, the supporting passage and nearby citations.

This equation gives an approximation: it relates the quantities while allowing an approximation. Read the equation part by part below; each part has a contextual explanation and a link to its mathematical background.

Read it piece by piece

PP

Symbol P

P is a part of this expression. Its role is fixed by the surrounding article and by the operations shown in the formula.

Understand this part →

wtw_t

Symbol w_t

wtw_t is a part of this expression. Its role is fixed by the surrounding article and by the operations shown in the formula.

Understand this part →

w1w_1

Symbol w_1

w1w_1 is a part of this expression. Its role is fixed by the surrounding article and by the operations shown in the formula.

Understand this part →

wt−1w_{t-1}

Symbol w_t-1

wtw_t-1 is a part of this expression. Its role is fixed by the surrounding article and by the operations shown in the formula.

Understand this part →

wt−n+1w_{t-n+1}

Symbol w_t-n+1

wtw_t-n+1 is a part of this expression. Its role is fixed by the surrounding article and by the operations shown in the formula.

Understand this part →

≈

≈

Approximately equal to; the equality is not exact.

Understand this part →

addition

addition

Add the term after the plus sign to the term or group before it.

Understand this part →

subtraction

subtraction

Subtract the following term or group from the preceding one. A leading minus marks a negative quantity.

Understand this part →

subscript

subscript

The lower label selects a particular version, component, or indexed member of the quantity. For example, x₀ and xₜ can be values at different positions.

Understand this part →

How to interpret it

Its accuracy depends on the assumptions and range of use described in the article.

What the article says around this equation

The obvious implementation of the chain rule is to condition on everything. That is impossible, so the Markov approximation truncates the history to a window of fixed width: P(wt∣w1,…,wt−1)≈P(wt∣wt−n+1,…,wt−1)P(w_t \mid w_1, \ldots, w_{t-1}) \approx P(w_t \mid w_{t-n+1}, \ldots, w_{t-1}). Estimate each conditional by counting. This is the n-gram model, and it dominated applied language modelling for roughly three decades because it is cheap, transparent, and surprisingly hard to beat on enough data.

Read the equation in its article →

Sources cited in the article section

These citations give research context. Read each source to check which claims it supports.

Return to From n-Grams to Reasoning Models: A Technical History of the Language Model

See this formula across 1 published context →

Browse the mathematical compendium →