← Mathematical compendium

Published equation contexts

P(w1,w2,…,wT)=∏t=1TP(wt∣w1,…,wt−1)P(w_1, w_2, \ldots, w_T) = \prod_{t=1}^{T} P(w_t \mid w_1, \ldots, w_{t-1})

Why this formula appears here

First, the object of study became prediction . A model of language is a probability assignment over what comes next. The chain rule makes this exact for any sequence of tokens: P(w1,w2,…,wT)=∏t=1TP(wt∣w1,…,wt−1)P(w_1, w_2, \ldots, w_T) = \prod_{t=1}^{T} P(w_t \mid w_1, \ldots, w_{t-1}). Second, quality became measurable without a task . Cross-entropy on held-out text is a number, and a lower number is unambiguously better. That gave the field a scalar to descend for the next seventy years, long before anyone knew what descending it would buy. Brown and colleagues later made the benchmark concrete, estimating an upper bound of 1.75 bits per character for English from a word trigram model measured against a balanced sample, and proposing a common corpus as a standard against which…

Read the full article-specific guide →

Read the representative guide

w1w_1

Symbol w_1

w1w_1 is an argument of the function-like quantity on the left; its role is set by that function’s stated inputs.

Read this term in its guide →
w2w_2

Symbol w_2

w2w_2 is an argument of the function-like quantity on the left; its role is set by that function’s stated inputs.

Read this term in its guide →
wTw_T

Symbol w_T

wTw_T is an argument of the function-like quantity on the left; its role is set by that function’s stated inputs.

Read this term in its guide →
tt

Symbol t

t appears in the bound of this product. The bound states where the repeated operation starts, ends, or which values it includes.

Read this term in its guide →
TT

Symbol T

T appears in the bound of this product. The bound states where the repeated operation starts, ends, or which values it includes.

Read this term in its guide →
wt−1w_{t-1}

Symbol w_t-1

wtw_t-1 is one of the signed contributions combined to compute the quantity on the left.

Read this term in its guide →
t=1t=1

Starting index or lower bound: t=1

This label says where the repeated addition, multiplication, or accumulation starts. Read its value or condition together with the article’s description of the index.

Read this term in its guide →
TT

Ending index or upper bound: T

This label says where the repeated addition, multiplication, or accumulation stops. It sets the last term or end of the range.

Read this term in its guide →

How to interpret it

Read it with the definitions, units, and assumptions supplied by the article.

Research cited beside this formula

Learn the mathematics behind it

Published contexts (1)

A symbol can carry a different meaning in another article. Each occurrence keeps its own guide and term definitions.

P(w1,w2,…,wT)=∏t=1TP(wt∣w1,…,wt−1).P(w_1, w_2, \ldots, w_T) = \prod_{t=1}^{T} P(w_t \mid w_1, \ldots, w_{t-1}).

Equation 1 · Foundation Models

From n-Grams to Reasoning Models: A Technical History of the Language Model

This equation states an equality: the expressions on both sides have the same value under the article’s assumptions.

First, the object of study became prediction . A model of language is a probability assignment over what comes next. The chain rule makes this exact for any sequence of tokens: P(w1,w2,…,wT)=∏t=1TP(wt∣w1,…,wt−1)P(w_1, w_2, \ldots, w_T) = \prod_{t=1}^{T} P(w_t \mid w_1, \ldots, w_{t-1}). Second, quality became measurable without a task . Cross-entropy on held-out text is a number, and a lower number is unambiguously better. That gave the field a scalar to descend for the next seventy years, long before anyone knew what descending it would buy. Brown and colleagues later made the benchmark concrete, estimating an upper bound of 1.75 bits per character for English from a word trigram model measured against a balanced sample, and proposing a common corpus as a standard against which…

Equation guide → · Article →