← Back to article

Equation 4 · The Hardest Unsolved Problems in Multimodal AI

What does this equation mean?

Ntok=HP⋅WP,N_{\mathrm{tok}} = \frac{H}{P} \cdot \frac{W}{P},

Read the formula alongside the article passage below. Each part has a deeper page with its role in the equation, the supporting passage and nearby citations.

Start withH
Divide byP
This relates toN_tok
How to read the two sides of this formula. Follow the article passage for the meaning of each quantity.

This equation states an equality: the expressions on both sides have the same value under the article’s assumptions. Read the equation part by part below; each part has a contextual explanation and a link to its mathematical background.

Read it piece by piece

NtokN_{\mathrm{tok}}

Symbol N_tok

NtN_tok is part of the quantity the equation computes from the expression on the right.

Understand this part →

HH

Symbol H

the height.

Understand this part →

PP

Symbol P

P occurs below the fraction bar. The quantity above the bar is divided by this expression; zero is excluded as a denominator.

Understand this part →

WW

Symbol W

W occurs above the fraction bar. The numerator is divided by the entire denominator below it.

Understand this part →

=

=

The expressions on both sides represent the same quantity under the stated assumptions.

Understand this part →

See an illustrated explanation →
fraction

fraction

Divide the expression above the line by the one below it.

Understand this part →

See an illustrated explanation →
multiplication

multiplication

Multiply the quantities on either side.

Understand this part →

subscript

subscript

The lower label selects a particular version, component, or indexed member of the quantity. For example, x₀ and xₜ can be values at different positions.

Understand this part →

How to interpret it

With a fixed numerator, increasing a nonzero denominator reduces the fraction. Read it with the definitions, units, and assumptions supplied by the article.

What the article says around this equation

There is a structural reason this compounds rather than merely adding up, and it is worth making explicit because it exposes an assumption rather than a vague sense of difficulty. A vision transformer front end tokenises an image of height H and width W at patch size P into NtokN_{\mathrm{tok}} = HP\frac{H}{P} ⋅\cdot WP\frac{W}{P}, and a video of T sampled frames multiplies that by T again. Any fixed context budget therefore forces a three-way trade between spatial patch size, temporal sampling rate, and clip duration — coarsen the patches, subsample frames, or truncate the clip. Whichever is chosen, information is discarded before a language model ever sees a token, and the TemporalBench failure…
Read the full surrounding passage
There is a structural reason this compounds rather than merely adding up, and it is worth making explicit because it exposes an assumption rather than a vague sense of difficulty. A vision transformer front end tokenises an image of height H and width W at patch size P into NtokN_{\mathrm{tok}} = HP\frac{H}{P} ⋅\cdot WP\frac{W}{P}, and a video of T sampled frames multiplies that by T again. Any fixed context budget therefore forces a three-way trade between spatial patch size, temporal sampling rate, and clip duration — coarsen the patches, subsample frames, or truncate the clip. Whichever is chosen, information is discarded before a language model ever sees a token, and the TemporalBench failure mode — missing whether an action happened twice or three times — is exactly what temporal subsampling below the rate of the action would predict. This is not a claim that either paper makes explicitly; it is offered here as the mechanism that makes their independently reported findings consistent with each other rather than coincidental.

Read the equation in its article →

Sources cited in the article section

These citations give research context. Read each source to check which claims it supports.

Return to The Hardest Unsolved Problems in Multimodal AI

See this formula across 3 published contexts →

Browse the mathematical compendium →