← All parts of this equation

Equation 1 · Part 5 · The Hardest Unsolved Problems in AI Agent Architecture

Symbol T

∇θJ(θ)=Eπ ⁣[∑t=0T∇θlog⁡πθ(at∣st)(∑t′≥trt′)].\nabla_\theta J(\theta) = \mathbb{E}_\pi\!\left[\sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t \mid s_t) \left(\sum_{t' \ge t} r_{t'}\right)\right].
TT

What this part means

the when.

Its job in the formula

T appears in the bound of this sum. The bound states where the repeated operation starts, ends, or which values it includes.

Where the article explains it

When T is small and rewards are dense, that crude attribution washes out quickly.

The passage around this formula

…textbook reward-to-go form is ∇θJ(θ)=Eπ ⁣[∑t=0T∇θlog⁡πθ(at∣st)(∑t′≥trt′)]\nabla_\theta J(\theta) = \mathbb{E}_\pi\!\left[\sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t \mid s_t) \left(\sum_{t' \ge t} r_{t'}\right)\right]. Read the inner sum literally: every action in the trajectory is credited with everything that happens from that point on, not with its own specific contribution. When T is small and rewards are dense, that crude attribution washes out quickly. When T is large and the reward is a single terminal signal — a multi-step coding task that either compiles and passes its tests or does not, a multi-turn support conversation that either resolves the case or does not — the…

Read this part in the article →

Learn the underlying idea

A variable is a named place for a value. Its letter is a local label: x can mean position in one formula and a data point in another.

Open the illustrated variables: a letter stands for a value guide →

See this notation across published equations →

Sources cited in the article section

These citations provide research context; check each source for the exact claim it supports.