← All parts of this equation

Equation 1 · Part 4 · The Hardest Unsolved Problems in AI Agent Architecture

Symbol t

∇θJ(θ)=Eπ ⁣[∑t=0T∇θlog⁡πθ(at∣st)(∑t′≥trt′)].\nabla_\theta J(\theta) = \mathbb{E}_\pi\!\left[\sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t \mid s_t) \left(\sum_{t' \ge t} r_{t'}\right)\right].
tt

What this part means

t appears in the bound of this sum. The bound states where the repeated operation starts, ends, or which values it includes.

Its job in the formula

t appears in the bound of this sum. The bound states where the repeated operation starts, ends, or which values it includes.

The passage around this formula

…textbook reward-to-go form is ∇θJ(θ)=Eπ ⁣[∑t=0T∇θlog⁡πθ(at∣st)(∑t′≥trt′)]\nabla_\theta J(\theta) = \mathbb{E}_\pi\!\left[\sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t \mid s_t) \left(\sum_{t' \ge t} r_{t'}\right)\right]. Read the inner sum literally: every action in the trajectory is credited with everything that happens from that point on, not with its own specific contribution. When T is small and rewards are dense, that crude attribution washes out quickly. When T is large and the reward is a single terminal signal — a multi-step coding task that either compiles and passes its tests or does not, a multi-turn support conversation that either resolves the case or does not — the…

Read this part in the article →

Learn the underlying idea

A function assigns an output to each allowed input. The expression f(x) means “apply f to x”.

Open the illustrated functions: inputs become outputs guide →

See this notation across published equations →

Sources cited in the article section

These citations provide research context; check each source for the exact claim it supports.