subtraction

The passage around this formula

A model-free update never references a transition model. A canonical form, temporal-difference learning, updates a value estimate purely from sampled transitions: $Q(s,a) \leftarrow Q(s,a) + \alpha \left[ r + \gamma \max_{a'} Q(s',a') - Q(s,a) \right]$. Nothing in this update requires knowing or estimating p(s' $\mid$ s, a) ; it only requires having experienced (s, a, r, s') . A model-based approach instead fits an explicit dynamics model $\hat{p}_\theta(s' \mid s, a)$ — a “world model” — and plans against it directly:

Read this part in the article →