superscript

The passage around this formula

Nothing in this update requires knowing or estimating p(s' $\mid$ s, a) ; it only requires having experienced (s, a, r, s') . A model-based approach instead fits an explicit dynamics model $\hat{p}_\theta(s' \mid s, a)$ — a “world model” — and plans against it directly: $a^{*}_{1:H} = \arg\max_{a_{1:H}} \; \mathbb{E}_{\hat{p}_\theta} \left[ \sum_{t=1}^{H} \gamma^{t} \, r(s_t, a_t) \right]$. A comprehensive survey of the model-based literature frames the trade this way: fitting and planning against $\hat{p}_\theta$ typically buys sample efficiency, because every transition teaches the model something reusable across many hypothetical future plans rather than updating one value estimate, and it buys interpretability, because the model can be queried and its predictions checked against reality…

Read this part in the article →