≈

The passage around this formula

By the time Lewis and colleagues published in 2020, learned dense retrieval and large pretrained generators both already existed as separate lines of work. The paper’s specific technical contribution was neither — it was making the whole pipeline trainable end to end through a single differentiable objective, treating the retrieved passage as a latent variable to be marginalized over rather than a fixed input handed to a frozen reader [ 8 ] . Writing x for the input, y for the output, z for a retrieved passage, and $\eta$ and $\theta$ for the retriever’s and generator’s parameters respectively, the RAG-Sequence variant approximates $p(y \mid x) \approx \sum_{z \in \mathrm{top}\text{-}k\left(p_\eta(\cdot \mid x)\right)} p_\eta(z \mid x) \, p_\theta(y \mid x, z)$. holding a single retrieved passage fixed…

Read this part in the article →