superscript

The passage around this formula

The retrieval literature said the opposite. Lewis and colleagues introduced RAG explicitly as a combination of parametric and non-parametric memory, and the architectural point is that the second is a different kind of object, not an extension of the first [ 1 ] . Their formulation treats the retrieved passage as a latent variable to be marginalised over. Writing x for the query, y for the output, and $\mathcal{Z}_k(x)$ for the top k passages returned by a retriever with parameters $\eta$ : $p(y \mid x) \approx \sum_{z \in \mathcal{Z}_k(x)} p_\eta(z \mid x) \prod_{i=1}^{|y|} p_\theta\left(y_i \mid x, z, y_{1:i-1}\right)$. Read the structure rather than the arithmetic. The generator is never conditioned on the corpus. It is conditioned on z — one span, or a handful — and its output distribution is a…

Read this part in the article →