addition

The passage around this formula

Change one term and the analysis changes with it. Consider instead $\hat{p}_{k+1} = \mathcal{F}\left( D_0 \cup S_1 \cup \cdots \cup S_k \right)$. where $D_0$ is the original real corpus and $S_j$ the synthetic output of generation j . Gerstgrasser and colleagues make exactly this substitution and report that while replacing real data with each generation’s synthetic data does tend toward collapse, accumulating successive generations alongside the original real data avoids it — across transformers, diffusion models and variational autoencoders — and they prove that under accumulation the test error has a finite upper bound independent of the number of iterations [ 2 ] . The accumulating case is also the more accurate description of the actual web, which…

Read this part in the article →