Model Collapse in Generative AI
This document summarises the key findings of the research paper “The Curse of Recursion: Training on Generated Data Makes Models Forget,” which introduces the concept of “model collapse”. This phenomenon describes how repeatedly training generative models (like large language models or LLMs) on data generated by previous iterations of those models can lead to a degradation of their performance and an inaccurate perception of the original data distribution. The paper argues that this is a serious issue for the future development of AI, and highlights the continued need for access to genuine, human-generated data.
Key Quote: “We find that use of model-generated content in training causes irreversible defects in the resulting models, where tails of the original content distribution disappear.”
2. The Rise of LLMs and the Problem of Recursion
The paper acknowledges the transformative impact of LLMs (like GPT series) and their ability to generate human-quality text and images. However, it points out that these models are currently trained on vast amounts of data scraped from the internet, which was largely created by humans. As LLMs become more widespread, their own output will inevitably start to dominate online content and therefore, future models may be trained on data largely produced by their predecessors. This creates a recursive loop that introduces problems.
Key Questions: The authors pose the question: “What will happen to GPT-{n} once LLMs contribute much of the language found online?”
3. What is Model Collapse?
The paper introduces model collapse as a “degenerative process” that affects generative models trained recursively. Models start to ‘forget’ the true underlying data distribution over time, even if that distribution doesn’t change. This happens because the models become “poisoned with [their] own projection of reality.”
Definition: “Model Collapse is a degenerative process affecting generations of learned generative models, where generated data end up polluting the training set of the next generation of models; being trained on polluted data, they then mis-perceive reality.”
Key Aspects:
- Early Model Collapse: Loss of information about the ‘tails’ of the data distribution – the less probable or rarer events.
- Late Model Collapse: Different modes (or types of variation) within the original distribution become mixed up, converging to a narrow distribution that resembles the original less and less (often with very small variance).
- This process is different from catastrophic forgetting as it is not about models forgetting learned data, but rather misinterpreting reality by reinforcing their own beliefs based on previous iterations of the model.
4. Causes of Model Collapse
The research identifies two primary sources of error that compound over generations, leading to model collapse:
- Statistical Approximation Error (Primary): Arises from using a finite number of samples to represent the true data distribution. Even with a large sample, some information is inevitably lost during resampling, especially regarding low-probability events. This issue is amplified with every generation.
- The authors explain that “a non-zero probability that information can get lost at every step of re-sampling.”
- Functional Approximation Error (Secondary): Comes from the limitations of the model’s ability to fully represent the true data distribution. This error only exists in the first generation if statistical error is absent and the output of the previous models belong to the image of the new models approximator.
- The authors point out that neural networks aren’t perfect functional approximators in practice: “In particular, a neural network can introduce non-zero likelihood outside of the support of the original distribution.”
These two sources of error cause issues individually and are typically amplified when combined over many generations.
5. Theoretical Intuition
The paper offers a theoretical analysis of model collapse through mathematical models:
- Discrete Distributions: In scenarios with discrete distributions, model collapse occurs due to the statistical error of repeated sampling, leading to the eventual disappearance of low probability events and causing the model’s output to become a delta function centred on one state. This happens even when the model can perfectly approximate the sampled distribution because the data is resampled from the approximated distribution instead of the original distribution.
- Single-Dimensional Gaussian: This analysis showed that errors are introduced each generation in the mean and variance parameter, and that these errors compound over time, causing the model’s representation of the data to drift further from the true distribution. It showed the variance diverges linearly, similar to a Gaussian random walk.
- Noisy Approximation Model: This model generalises the analysis to show that even if a model is capable of adapting, a similar lower bound on the divergence from the original distribution is still applicable, suggesting that super-linear increases in sampling rate are needed to mitigate model collapse. The analysis showed that errors accumulate and cause deviation from the original distribution even when errors in the model are minimised.
Key Conclusion: The analysis suggests that model collapse is an intrinsic problem due to the combination of finite sampling and functional approximation error when models recursively train on their own generated data, and that it is a universal phenomenon.
6. Experimental Evaluation
The researchers tested the model collapse theory on several generative models:
- Gaussian Mixture Models (GMMs): Experiments showed how GMMs progressively lose their ability to represent the underlying distributions, ending up at a point estimate with low variance, where the model is no longer able to discriminate between two distinct modes.
- The paper notes “The performance worsens over time and by iteration 2000 we arrive at a point estimate of the distribution with very little variance.”
- Variational Autoencoders (VAEs): Similarly, VAEs trained on the outputs of their ancestors began to generate “unimodal” and indistinguishable data, a mix of all original training data, and their internal representation of the data becomes ever closer to the mean, causing the disappearance of information from the model.
- The paper highlights that the “original encoder perceives the generated data from its descendant with ever-growing confidence.”
- Large Language Models (LLMs): Fine-tuning experiments with a version of the OPT-125m model, show that performance on the original task degrades, indicating that the effects of Model Collapse are observable even in LLMs. Further analysis showed that models started generating higher probability sequences from the training data and simultaneously started introducing new low-probability sequences, which represent errors due to the recursive training. The paper showed that even when explicitly penalising repeating phrases in generated text, the Model Collapse still occurred, suggesting this wasn’t the cause.
- The paper notes that “over the generations models tend to produce more probable sequences from the original data and start introducing their own improbable sequences.”
7. Implications and Conclusion
The paper concludes that model collapse is a significant problem that must be addressed to ensure the long-term viability of generative AI.
Key Points:
- First-Mover Advantage: Early models trained on genuine human data will likely have an advantage in the long term, as subsequent generations become susceptible to model collapse.
- Importance of Original Data: Access to the original data distribution is crucial to maintaining the quality of generative models. It is essential to collect data from human interactions to counter the pollution of the data used to train models.
- Need for Content Provenance: The ability to distinguish between human-generated and model-generated content is becoming critical, but it is unclear how content generated by LLMs can be tracked at scale. The paper suggests that community-wide co-ordination and data sharing between different actors will be needed to solve this issue.
- Low probability events, that may be very important, are often the first to be forgotten in Model Collapse, which can introduce fairness issues into the models’ behaviour.
Overall Message: Model collapse is a fundamental challenge for generative AI that needs to be addressed, through the preservation of access to original data and consideration to the provenance of data sources. Without these considerations, the capabilities of generative models will degrade over time as models recursively train on their own outputs.
Let me know if you would like a deeper dive into any of these sections or aspects of the paper.