What happens when AI trains on AI (model collapse)?

Make a photocopy of a photocopy of a photocopy, and the faint lines go first. A pencil note in the margin fades, then a light gray shadow. Each copy looks almost like the one before, but after enough rounds the page has lost its detail. Researchers found something similar when artificial intelligence (AI) models learned from other models' writing, round after round, with too little written by people. They call it model collapse.¹

Language models learn from enormous amounts of text, and most of it was written by people: books, articles, forum posts, web pages. More and more of what you read online is now written with help from AI. So researchers asked a simple question. What happens if future models learn mostly from text that earlier models wrote?¹

A team led by Ilia Shumailov at the University of Oxford tested it, and published the results in the journal Nature in July 2024. They took a language model and trained a fresh copy on text the first one had written. Then they did it again. And again. In the main test, each new model learned only from the writing of the one before it, generation after generation.¹

The first thing to fade was the unusual. Rare words, uncommon facts, and less likely ways of saying things showed up a little less in each generation's writing, so the next model saw them even less. Common patterns got more common. The researchers describe the edges of the original data disappearing. Those edges are where the rare cases live.¹²

Given enough rounds, the writing drifted into nonsense. In one example, the first model was given a passage about church towers built in England in the Middle Ages. By the ninth generation, its descendant was producing a list of jackrabbits with different colored tails. Every model makes small mistakes, and here each generation copied the last one's mistakes and added its own.¹²

One detail changes the story. The collapse came when AI-written text replaced the human-written data, round after round. When the team kept a tenth of the original human data in every round, performance dropped only slightly. A separate 2024 study led by researchers at Stanford University went further: when new data was added to the old instead of replacing it, the decline leveled off. So model collapse is a real effect, and it depends on how the data is handled.¹³

Careful teams plan around this. They keep human-written data in the mix, filter out low-quality machine text, and try to track where their data came from. The Nature paper's authors say access to original, human-made data needs to be preserved over time. That's one reason writing by real people, about real life, keeps its value. Next time you see a headline saying AI is getting worse by learning from itself, look for one detail: did the study replace the human data, or add to it?¹³