Understanding Model Collapse in Artificial Intelligence

When AI models are trained on AI-generated data, they forget. A Nature paper by Shumailov et al. calls this model collapse. The tails of the distribution disappear first. Rare events vanish. Then variance collapses. What remains? Maximum confidence. Minimum reality. The mechanism is simple: Finite sampling loses rare signals. Each generation compounds that loss. The math shows convergence toward a delta function. A model that becomes increasingly certain… about a shrinking version of truth. Here’s why this matters now: LLM-generated text is flooding the internet. Future models will be trained on that text. GPT-{n+1} will inevitably contain output from GPT-{n}. This recursive contamination isn't hypothetical. It's a degenerative process with predictable dynamics. The implication: Human-generated data becomes more valuable over time, not less. Access to the original distribution isn’t optional. It’s foundational. The parallel to clinical research: We've spent decades training our intuition on summary statistics. Group means. Terminal endpoints. Compressed representations of complex biology. Each generation of researchers inherits the compressed version, not the raw signal. The tails disappear. The rare phenotypes. The unexpected responders. The patterns that weren't in the scoring rubric. Model collapse isn't AI-specific. It's information-theoretic. It applies to any system that learns recursively from its own simplified outputs. The antidote in AI: preserve access to original, human-generated data. The antidote in clinical research: preserve access to continuous, high-resolution, unfiltered biological signals. Both require the same thing. Infrastructure that captures reality before someone decides what to discard. That's not a technology problem. It's a design choice. As healthcare AI scales, this becomes critical. If we train on AI-summarized notes, filtered datasets, and compressed clinical narratives… We won’t notice the loss at first. The rare cases disappear quietly. Then clinical variance collapses. And eventually: We’ll have extremely confident systems… That no longer understand real patients. — Source: Shumailov et al., Nature 631, 755-759 (2024) 📌 Save this. Share with teams building healthcare AI training datasets. ♻️ Repost, someone designing clinical AI infrastructure needs to understand model collapse. 🔔 Follow Bhargav Patel, MD, MBA for healthcare AI insights on data quality and long-term model integrity.

Pandora’s Corpus: The Model Collapse Already Happening Inside Your Enterprise In 2023, Shumailov et al. (“The Curse of Recursion”) showed that when models train on model-generated data, the distribution collapses. Rare modes disappear first. The tails go silent. Now apply that inside your company. Your GenAI is generating executive summaries, CRM notes, underwriting commentary, code documentation, data annotations. At scale. Those artifacts land in SharePoint, Confluence, data lakes, vector stores. Within 12 to 24 months, synthetic and human-authored knowledge become indistinguishable. Then you fine-tune your next model on that corpus. That is recursive synthetic training inside your own enterprise boundary. What disappears first isn’t fluency. It’s edge-case judgment: the atypical claim, the nuanced regulatory interpretation, the exception underwriting logic. In regulated industries, that tail knowledge is where capital, compliance, and litigation risk live. The model won’t signal degradation. Your dashboards won’t turn red. Collapse is distributional, not dramatic. Three controls. Architectural, not aspirational. 1. Immutable provenance at write-time. AI-assisted artifacts must carry enforced, machine-readable origin metadata at the storage layer. 2. Synthetic exclusion partitions. Physically separate primary human corpora from AI-generated content. Training pipelines must default to exclusion. 3. Protected tail domains. Ring-fence human-validated underwriting, adjudication, and regulatory logic from recursive ingestion. Speed was correct. Uncontrolled recursion is not. #ModelCollapse #AIGovernance #EnterpriseAI

Just read LeCun's latest paper. His team trained the first world model that can't collapse. Let me explain why this matters. It's called LeWorldModel. World models predict what happens next physically. Objects moving, falling, colliding. That's the base layer for robots that plan, cars that simulate before they steer, any AI that acts in reality instead of just talking about it. The catch is nobody could train these reliably. The models kept cheating. They'd map every input to the same output. Like a weather app stuck on "sunny" forever. Technically predicting. Completely useless. So teams piled on fixes. Frozen encoders, stop-gradient hacks, 6+ loss hyperparameters. A fragile stack too brittle for production. This team asked a different question. What if you make collapse mathematically impossible? An encoder turns each video frame into a small vector. A predictor takes that vector plus an action and guesses the next one. First loss: how wrong was the guess. Second loss: a regularizer called SIGReg that checks if vectors spread out like a bell curve. If they start looking the same, the loss spikes. The model can't cheat because the math won't let it. Six hyperparameters became one. 15M parameters. Trains on one GPU in hours. Plans 48x faster. Encodes with ~200x fewer tokens. Open-source. I could run this on my own hardware. Which changes who gets to build physical AI. Not just big labs anymore. Any team, any startup, any grad student. LeCun has pushed JEPA as the path forward. The criticism was always training instability. This paper removes that objection. Two directions compete in AI right now. Bigger LLMs with more compute. Or small models learning physics from raw pixels.

There's much hand-wringing about "model collapse" - what happens when AI starts doing the Ouroboros thing and gets trained on its own output more than from human-generated content. This is especially nasty in AI code generation, I think. AI and human-generated code all ends up in the same repositories used for training with no way to distinguish it. Today's AI-generated code is of inconsistent quality, and human review of that code is spotty. It seems as though this must lead to AI-generated code getting worse instead of better over time. But what worries me more is humans relying more on AI-generated code, and losing their coding chops - and ability to review and correct that code - at the same time. This doesn't end well. Do we weigh training data more if it comes from before 2024 and was more likely to be human-generated? That's not sustainable; as new technologies and API's are introduced, how does AI learn how to use them properly? Meanwhile tech CEO's - even those who should know better - seem quite intent on relying on AI for coding in place of expensive and moody software engineers. There's a lot of power and money behind making model collapse happen. Perhaps it just has to get worse before it gets better. At some point, skilled human software engineers will have to swoop back in and clean up the mess. Or maybe your new job is writing code to train AI with explicitly. We know OpenAI already contracts programmers for this purpose, and it's not a fun job. And if you're working for a company that is producing something truly new and innovative, producing documentation for your API's won't be enough - you'll also have to produce code examples to feed AI for training purposes. Strange times ahead no matter how you cut it.

The reality check AI Builders need: Apple's 'Illusion of Thinking' research. Apple just dropped some groundbreaking research that every AI builder should read. Their study on Large Reasoning Models (LRMs) reveals something counterintuitive that could reshape how we build agentic AI solutions. The unexpected finding? These "thinking" models hit a wall faster than we thought:- 🔍 Three Performance Zones Discovered:- - Low complexity: Standard LLMs actually outperform reasoning models. - Medium complexity: Reasoning models show their advantage. - High complexity: Both models completely collapse. The most surprising insight:- LRMs increase their reasoning effort up to a point, then give up entirely—even when they have plenty of compute budget left. It's like watching someone work harder and harder on a puzzle, then suddenly walk away. What this means for AI builders:- → We need better evaluation beyond just 'final answer accuracy'. → Data contamination is masking real capabilities in benchmarks. → There's a sweet spot where reasoning helps, but it's narrower than expected. → We're still far from true algorithmic reasoning. As someone building with agentic AI, this research validates what many of us suspected - current "reasoning" is more pattern matching than genuine problem-solving. The models struggle with exact computation and fail to apply consistent algorithms across similar problems. The bottom line:- We're in the "medium complexity" sweet spot right now, but we need to be realistic about limitations as we scale up our AI solutions. What's your experience been with reasoning models in production? Are you seeing similar patterns? #AI #MachineLearning #ReasoningModels #Innovation #AppleResearch #AgenticAI

The Illusion of the Illusion of Thinking 🤯 Claude's team just dropped a comment on the Shojaee et al. (2025) paper about Large Reasoning Models (LRMs) hitting an "accuracy collapse." And it's a must-read for anyone building or evaluating AI. The findings suggest the "collapse" isn't a fundamental reasoning failure. It's an experimental design failure. No more misinterpreting model capabilities. No more flawed automated evaluations. No more penalizing AI for being smart. 𝗛𝗲𝗿𝗲'𝘀 𝘄𝗵𝗮𝘁 𝘁𝗵𝗲 𝗻𝗲𝘄 𝗽𝗮𝗽𝗲𝗿 𝗿𝗲𝘃𝗲𝗮𝗹𝗲𝗱: ↳ The models actually hit their token limits and explicitly stated they were truncating their answers. ↳ The automated evaluation was flawed, unable to distinguish "cannot solve" from "chooses not to list 10,000 moves." ↳ They tested the models on IMPOSSIBLE puzzles and scored them as failures for not solving them. ↳ A simple change in the prompt (asking for a function instead of a move list) restored high performance. 𝗧𝗵𝗲 𝗯𝗲𝘀𝘁 𝗽𝗮𝗿𝘁? The models KNEW they were hitting the limits. They understood the solution pattern but chose to stop due to practical constraints, a nuance the original study missed. 𝗕𝘂𝘁 𝗵𝗲𝗿𝗲'𝘀 𝘄𝗵𝗲𝗿𝗲 𝗶𝘁 𝗴𝗲𝘁𝘀 𝗿𝗲𝗮𝗹𝗹𝘆 𝗴𝗼𝗼𝗱: Models were scored as FAILURES for not solving mathematically UNSOLVABLE problems. This is like penalizing a calculator for correctly telling you that you can't divide by zero. 𝗣𝗿𝗼𝗽𝗲𝗿 𝗔𝗜 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 𝘀𝗵𝗼𝘂𝗹𝗱: → Distinguish between a model's reasoning capability and its output constraints. → Verify puzzle solvability before running tests on a model. → Use complexity metrics that reflect computational difficulty, not just solution length. → Separate algorithmic understanding from the mechanical task of typing out long answers. No more drawing incorrect conclusions about fundamental capabilities. No more mischaracterizing model behavior. No more overlooking the obvious flaws in the test itself. This is what happens when we test the experiment, not just the model. Instead of finding the limits of AI reasoning, the original study may have just found the limits of its own flawed evaluation framework. The question isn't whether AI can reason. It's whether our tests can. 𝗪𝗮𝗻𝘁 𝘁𝗼 𝗱𝗶𝘃𝗲 𝗱𝗲𝗲𝗽𝗲𝗿 𝗶𝗻𝘁𝗼 𝘁𝗵𝗶𝘀? Check out the paper in the first comment. 𝙊𝙫𝙚𝙧 𝙩𝙤 𝙮𝙤𝙪: What’s the biggest mistake you see people make when evaluating AI? 𝙋.𝙎. I break down cutting-edge AI research like this every week. Your 👍 like and 🔄 repost helps me share more. Don't forget to follow me, Rohit Ghumare, for daily insights where AI Research meets Technology. For founders, builders, and leaders.

Interesting observation! When new AI models learn from the output of older AI models instead of real-world data, they could lose real-world detailing and nuance with each generation. From the paper: "Here we consider what may happen to GPT-{n} once LLMs contribute much of the text found online. We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear. We refer to this effect as ‘model collapse’ and show that it can occur in LLMs as well as in variational autoencoders (VAEs) and Gaussian mixture models (GMMs)."

Elon Musk has recently highlighted a significant challenge in the AI industry: the depletion of human-generated data for training AI systems. Generative AI, which relies on vast quantities of data to function effectively, has already consumed virtually all human-produced content, including the entire internet, accessible books, and publicly available videos. With this finite supply exhausted, AI companies are increasingly turning to synthetic data—artificially generated datasets created by computers—to continue training their models. While synthetic data offers a scalable alternative, it comes with critical drawbacks. Research studies have shown that heavy reliance on synthetic data increases the risk of “hallucinations,” where models generate false or nonsensical outputs. Moreover, this dependence can lead to “model collapse,” a phenomenon where models degrade in quality due to the lack of creativity, diversity, and complexity inherent in human-created data. Despite these concerns, AI companies remain optimistic about the potential of synthetic data. Recent advancements, such as models equipped with built-in fact-checking capabilities like O1, provide hope for mitigating these challenges. However, the debate continues over whether synthetic data can sustain innovation in AI without compromising accuracy and creativity.

NeurIPS 2025 just wrapped, and one paper caught my eye. Jiang et al. ran an extensive empirical study on something many of us have been muttering about for a while - what I've called the "beigeification" of large language models. Their finding is stark: open-ended questions are collapsing to the same narrow set of answers across ALL major models. Take their example: "Write a metaphor about time." This should invite wild exploration. Instead, every model collapsed onto two metaphors: time as a stream, or time as a weaver. Different labs. Different training pipelines. Different architectural tweaks. Same answers. The culprits appear to be: 🔹 Shared underlying training data   🔹 Aggressive RLHF tuning that suppresses outliers   🔹 Overlapping pools of human preference labellers   🔹 And increasingly, LLM-as-judge for scalable evaluation That last one matters most. The study found that LLM-as-judge doesn't value diversity. It rewards the statistically obvious answer - the "safe" one - even on tasks where distinctiveness is the entire point. This is the algorithmic root of AI slop: homogeneous content generated at scale, feeding back into training data, tightening the collapse further. And this is bigger than mode collapse within any single model. It's systemic semantic collapse - the erosion of diversity of meaning itself. From an algorithmic standpoint alone, this is a disaster. Some of the most exciting recent breakthroughs - JEPA, AlphaEvolve - depend on diversification. Evolutionary methods need variation to explore the frontier of what's possible. If the meaning-space collapses, the search space collapses with it. My work sits at the intersection of LLMs and knowledge graphs, so I see this through a particular lens: The way I see it, the only real defence against semantic collapse is for organisations to discover and formalise their ontological cores. Everything inside the general training distribution is being commoditised. What remains valuable is the uncommon - the structured, defensible edge of understanding that only you possess. More than that, every organisation has a unique semantics: a unique way of understanding its world. When you capture that formally - not as branding, but as ontology - you create a semantic boundary around yourself. A structure of meaning that persists even as the external world beigeifies. A strong core creates a strong boundary. Your organisation will need one for what's coming. And if enough organisations do this - using open standards - we end up with a network of networks. A rich, diverse semantic ecosystem rather than a single homogenised hive mind. ⭕ Artificial Hivemind Paper: https://lnkd.in/e9ERs7KB ⭕ Distribution: https://lnkd.in/eRCrEKnt ⭕ Network of Networks: https://lnkd.in/efrpa3q9

✨ NEW Paper from Apple: The Illusion of Thinking in LLMs Apple researchers discuss the strengths and limitations of reasoning models. Apparently, reasoning models "collapse" beyond certain task complexities. Lots of important insights on this one. (bookmark it!) Here are my notes: ⭐ Paper Overview Investigates the capabilities and limitations of frontier Large Reasoning Models (LRMs) like Claude 3.7, DeepSeek-R1, and OpenAI’s o-series by systematically analyzing their performance on reasoning tasks as a function of problem complexity. Rather than relying on conventional math benchmarks, which suffer from contamination and lack structure, the authors evaluate LRMs using four controllable puzzles (Tower of Hanoi, Checker Jumping, River Crossing, and Blocks World) that allow fine-grained complexity scaling and transparent trace analysis. ⭐ Three complexity regimes The study identifies distinct performance phases. In low-complexity tasks, non-thinking LLMs outperform LRMs due to more efficient and direct computation. In medium complexity, reasoning models show an advantage, leveraging longer chain-of-thoughts to correct errors. However, in high complexity, all models, regardless of their reasoning scaffolds, collapse to near-zero accuracy. ⭐ Counterintuitive reasoning collapse Surprisingly, LRMs reduce their reasoning effort (i.e., number of tokens used in thoughts) as problem complexity increases beyond a threshold. This suggests an internal scaling failure not caused by token limits but by intrinsic model behavior. ⭐ Reasoning trace inefficiencies LRMs frequently “overthink” on simple problems, finding correct answers early but continuing to explore incorrect paths. For moderate tasks, they correct late; and for complex ones, they fail to find any valid solution. Position-based accuracy analysis of thoughts reveals systematic shifts in when correct solutions are generated within the trace. ⭐ Failure to execute explicit algorithms Even when supplied with correct pseudocode (e.g., Tower of Hanoi recursion), models still failed at similar complexity points. This indicates that LRMs don’t just struggle to find solutions; they can’t reliably execute logical instructions either. ⭐ Inconsistent behavior across puzzles Models could perform >100 correct steps in Tower of Hanoi (N=10) but fail after 4 steps in River Crossing (N=3), suggesting performance correlates more with training data familiarity than inherent problem complexity. Overall, this paper challenges the assumption that LRMs are progressing steadily toward generalizable reasoning. It argues that existing “thinking” enhancements provide local, not scalable, benefits, raising critical questions about inference-time scaling, symbolic reasoning, and robustness of these emerging systems.