Training on Your Own Output

In one line: a model trained on data its predecessor generated loses the rare cases first, and the loss is invisible on average-case metrics until the distribution has already narrowed.

Once a loop is running, its own outputs are the cheapest training data available. They are on-distribution, plentiful, already formatted, and free. Using them is not automatically wrong, but doing it naively has a measured failure mode.

What happens, and in what order

Training a generative model repeatedly on data produced by the previous generation causes the tails of the distribution to disappear first, variance to shrink, and errors to compound across generations until the model converges on a narrow region of what it started with. The effect has been demonstrated across language models and simpler generative families alike, which tells you it is a property of the recursion rather than of any architecture.

The ordering is the useful part. Rare events are rare in any finite sample, so each generation slightly under-represents them; the next generation trains on that under-representation and produces even fewer. The tail thins geometrically while the mean holds steady, which is why aggregate evaluation misses it completely and a per-segment view catches it.

Replacing versus accumulating

The result that makes this actionable: collapse follows from replacing real data with synthetic data each generation. When data accumulate — the real corpus is kept and synthetic data is added alongside it — test error has a finite bound independent of the number of iterations, and collapse does not occur. This has been shown across language, image and molecular settings with an accompanying proof.

So the design rule is simple to state and easy to get wrong in an operational way: never let a data pipeline silently substitute generated data for the real corpus. Keep the real data, mark provenance on every record, and hold the synthetic share as a stated fraction rather than whatever a retention policy happens to leave behind.

Where a loop eats itself without meaning to

The obvious version — fine-tuning on the system's own generations — is easy to spot. The quiet versions are worse:

• Demonstrations bootstrapped from the system's own successful traces, refreshed each round from a system that was itself built from the last round's demonstrations. Several generations in, the examples describe the system's habits rather than the task.
• A judge distilled from a larger judge, then used to select data that trains its own replacement.
• Production traffic that is already partly generated by the system — support replies, code, summaries — flowing back in as "real" data because nothing recorded where it came from.
• Retrieval corpora containing documents the system wrote earlier, so the retriever confirms the system's previous outputs.

Every one is prevented by the same discipline: a provenance field on every record, and a policy that mixes by provenance rather than by recency.

When it is safe, and why

None of this makes synthetic data unusable. Two conditions separate the cases that work from the ones that collapse.

The first is a filter with an outside signal. Rejection sampling on generated data against an independent check — a test suite that runs, a verifier, a human label on a sample — is not recursion in the harmful sense, because the selection is being made by something that is not the model. What collapses is generation feeding training with the model as its own arbiter.

The second is the mixing ratio, held as a stated fraction with the real corpus retained rather than as whatever survives a retention policy. The failure in practice is almost never a decision to train on synthetic data. It is a pipeline where old records expire, new ones arrive partly generated, and the ratio drifts with nobody's approval.

Detecting it

Track output diversity explicitly — distinct n-grams, embedding spread across responses, entropy of chosen actions — as a first-class metric rather than a curiosity. Evaluate on a rare-case slice held for the purpose. And compare the current generation against a stored generation from several rounds ago on the same inputs, which turns a slow drift into a visible diff.

Next: what makes an automated promotion safe.