On September 24, 2026, preprint 2609.30063 "Self-Play Pretraining with Zero Data" was released on arXiv — the first proof-of-concept of language model pretraining entirely without human data. The generator and student start from random initialization and are trained exclusively on data that the generator itself creates through programs for a universal Turing machine, adjusting them to the edge of the student's capabilities. On several natural datasets, zero-shot loss predictably decreases with increasing self-play compute. This is currently a small-scale proof-of-concept, but with a reproducible scaling signal worth watching.
What happened
The preprint was prepared by Aditya Kousik, Kfir Dolev, Michael Y. Lee, G. Bruno De Luca, Nurja Cohen, Noah D. Goodman, and Yoav Levin; the first three made equal contributions. In the experiment, neither the generator nor the student is trained on natural data even once: the generator receives an RL reward for creating sequences exactly at the edge of the student's capabilities, which forms an adaptive learning program (curriculum), and the student autoregressively predicts byte sequences generated by programs executed on a universal Turing machine (UTM) using standard cross-entropy. The authors show predictable scaling of zero-shot loss on several natural datasets depending on self-play compute, the emergence of in-context learning, and the models independently finding recognizable mathematical sequences.
Context
The procedure is inspired by Solomonoff induction — a setting in which the search is conducted over the space of all computable generative processes; in this logic, self-play means finding the shortest programs that generate useful training data, and the UTM defines the search space covering all such processes. The honesty of the design is important for measuring transfer: since neither the generator nor the student sees natural data throughout the training, the growth in zero-shot quality on natural datasets is a pure metric of generalization, not a leak from human corpora. This work should be distinguished from Absolute Zero (arXiv 2505.03335, May 2025): there, self-generated RL tasks without data were used only for fine-tuning reasoning, whereas here pretraining is done from scratch — a significantly stronger claim that the proof-of-concept only opens, not closes.
Why this matters for the industry
Pretraining today is limited by the finite volume of human-selected data, and the work proposes a specific mechanism for removing this ceiling: an RL reward to the generator for sequences at the frontier of the student's capabilities ties the volume of useful training data to compute, not to text reserves. For business, this is a hypothesis of dismantling data moats and shifting value from accumulated corpora to compute — currently a hypothesis, not an established fact. For builders, this is an early signal of a shift in constraints: if pretraining without human data scales, custom and niche models for domains and languages without corpora become trainable with compute. If the trend is confirmed, on a horizon of about two years, a complete abandonment of human data is not realistic, but hybrid pretraining is, in which self-play data covers part of the data budget and scarce domains.
Why this matters for users
For the reader, this is a research result, not a product: the work has no weights, API, or ready-made tools. The article is open for reading — PDF and HTML versions are available on arXiv, and Figure 1 with the self-play cycle scheme (generator → programs → byte sequences → student) allows understanding the mechanics without diving into the mathematical part. If pretraining is in your area of interest, the preprint is worth adding to your monitoring: in the coming months, reproductions by independent groups, an increase in parameters and self-play compute, as well as ablations of the generator and reward function are expected. A practically useful marker is whether the predictable scaling of zero-shot loss holds as compute increases by orders of magnitude.
What is still unknown / limitations
The claim that "the volume of useful training data has become a function of compute" cannot yet be considered an established fact: the work shows predictable scaling of zero-shot loss on a small proof-of-concept, and extrapolating the curve beyond the verified compute is an interpretation, not a result. Between the small proof-of-concept and industrial pretraining stand unresolved questions about the efficiency of the RL generator. Key checks are ahead: comparisons with pretraining on curated human data at equal compute and transferring the effect to downstream tasks, not just zero-shot loss. In addition, the preprint is not accompanied by code, weights, or API, and the scale of the experiments is small, so it is premature to talk about industrial applicability.
Sources
- [Self-Play Pretraining with Zero Data — arXiv:2609.30063 [cs.AI] (abstract, authors, submission date)](https://arxiv.org/abs/2609.30063)
- Self-Play Pretraining with Zero Data — article PDF (arXiv)
- Self-Play Pretraining with Zero Data — arXiv HTML version (v1, including Figure 1)
Author
Look at AI, editorial team