Cover illustration

  • Date: 2026-10-19
  • For graduate students, early-career researchers and research engineers
  • Data deadline: 2026-10-19; the actual search has been working for nearly 12 months, and the classical research required to explain the problem is reserved. This article is deeply read as evidence, without executing model training or reproducing experiments, and does not claim to be fully covered.

Summary: Who is writing textbooks and who is deciding right and wrong?

A model to ask questions, answer them, and then use their own answers for training. This closed loop doesn't seem to need a teacher anymore. But generating training materials and determining which materials are worth learning are two different tasks. Teachers may no longer write questions one by one, but still participate in the form of standard answers, scoring specifications, program execution results or development sets.

The judgment of this paper is: self-improvement can reduce continuous manual annotation, and may also improve the ability to transfer structure; whether it depends on external feedback, it is necessary to explain which stage and which information it refers to."Self-generated data", "self-scoring" and "starting from random initialization" are not the same intensity claims. STaR, self-reward model, Absolute Zero and recent zero-data pre-training research show different closed loops, and do not constitute two direct oppositions of "self-study" and "can't self-study". STaR; Self-Rewarding; AZR; Self-Play Pretraining.

First draw the feedback to the position of the system

The "self-improvement" here refers to the experience of model generation or screening, and then changes the subsequent ability through parameter updates; rewriting the answer in a single conversation belongs to another type of problem. validator It is a mechanism for judging signals for candidate answers, which can be a model, or a rule or execution environment. Course It is the distribution and difficulty arrangement of training tasks, and does not specifically refer to artificially compiled textbooks.

Might as well break the external dependencies into four layers. The starting point layer includes pre-training weights and seed demonstrations; the training layer includes answers, rewards and environmental observations in each round; A system can no longer request human tags at the training layer, and continue to rely on several other layers at the same time. Combining these four layers into a "zero external information" will make the most valuable contribution of the paper difficult to identify.

Self-improvement feedback sources and evaluation boundaries

Figure: This is the reading framework of this article, not the measured cause and effect diagram of any single paper. The solid line represents the training information path that needs to be tracked, and the orange dotted line represents the possible development selection feedback; each method does not necessarily use all entrances. The test on the right side should only accept the model and not reverse the parameter adjustment. This is the design requirement of the subsequent experiment, and it cannot be claimed that the papers read have been met.

Three differentiable explanations can be proposed from this. first, Reorganize the ability: The model can occasionally give good answers, screening and training to make it multiplexed. second, Explore new experiences with executable rules: No one gives the answers one by one, but the environment can distinguish the candidates. third, Adapt to your own referee or development indicators: The internal score has improved, and the independent tasks may not be improved synchronously. The three can coexist; the purpose of subsequent experiments is to identify their respective contributions, not just use the prefix of "self".

Four closed loops, four evidence boundaries

STaR: Write the process yourself, the answer is still provided by the dataset

STaR starts with pre-training GPT-J and a small number of inference demonstrations. The model generates the problem-solving process, and the final answer matches the standard answer before entering the Training Set; when the answer is wrong, the correct answer will be given as a prompt, and the process will be regenerated, and the prompt will be removed during training. The original model of the new material Fine-Tuning is used for each round, and it is not always continued on the weight of the previous round. Original §3, Algorithm 1.

The paper improves on arithmetic, common sense questions and mathematical applications; its main contribution is to reduce the demand for gradual reasoning and labeling. It does not cancel the result supervision: just look at the "process is generated by the model", and the two entrances of screening and answer prompts will be missed. The final answer is not guaranteed to be correct in every step; the failure case of the paper retains this. §4. §4.4 Failure cases.

This shows that when reviewing a self-training paper, you should separate the "how many manual processes required" and "how many known answers are needed". The reduction of process supervision is a substantial progress; writing it as unsupervised discovery changes the problem of being tested by experiments.

Self-rewarding model: Referees can make progress together, but do not appear out of thin air

Self-Rewarding starts with Llama 2 70B, and trains seeds with Open Assistant's human demonstration and sorting. Subsequently, the same model generates candidate answers and scores according to the specification, and directly prefers to optimize with preference, that is, DPO. The evaluation demonstration is part of the main formula, and the paper also examines variants that do not add it, so it cannot be said that it is a necessary condition for all Self-Rewarding methods. §2—3.1, Appendix A.3.

The consistency of the instruction evaluation and the reserved human preferences has been improved, but only three rounds of the model settings have been examined; some reward evaluation indicators are not increased by round, and the answer length has also increased. The authors clearly discuss the limitations of model referees and reward speculation. The problem generator of the main experiment is still pre-fixed, and each link cannot be described as co-evolution. §2.2 Footnote, Table 4, §6.

The more direct development layer dependence is: the early stop selection of the training checkpoint uses the Claude 2 evaluation verification sample; the main experimental topic is generated by the fixed Llama 2-chat. Therefore, "self-reward" accurately describes the source of training scoring for candidate answers, and cannot be expanded to only rely on one model for the whole process. §3.1.3.

Based on this, it is believed that "the same model as a referee" does not automatically invalidate the feedback: the answer and comparison candidates may have different difficulties. The real guarantee that the referee becomes more confident is whether it is still calibrated to the target task. It is also not possible to improve the extrapolation of infinite recursive growth from three rounds of improvement.

Absolute Zero: After removing the question bank, the execution environment is still there

AZR allows the pre-trained model to issue and solve problems at the same time, construct tasks around the program, input, and output, and Python performs verification and generates rewards. The course will bias the model that has not yet fully grasped and has the opportunity to succeed. What is omitted here is the artificially sorted post-training question bank, and the pre-training ability and executable semantics have not disappeared. §2—3, Algorithm 1.

The paper reports improved mathematical and code evaluations, but different starting models, different benefits, and individual test items decline; the data and starting points of other methods in the table are not uniform. It supports bootstrapping training in a specific environment, and does not prove that all tasks can make reliable answers on their own. Program screening is also approximate: the checks are consistent with limited repetitions, and it is not possible to prove that all inputs are determined. §3.3.3, §4, Table 1.

From the boundaries of the parameter model, the actuator provides external feedback; from the entire system containing the executor, it is an internal component. This is not a word game: the former asks whether the model can judge alone, and the latter asks whether the system can get rid of manual supervision. Both research objectives are reasonable, but the evidence cannot be mixed.

Zero data pre-training: there is no natural data gradient, there are still development options

The Self-Play Pretraining of 2026 allows both the generator and the learner to start randomly: the former generates an executable program, and the latter predicts the bytes output by the program. Generator rewards are based on the relationship between the learner gradient and historical parameters, rather than the correct answer to natural language. The paper reports the improvement of predicted loss on natural data; fixed program priors and specialized grammar generation have different baseline performances, and self-games are not the best in all fields. §2—3.

Section 3.1 gives an important qualification. Hyperparameter selection used validation losses on DCLM text and DNA, a limited information leak that the authors explicitly disclose. Natural data influenced the choice of training recipe even though they supplied no gradient updates. The reported compute frontier also spans different model sizes, checkpoints, and ensemble sizes. It cannot establish that a single model improves indefinitely along one training trajectory. §3.1 Scaling recipe.

Therefore, this preprint provides a powerful incision for the migration of the program structure, but predicting the next byte is different from grasping the world facts. Its discussion also retains this boundary: the fact of the specific world still needs to be obtained through contact with the world. §6.

Which comparisons can be done and which cannot be done?

Research Start and task Supported observations cannot be launched accordingly
STaR Pre-training models, questions with answers Self-generated processes can be used to improve No standard answer is required
Self-Rewarding Pre-training and human seeds, command preferences The self-scoring training for limited rounds is effective Self-assessment is always consistent with the facts
AZR Pre-training model, code environment From the question to form an effective post-training experience No pre-training, no environment is also established
Self-Play Pretraining Random initialization, program byte prediction Synthetic structures can be migrated to predictive tasks The entire development process has no contact with natural data

This table summarizes the aforementioned evidence and cannot be used for ranking. The indicators of the four studies involved answering the question Accuracy, preferences, code and mathematical tests, and byte prediction losses, respectively; there is no common budget for training samples, execution calls, generation length and integration costs. Even if all the curves rise, it is not four measurements of the same "smart growth".

It is more useful to the reader to distinguish two questions: Is this recipe useful?, and Whether the proceeds are caused by the claimed mechanism. The former needs to be tested independently; the latter also needs to match the starting point, training volume and reasoning budget, and intervene in a certain feedback source. If the number of samples drops sharply after removing the validator, the performance degradation cannot be attributed to the label quality; if the referee is only frozen but the number of candidates is increased, the role of the referee to update cannot be isolated.

A follow-up experiment to distinguish competing explanations

The following are research design recommendations, which have not been implemented yet. You can first choose a small-scale task with independent criterion, fixed a candidate pool at the same starting point, compare random screening, freeze model referees, update referees with rounds and perform criterion screening. Try to match the reserved sample number, difficulty and training token; if it cannot be exactly matched, report it separately, and add a subset control.

In the first stage, a fixed candidate pool is used to test the value of the screening information; in the second stage, the questioner is allowed to update, and the adaptive course is compared with the resampling from the same historical question pool. This avoids mixing "good referees" and "more data" and "better courses" into a change. The candidate pool itself should also record the generation cost, not just the learner to calculate the bill.

The evaluation also retains the new task of the original task, the new task of the combination method, and the counter-example that is easy to make the referee misjudgment. Each round records the actual execution correctness, self-score and coverage, and then analyzes when they are separated. If the self-scoring rises, the interpretation of the adaptation to the referee is more credible; if the independent correctness of the fixed candidate is improved, the screening has been supported by successful explanations; These results still cannot prove the birth of an internal mechanism alone.

The hyperparameters and stop rules are first determined in the development task, and then frozen to the test task that is not used for selection. In particular, it is necessary to record "update weights without test samples" and "researchers did not use test feedback through repeated trials" separately. If the actual budget is not enough to complete this step, the migration conclusion should be narrowed instead of renamed the development set to test set .

Summary: The boundaries of self-learning are in the feedback ledger

Models can now take on more of the work of generating training materials, screening candidates, and organizing a curriculum. Those advances can be tested. Stronger claims require accounting for where the information comes from. A process that adds no new human labels can still rely on earlier human input. One without a human-written question bank can still use an environment’s correctness criterion. Even if natural data contribute no gradients, they may still guide development choices.

When reading the next paper "The model teaches yourself", you first track the source of information generation, judgment, update and evaluation, and then ask what kind of abilities the benefits fall on. The most worthy of the dispute is which feedback can be replaced and which dependencies still determine the boundaries, rather than compressing the entire system into a sentence "with or without a teacher".

References

  1. Eric Zelikman、Yuhuai Wu、Jesse Mu、Noah D. Goodman(2022). STaR: Bootstrapping Reasoning With Reasoning. The official full text of NeurIPS 2022.
  2. Weizhe Yuan et al. (2024). Self-Rewarding Language Models. ICML 2024; Use of Methods and Restriction Verification arXiv v3,2025-03-28, do not pretend to be the original meeting of the conference.
  3. Andrew Zhao et al. (2025). Absolute Zero: Reinforced Self-play Reasoning with Zero Data. NeurIPS 2025; method verification use arXiv v3,2025-10-16. There is a difference between the official page and the arXiv author list, and save it in the source record respectively.
  4. Aditya Cowsik、Kfir Dolev、Michael Y. Li、G. Bruno De Luca、Nourya Cohen、Noah D. Goodman、Yoav Levine(2026). Self-Play Pretraining with Zero Data. arXiv V1, 2026-09-24; this time has not been verified for the official employment, and is quoted according to the pre-print.