
- Date: 2026-10-12
- For graduate students, early-career researchers and research engineers
- Deadline: 2026-10-12. Focus on retrieving the work of the past 12 months, and retain the starting point of the dispute in 2025; this article is a deep reading of the evidence of three first-hand studies, no model training, and does not claim to be fully covered.
Abstract: The name of the learning method cannot replace the experimental conditions
Give the model a standard answer, and let the model receive feedback after trying, why does it have different migration performance? "SFT memory, RL generalization" gives an easy-to-remember answer, and it is easy for people to ignore: whether the two groups of models have seen the same rich solutions, whether they are in the same training stage, and whether they have accepted the same test.
The judgment of this paper is: reinforcement learning shows stronger migration on some controlled tasks, but the existing evidence is not enough to attribute the generalization ability to a certain training goal. After data selection, supervised content and optimization degree change, SFT can also have cross-distribution benefits. This does not mean that the two methods are equivalent; it means that the "observed gap" and "explained source of the gap" need to be separated. Chu et al., 2025、Lu et al., 2026、Ren et al., 2026 Three incisions are provided for task comparison, data intervention and training dynamics.
"Generalization" must first explain what to change
Supervising fine-tuning (SFT) enables the model to increase the probability of the next tokens in the demonstration when given the problem and demonstration prefix. Reinforcement Learning (RL) adjusts the generation strategy according to the rewards obtained from the generated results. Here we mainly discuss RLs in verifiable tasks, and do not include all conclusions of human preference optimization.
Intuitively, SFT is like reading the process of reading, and RL is like getting feedback after handing in the paper. However, the metaphor cannot judge for us: the textbook can contain multiple solutions, and the feedback may only allow students to repeat the most familiar path. "Explore" describes the training process, and does not automatically guarantee that a transferable method is found.
Intra-distribution (ID) tests follow the main conditions of training tasks; out-of-distribution (OOD) tests change some of these conditions. Change the card rules, change the image style, and switch from math questions to code questions, all of which are distributed changes, but not the same ability. What's more, OOD is usually only relative to this post-training data, and it cannot be asserted that pre-training has never included similar knowledge.
This article separates the "Memory Training Rules", "Retain the original ability" and "Get migration benefits". After changing the rules, the performance declines, and only the adaptability is damaged first. To further claim to memorize verbatim, additional measurements such as retelling need to be approximated. In turn, the migration performance is higher than before the training, and it is not enough to locate what algorithm is formed inside. This is the standard of evidence used in this paper, not an observation report on the internal state of the model.
The earliest strong evidence also has a clear scope of application
Chu et al. change rules or visual input in GeneralPoints card operation and V-IRL navigation. For example, j, q, and k are all interpreted as 10 during training, and 11, 12, and 13 are changed during the test, and the new rules are notified in the prompts. The model needs to be re-acted according to the current instruction, rather than the training map. The paper reports that RL improves under these OOD conditions, and continues SFT to degenerate. Original §4, §5.1—5.2
This control is not simply to give RL more training: two start from the shared SFT initialization checkpoint, and Figure 6 compares the estimated calculation of the same position. The backbone is Llama-3.2-Vision-11B; RL allows the use of validators to feedback by rounds of correction, while the main text SFT mainly uses single-wheel expert demonstrations. The appendix also tests the SFT track with errors and verification information, and the local results are still not ideal. Figure 6. Appendix C.1—C.3
Therefore, the observation cannot be erased with "no fair comparison at all", nor can it be upgraded to "SFT is essentially memorized in essence". The calculation amount is similar does not make the feedback information, trajectory distribution and the objective function automatically the same. The paper also reports that RL cannot restore OOD performance of an excessive SFT checkpoint. §6 This constitutes a boundary for the statement that "just add RL to generalize".
Which samples may the gap really come from really pushing for updates
Lu et al. to investigate the classification and positioning tasks of the VISION-LANGUAGE MODEL, and it is difficult to divide according to the results of multiple answers in the initialization model: it is easy to all correct, all errors are difficult, and there are right and wrong as medium. The "difficult" here is the observation label relative to the model and sampling protocol, and is not an eternal attribute of the topic. Original §4.1—4.2
The proposed explanation draws on group relative policy optimization (GRPO), which compares rewards for candidate answers to the same question. When all candidates in a group receive the same reward, their relative advantages are zero and that reward contributes no policy update signal. Other terms can still contribute gradients, including a KL penalty that limits how far the model moves from its reference. The authors describe the effect as implicit filtering by difficulty and test that explanation with SFT using explicit filtering. §3—4
In the main settings of the paper, DC-SFT retains medium difficulty, or at the same time retains easy and medium difficulty data, and its average OOD performance exceeds that of the control GRPO. The research uses the two scales of Qwen2.5-VL, first to initialize the shared format, and then mainly carry out low-rank fine-tuning; compared with the number of fixed training steps, the batch setting of SFT and GRPO is different, and it cannot be directly called iso-equipped power experiment. Table 2, §5.1—5.2
This intervention shows that at least a portion of the gap can be changed by changing the data of the SFT contact. It has not ruled out other common changes, nor has it proved that all RL benefits are caused by difficulty screening. In particular, a static screening is not the same as a continuous online sampling; the questions that are not in the early stage may become effective training questions in the later stage. Reading the paper as "difficult samples is always harmful" will turn conditional discovery into a new slogan.
A training curve can also change the judgment of "ungeneralized"
Ren et al. Started from the pre-training checkpoint of the command FINE-TUNING, and carried out SFT with the long-thinking-thinking (CoT) mathematical demonstration proved by the answer. CoT refers to the intermediate inference text that is expanded in the output. In some cross-domain tasks, the paper observes the trajectory of first decline and then recovery: if only the earlier checkpoint is seen, it is easy to underestimate the final migration performance. Original §2, §3.1, Figure 3
This is not "the more training the better". The authors observed late degradation with a more aggressive learning rate and training arrangement, and also found that data quality and model starting point affect recovery. The subsequent GRPO comparison matches the starting point, problem pool, batch, response upper limit and number of updates, but it is clear that there is no matching total generated tokens and total floating-point operations; the results of SFT dominant cannot establish a general ranking. §3.4, Appendix A and C.14
My inference is that attributing a difference to an algorithm requires looking across the learning trajectory. A single training checkpoint can mislead. Even a later recovery in performance cannot establish that the model learned to backtrack or verify its work. The paper’s detector identifies observable patterns in the text, and the authors explicitly acknowledge that this behavioral proxy supplies no parameter-level mechanistic explanation. Appendix A
Put the three interpretations into the same comparison frame

Figure: The comparison protocol proposed in this paper. The arrows indicate that the same set of tests are trained and received in the same set after sharing the starting point; Data and budgets are conditions that need to be checked. There is no winning method or measured trend in the figure; the "exploration, screening, and optimization stage" still needs to be distinguished.
| Explanation | Distinguished Observation | Existing evidence and boundaries |
|---|---|---|
| Feedback exploration is more favorable | Under the same data coverage and budget, online feedback still has additional benefits | Chu et al. support local RL advantages, but there are still differences in training signals |
| Effective data distribution is different | Keep the SFT goal, and the gap narrows after changing the sample selection | Lu et al. provide evidence of intervention and cannot explain all RL mechanisms |
| Compare different training stages | The same method later checkpoint changes the original judgment | Ren et al. observed recovery, and also observed degradation under radical training. |
These three explanations can be established at the same time. They are concerned with "how to gain experience", "what experience to enter the update", and "when to measure the results", and they are not three mutual exclusion camps. The tasks, models, training data and evaluation indicators of the three papers are all different, so this article only compares the structure of evidence, and does not put their list numbers into rankings.
When reading the table, you should also distinguish the ledger of the budget. There is no limit to the number of attempts per question; the number of the same gradient steps does not limit how many tokens are processed per step; the same training tokens is not included in the teacher generation, difficulty screening and online verification. For the research objectives, "the training cost of the existing demonstration time" and "the total cost of data from scratch" can be reported separately, but the two cannot be mixed to declare the outcome of efficiency.
There is no ruler to make all variables equal at the same time. If you study "who is more effective under the same computing power", you are allowed to use different quantities of experience, but you must settle the calculation cost honestly; if you study "how the same experience is used", try to share the trajectory as much as possible, and then accept the cost difference caused by the optimization method. The former is a resource allocation problem, and the latter is closer to the learning mechanism problem. The paper can answer one of them, but should not wrap one answer into another. In particular, the knowledge of additional teachers of SFT cannot be given first, and then all the improvements are attributed to the supervisory goal; also the RL cannot be allowed to repeatedly access the validator, but the source of information is denoted as zero.
Evaluation also needs to be dismantled. The average accuracy of multiple answers, and the ratio of at least one success to answer different questions in multiple times. Several reasoning indicators such as Ren et al. use the former; the card calculation success of Chu et al. includes the AT Inference time verification opportunity. Ren et al. § 2.2, Appendix B.4、Chu et al. Appendix C.3 It also says "success rate", and the decoding and feedback protocol cannot be ignored.
A follow-up experiment to distinguish competing explanations
The following are the experimental recommendations, which have not been implemented yet. Select a task that can automatically check and change the rules, fix the starting point, training problem pool and independent development set, and reserve the new rule test set that does not participate in the adjustment of parameters in advance. First compare the ordinary demonstration SFT, difficulty screening SFT and online RL, and then add a set of SFTs, and use the RL to actually generate and pass the verification trajectory. This group can be checked whether the part of the proceeds obtained by RL can be demonstrated and delivered.
This is still not a one that can be solved in one go. The generation cost of the RL trajectory should be recorded in the data ledger that is supplied to this group of SFTs; if the trained RL model is used as a teacher, the conclusion can only be "whether these experiences can be passed", and cannot claim that RL is not needed. Random screening groups with the same number and approximate length should also be retained, so as not to reduce the number of samples or shorten the output as a difficulty screening.
Pre-fix several budget nodes, and record training tokens, to generate tokens online, and validation times and time; in the development collection checkpoint, the final test is only used for reports. In addition to the total ACCURACY, you should also separate statistical format failures, continue to apply old rules and execute new rules. This is where you know where the score changes come from. All methods use the same decoding protocol and report fluctuations in multiple independent training.
If only the SFT data is changed to stabilize the gap, the data interpretation will be strengthened; if the online RL still benefits additionally when the coverage is similar to the budget, the feedback explanation will be strengthened; if the method ranking changes with the checkpoint, the training dynamic should be given priority to report. On the contrary, if a certain operation does not work, we can only rule out the interpretation under which one implementation, and the negative result cannot be upgraded to an algorithm impossible.
Summary: Restore method labels to verifiable problems
Existing research allows us to say that RL is sometimes more adaptable to change than some kind of SFT configuration; they also clearly suggest that the migration performance of SFT will change with data and training conditions. For newcomers to scientific research, it is not worth asking which method to stand in line, but: which controls still exist, which explanations have been weakened, and which variables are still entangled.
"Memory, exploration, generalization" should be the object waiting to be measured. Only when the starting point, experience, budget and test distribution are clearly explained, the strong judgment in the title of the paper can be transformed into a research conclusion that can be accumulated and can be revised.
References
- Tianzhe Chu, Yuexiang Zhai, Jihan Yang, etc., SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training,2025. ICML 2025 Official Records; Method and result verification use arXiv v2,2025-05-26. The official PDF has not been successfully opened this time, and the two versions are not claimed to be checked word by word.
- Aojun Lu、Tao Feng、Hangjie Yuan、Wei Li、Yanan Sun, Why Does RL Generalize Better Than SFT? A Data-Centric Perspective on VLM Post-Training,2026. arXiv V1, 2026-02-11 full text. retrieved CVPR 2026 official PDF record, but this direct opening failed; the experimental judgment is only based on the read arXiv version, and the final draft is not presumed to be completely consistent.
- Qihan Ren, Peng Wang, Ruikun Cai, etc., Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capability,2026. arXiv V2, 2026-08-15 full text; Version and Status. The author is in arXiv and Official code base The statement was received by COLM 2026; the official entry of the independent meeting has not been obtained this time.