
- Date: 2026-10-08
- For graduate students, early-career researchers and research engineers
- Data deadline: 2026-10-08; focus on searching for nearly 12 months, and retain the 2025 study of the disputed starting point. This article is deeply read as evidence, and the model is not run to train or reproduce the experiment, and does not claim to be fully covered.
Abstract: The answer is more correct, not only one explanation
After reinforcement learning, a model can solve more questions the first time it is answered. Can we say that it has obtained new reasoning ability? The key is not to admit or deny progress, but to figure out where the progress is happening: the question that was answered correctly occasionally becomes stable, the question that was originally unsolved can start to be solved, or is the output format more in line with the scorer? These changes have value, but require different evidence.
Four studies were compared. The current relatively safe judgment is that both improving the sampling efficiency and expanding the solvable scope within the limited budget may occur, and the effect depends on the starting point, task and training method; they are not enough to prove a certain internal reasoning mechanism from scratch. This is not to leave half of the space for the positive and negative point of view, but to put the conclusions of different intensity back to the experimental conditions that each can support. Yue et al、ProRL、MATH-Beyond and Intensive Learning in Courses Four different cuts are provided.
Let's do it into three questions
It can be verified reward reinforcement learning, referred to as RLVR, which is to let the model generate candidate solutions, and then update the parameters according to the feedback such as answer check or program test. The reward tells the model which outputs are more worthy of retention, but does not necessarily provide a complete solution. This article refers to the checkpoint before receiving this reinforcement learning as the "starting point model"; it may have been distilled by the instruction fine-tuning or inference, and the word base in the paper cannot be seen to be identified as the original pre-trained model.
The first "will do" is Single Reliability: How likely is it to be correct when you try it randomly. The second is In-budget accessibility: Allows multiple attempts, can you get at least one correct answer. The third is Migrate problem-solving mechanism: Can a method still be used in the face of new structures, conditions to be intervened, or more difficult. These three statements have gradually increased, and the latter cannot be automatically launched by the former one.
Common pass@k before measurement In the case of at least one success in the sub-sampling, the topic will be summarized. It is different from the ability of "acting the correct answer from the candidates": the evaluator knows which answers are verified, and the real user may not know. An intuitive probability model for a single question is:
here is a fixed topic, It is a single success probability under a fixed model, prompt and decoding protocol. is the number of independent and same distribution samples. The formula is only an explanation of indicators, not this measurement; when there are candidates for adaptive search or interdependent, it cannot be applied directly. of different topics It is also different. You can't average the accuracy first, and then substitute the formula into the pass@k of the entire dataset. For the definition of indicators, see Yue et al.§2.2.
There is a fact that is easily overlooked: Even if only the original minimal probability of success is increased, the coverage rate under the high sampling budget may increase. Conversely, continuous failure can only indicate that no success has been observed under the current protocol, and it cannot be put into judged as a strict zero. The "boundary enlargement" mentioned in this article first refers to the observable inner boundary of the budget, rather than mathematically proof that the original model can never generate this answer.
The disagreement of the paper, which come from different settings?
When the starting point is strong, the reward may mainly make the answer more concentrated
Yue et al. compare the pass@k before and after training in mathematics, code and visual reasoning. The paper reports the small The RL model is better and larger The starting point model catches up or even exceeds the phenomenon. Mathematical experiments start with the pre-trained model, and the code and visual parts are not all this setting. §4.1 also distinguish whether to solve the problem before and after training according to the same question, instead of only comparing the average score. Original §3–4.1
This supports the explanation of "in the measured formula, reinforcement learning often makes it easy to get the existing success". The limitation of this article is: the observation of the limited model and the limited budget cannot be upgraded to the upper-limit theorem of the ability of all reinforcement learning algorithms. In particular, it is impossible to rewrite the "problem-solvable set after training is close to the subset of pre-training", and each reasoning process has been proved to exist one by one.
The benefits of extending the training after training are not consistent everywhere
ProRL joins the design of stable training and maintenance exploration, including the KL items of the distribution offset of the constraint policy, the reset reference strategy, and the multi-domain task training. It starts with DeepSeek-R1-Distill-Qwen-1.5B, and §4 divides the results into three categories: coverage decline, earlier platform and continuous growth, and report improvements on training unseen Boxnet tasks. Original §2, §4.2–4.3
Therefore, it should not be asserted that RL always only improves low-budget performance, but also cannot attribute all improvements to training duration: task mixing, exploration control and starting point are different from the previous study. The starting point that has been distilled means that the experiment answered "how much can this checkpoint improve through RL", not "how much can you learn from zero without prior reasoning and supervision". Cross-task migration is useful evidence, and it is still not a direct observation of the new internal algorithm, and it does not rule out the possibility that the extremely low probability solution is amplified.
To find boundaries, the topic itself has to leave space
MATH-Beyond screened out the mathematical question bank that the specified model was still unsolved under 1024 attempts, and provided the model-specific subset, union and intersection. It separates the "newly solved questions" and "the original can be solved and later lost" separately; the evaluation of some RL models shows limited expansion. Original §2–4; ICLR 2026 official version
It is more suitable for the diagnosis of difficult cases, but it is not an unsifted daily distribution of math problems. The expansion rate of the exclusive set of a model is different from the expansion rate on the other set of another model, and the denominator is already different, and it cannot be directly ranked in the ability list. This article also recommends using a new sampling batch to retest the starting point model: If the question of zero success during screening, there may be rare success during the retest; this will affect the attribution of "newly", and do not necessarily mean that either party is fake.
Teacher guidance is effective, it does not mean that pure rewards are discovered alone
A 2026 preprint on curriculum reinforcement learning first probes the limits of what the model can currently solve, then uses a small number of teacher reasoning traces for supervised fine-tuning and consolidates the gains with RL. The authors report improved pass@256 on math evaluations. They also note that under group-relative rewards, if every candidate in a group fails, the reward-driven advantage terms may all be zero. Original §3–4
This gives a clear learning signal problem, but the improvement comes from a combinatorial process that contains teacher information. If you want to answer "what the RL has contributed", you also need to control the teacher's content and budget, and compare the teacher's fine-tuning, only RL and combination training. All failure groups do not have this signal, nor does it mean that the entire model will never progress: the parameter updates, regular items and migration of other questions may still change it.
| Research angles | critical condition | observation or claim | cannot be proved accordingly |
|---|---|---|---|
| Yue et al | Various starting points with existing RL formulas | Small budget improvements can be accompanied by coverage contraction | RL can never be expanded |
| ProRL | Distillation starting point, multi-tasking, extended training | Shrinkage, platform and growth coexist | Duration is the only reason |
| MATH-Beyond | Difficult to screen out by model failure | Success in quarantine budget | Overall Ability on Universal Tasks |
| Course RL | Faculty fine-tuning alternates with RL | Combined Process Expanded Observation Coverage | New ability to create without external information |
Each row in the table corresponds to the above original text, and the amplitude cannot be compared horizontally. For the first three items, there are formal meeting minutes; the course RL only confirms the arXiv preprint in this verification.
What really needs to be aligned is a complete measurement

The diagram illustrates the evaluation proposed here. It depicts neither data from the paper nor an established causal mechanism. For each pair of marks, the left gives the starting model’s outcome and the right the RL model’s. The diagram shows the three categories in which at least one model succeeds. The fourth, where both fail, must also be recorded. “RL ONLY” means that only the trained model succeeded within the given budget. That observation alone cannot establish a new reasoning mechanism.
The same number of samples does not guarantee the same calculation cost. The general evaluation limit of Yue et al. is 16384 tokens, and the evaluation of ProRL is set to 32K; even if both draw pass@k, it is not a strict and other budget comparison such as cross-paper. Yue et al§3、ProRL§3.4. The budget for training data, teacher generation, and trial recipe selection consumption should also be reported separately from the cost generated during the test.
The prompt template, temperature and answer extractor should also freeze. Otherwise, a model may just be more familiar with the output format, or get more suitable decoding parameters for you. The common agreement is convenient for explaining the cause and effect, and the tuning protocols are easy to compare their respective best performances; both experiments can be done, but the tuning budget and goals must be clearly written, and they cannot be mixed into the same conclusion.
Comparability also includes data identity. It is necessary to distinguish between training questions and formula selection verification questions and final test questions, and check whether the questions and the near-repetitive solution are crossed. ProRL shows that the mixed Validation Set monitoring training has been used from the evaluation benchmarks; this article does not refer to all report benefits as completely independent confirmation results. Original §3.3. This is an interpreted limitation and does not mean that data leakage has been proven.
Finally, there is still a distance between the correct answer and the correct reasoning. The math problem may guess the last number, and the program may only pass an inadequate test. MATH-Beyond also shows the misjudgment caused by the answer resolution rules. Original §3.1. If you advocate a new problem-solving method, you need to check the key steps, supplement tests or controlled variants; just relying on smooth text explanations is not enough to complete this step.
In the next experiment, how to make the two explanations give different predictions?
The following is the design of the follow-up research, which has not yet been implemented. First, fix a starting point checkpoint and a set of independent questions, and keep the complete candidates, score records and generate costs. Use a common sampling budget grid before and after training, and do the same comparison of the same total generated token budget, respectively report reliability, new coverage and loss of coverage, to avoid only picking one that is beneficial to you. .
Test 1: Does the income mainly come from the redistribution probability? If the low budget gap is obvious, the high budget gap gradually disappears, and the new question is also solved by the starting point model after the independent resampling, this explanation is more supported. If the high budget gap continues, the evidence will support the widening of practical reach; but the original probability cannot be asserted by limited failure alone.
Test 2: What training information does the extension depend on? Set the condition that only do RL, only use the teacher track Fine-Tuning, and combine the two. Keep the teacher's questions consistent with the trajectory, and record the number of teacher calls, training tokens and updates. If the combination is better than the teacher-only condition, it can support RL to have an additional effect on the consolidation or migration; if it is only improved on the template covered by the teacher, the conclusion should be narrowed.
Test 3: Can the new success be migrated to the solution structure? The newly solved question type makes a pre-determined variant, changes the surface wording, irrelevant clues and combination structure, and determines the score by independent verification method. If you improve the span-variable maintenance and the key steps are effective, it is closer to the proposition of "learning the reusable method". The different interpretation texts are not mechanism evidence in themselves, and should not be pretended to be artificial semantic gold labels by the assistant who generates the article.
Before the experiment, the upper and negative results should also be agreed upon to stipulate the upper limit and stop rules. If the budget is exhausted, the two explanations cannot be distinguished, and the conclusion is that the current measurement resolution is insufficient; no additional sampling until the expected answer occurs, and the question that is still open is not written into an algorithm.
Summary: Write "new ability" as a testable proposition
The most useful position for the reader is the hierarchical report: a single accuracy Improvement, the description is more reliable; the number of new additions and budgets of the same topic has increased successfully, indicating that the observed changes in the reach; after controlling the training information, there is still a cross-structural migration, and the learning of reusable methods is further supported. Every step forward, new evidence is needed.
Therefore, when reading "RL Creation Ability" or "RL will only sample", four limits are added: relative to which starting point, which type of task, how much budget is used, and what verification is used. The value of the dispute is not to choose a slogan, but to design the next experiment to be specific enough so that the competitive explanation has the opportunity to be distinguished.
References
- Yang Yue、Zhiqi Chen、Rui Lu、Andrew Zhao、Zhaokai Wang、Yang Yue、Shiji Song、Gao Huang. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? 2025,NeurIPS 2025. Published paper; The main targeting version of this article: arXiv v5. The two Yang Yue are different authors.
- Mingjie Liu、Shizhe Diao、Ximing Lu、Jian Hu、Xin Dong、Yejin Choi、Jan Kautz、Yi Dong. ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models. 2025,NeurIPS 2025. Conference page; Published paper; arXiv v1. This article discusses the first generation ProRL, which does not mix the number of subsequent versions.
- Prasanna Mayilvahanan、Ricardo Dominguez-Olmedo、Thaddäus Wiedemer、Wieland Brendel. MATH-Beyond: A Benchmark for RL to Expand Beyond the Base Model. 2025 preprint, ICLR 2026. Conference page; Published paper; arXiv v1.
- Pengxiang Cai、Tianchen Fang、Xiaohan Li、Qingyuan Zeng、Guocong Li、Jintai Chen. Curriculum Reinforcement Learning Can Incentivize Reasoning Capacity in LLMs Beyond the Base Model. 2026, arXiv preprint V1, the first submission on 2026-06-21. Original; Version history. As of the search date, the employment information of the official meeting has not been verified.