
- Date: 2026-10-11
- For graduate students, early-career researchers and research engineers
- Deadline: 2026-10-11; Search for work in the past 12 months and keep the classic control. This article is a deep reading of the literature, no model training or reproducing experiments are run, and it does not claim to be fully covered.
Summary: First ask who has improved
When correcting math homework, only check the final answer, you may miss the mistakes in the middle; it takes more time and professional judgment to check it by line. Moving this to the large model will come to an alluring conclusion: the finer the supervision, the more reliable the reasoning is. But the "better" in the experiment may just be more accurate from the existing answers, or it may be that the validator will find the wrong one, or the generative model itself will become more stable. These three progresses require different evidence.
The judgment of this article is that the process label can provide a more clear wrong position, but its benefits depend on the meaning of the label, how it is used and the evaluation target; the result tag may also train a useful process scorer. What is really worth comparing is not who wins the two abbreviations, but which link of the newly added information and calculation has improved. The following four studies are used to disassemble these problems, and the results under different settings are not combined into camp confrontation.
"Processes" have at least three positions
First is Supervised source: Does the label say only whether the final answer is correct, or does it indicate whether an intermediate step is valid? Process reward model is usually abbreviated PRM, and outcome reward model ORM. Usage varies across papers, so check what the model was actually trained to predict.
Secondly score position: The model can output scores at each step, but these scores do not necessarily indicate that the logic of this step is correct. If the tag comes from "continue to generate from here, can you answer correctly in the end", it is closer to the potential for future success. The result supervised model may also read the complete solution instead of just seeing the last number. Uesato et al.§2.4、Zhang et al§3.1.1
Finally use location: Use the scorer for the candidate reranking, the change is the selected answer; use it as a reinforcement learning reward, and the generation strategy will be updated. "The reward model is better trained" and "the model after training with the reward" is not the same experiment. In this paper, the model that actually generates the solution is called the generator, and the model that checks the candidates is called the validator.
Best-of-N refers to the fact that the gentleman has multiple candidates, and then selects the one with the highest rating. It tests the selection ability; wrong positioning requires pointing out the earliest error position, or determining that the entire answer is correct. Both have value, but cannot guarantee each other with the same total score.

The solid line in the figure represents the training or use of the path, which does not indicate that there must be benefits; the dotted line reminds the "update generation strategy" that needs to be verified separately, not a natural extension of the existing rearrangement results. The bottom three items are independent evaluation questions, and the hook fork is only used as an icon, not a measurement result. The track inspection device on the cover is also a conceptual metaphor, not a paper experimental device.
The classic advantage is established, but let's see where the experiment is moving first
"Let's Verify Step by Step" by Lightman et al. compares process supervision and result supervision on MATH problems. Its large-scale experiment starts with GPT-4, fixes the generator, and uses different validators to select candidates; the paper clearly states that there is no use of reinforcement learning to improve the generator. In the official version 3 report, under the condition of 1860 candidates for each question, the answer selected by the PRM is 78.2%, and the ORM is 72.4%; the evaluation adopts the reserved 500 questions. Official version §2.1, Figure 3, Appendix C
This first supports The validator of process supervision under this condition will even pick the answer. It cannot be re-written to the model to generate accuracy to reach the same value at a single time. The paper also acknowledges that large-scale two sets of training sets are not strictly comparable; small-scale experiments use large-scale models to provide synthesis supervision for more direct comparisons. Original §2, §3–4
Therefore, there is still a step between the observation and the cause of determination: fine-grained labels, sample selection and training data distribution may change at the same time. For the researchers, the next question should be "how much advantage is left under the same problem and cost", rather than deducing all tasks from this result that should be scored gradually.
Usato et al. compared a variety of supervisory combinations on GSM8K earlier, including both rearrangements and update generators through expert iteration. Expert iteration is to search for better samples first, and then use these samples to train the model. In the paper, the result-oriented approach can obtain similar final answer performance, and the quality of the process needs to be checked separately; the learned ORM may also approximate the process feedback. Original §2.6, §3 and Table 1
The process error rate here is especially misunderstood: the denominator is The correct answer to the final answer, the proportion of the error steps in the statistics, not all the error rates of all answers. It reveals that "an answer is correct but wrong", and it cannot directly replace the overall correctness. The author also selects the iteration checkpoint according to the final answer test error rate, so these results should not be regarded as final tests that are not completely involved in the selection. Original §2.1, §2.6
The two studies do not constitute a clean opposite conclusion: primary school application problems are different from more difficult math problems, and the model starting point, label scale, update method and decoding protocol are also different. The latter results cannot overturn the former; the local advantages of the former cannot answer the generation training question for the latter.
Each step has a label, and it is possible to supervise the wrong object
Zhang et al. analyzed the data structure and evaluation of the process reward model. The common Monte Carlo annotation is to continue writing from a prefix multiple times, and estimate the score with the final success situation; but the failure to continue the writing may come from the lack of follow-up ability, and the success of the continuation may also be hit by error correction or luck. This makes "whether this step is right" and "can you still do it from here"? Original §2.1, §3.1
The paper also checks the candidate screening and wrong positioning, and finds that only looking at the Best-of-N will cover up the defects of the process: if the candidate book contains the correct answer but the derivation is wrong, the loose validator can also score points on the answer indicator. Therefore, the author adopts the consistency filtering of the wrong position with the continuation estimation and the model referee, and evaluates the two types of abilities at the same time. Original §3.1.3, §3.2
This article inference: The improvement after filtering still cannot prove that a certain annotation method is natural and superior. Retained questions may be easier to reach, and errors that are difficult to determine are excluded. When evaluating the quality of the label, you should look at what is screened out at the same time; otherwise, the clean data and the narrower coverage may be mistaken for the same benefit.
Can you learn to make mistakes just by labeling the results?
The 2026 LCA study explores a counterexample. Its main experiment retains Math-Shepherd’s reasoning traces and final-answer labels while discarding the step-level pseudo-labels. It learns to attribute errors by aggregating predictions over successive prefixes, then evaluates error localization and candidate reranking separately. The reported averages support the feasibility of the approach without establishing that it leads on every subtask. Jia et al.§4–5, Table 1–2
However, "training without step labels" cannot be expanded into "research without process information": hyperparameters are selected on the MATH subset of ProcessBench; another group of the aggregate method, which aggregates the manual step labels, is summarized into trajectory labels. The data sources of these two experiments must be separated. Original §5.1, §5.3, Appendix C.5
Its theory also relies on assumptions such as the overall label and the local error relationship. When the answer is correct and the process is wrong, this correspondence will be distorted; the nature of the overall goal does not guarantee that limited sample training will inevitably converge to an ideal solution. Original §3.3, Appendix C.5 and D.1
Therefore, the problem of recent evidence has narrowed: process supervision does not have to be equal to gradual manual labeling, but to prove that the misjudgment that can be migrated has learned, it still needs an independent and credible process evaluation. "Eliminating the training label" is a cost proposition; "finding a true error" is a competency proposition, and the two should be accepted separately.
Before comparing, put the budget and denominator back to the desktop
| Research | main condition | Supported observations | Conclusions that cannot be pushed |
|---|---|---|---|
| Lightman et al. | Fixed generator, MATH candidate screening | Process Oversight Validator has a selection advantage | The generator has become stronger due to RL |
| Uesato et al. | GSM8K, a variety of training and rearranging combinations | Answer performance and conditional process errors are separable | The two categories of supervision are equivalent in all tasks |
| Zhang et al | Automatic labeling, rearrangement and positioning evaluation | Label goals and evaluation goals may be misplaced | High rearrangement is reliable process verification |
| Jia et al | Result label training, structured aggregation | Learn useful process scores | There is no process information in the whole process or a global leader |
The observation in the table corresponds to the above original text, and does not rank horizontally. The difficulty of GSM8K and MATH cannot be regarded as an aligned variable; just writing "the same parameter of seven billion" cannot guarantee that pre-training, MATH fine-tuning and supervision history are consistent.
For follow-up research, this article proposes to establish two cost calibers. information caliber Fixed questions, trajectories and visible feedback to compare the target design; equal cost caliber Fixed labeling time or total calculation, allowing each method to allocate by itself. The former is beneficial to the research mechanism, and the latter answers how resources are used; the two do not have to choose the same winner.
Manual step-by-step labeling, model referee calls and sequential sampling are all included in the cost. The same number of candidates does not mean the same reasoning budget: the candidate length, validator size and number of ratings will change overhead. If you also want to train the generator, you should list the training budget separately, and the multi-sampled benefits during the test cannot be recorded into the learning ability.
Three subsequent experiments that can distinguish interpretation
The following is a design proposed in this article, which has not yet been implemented, and is not a reproduction result. They are respectively "more choice", "more wrong" and "more will generate" three explanations.
First, freeze the candidate pool, just replace the validator. Preserve the candidates of the same question and batch, compare the majority of voting, result scores and process scores, and record who chooses the right and who is wrong for the same question. The derivation of the selected answers is then independently checked. If the answer benefits exist and the process errors are not reduced, the conclusion should be left at the selection effect level. Problems that are not correctly solved in the candidate pool should also be reserved, so as not to hide the generation failure.
**Second, hold the final answer fixed and intervene on the intermediate reasoning.** Construct meaning-preserving rewrites, steps containing actual errors, and traces in which an initial error is later corrected. Match length and wording as closely as possible. If the score mainly tracks the final answer or the writing style, it provides little evidence of reliable error localization. Semantic labels need justified annotations and a way to resolve disagreements. The model being tested cannot serve as its own gold standard. We have yet to collect those annotations.
Third, after freezing the reward, retrain the generator. Starting from the same checkpoint, use the result reward, process reward and combination reward respectively under the same training budget. List the correctness of the independent answer, process error and cross-generator verification performance at the same time, and finally change the type of question that did not participate in the selection. If only the training reward is rising and the external indicator remains unchanged, this negative result should be retained, rather than continuing to adjust the evaluator until there is an advantage.
These designs should be pre-conventional development sets, final test sets and stop rules. In particular, it is necessary to put "Select hyperparameters with process labels" in the information budget, not just check what the gradient update has read. In this way, it can be judged that the lower labeling cost comes from the real savings, or is it transferred to the method selection stage.
Summary: Supervision granularity is not conclusion granularity
The current evidence supports process information that is valuable for validation and selection, and also supports learning part of the process judgment from the coarse result feedback. But the gains are not automatically passed along "Step Score - Selected Answers - Generation Ability - Generalization Mechanism". The most valuable report should explain which layer of improvement occurs, what information is consumed, and leave experiments that can overturn its own interpretation. When reading "The process reward is better", make up for the object, budget and indicators first, and then the dispute becomes a question that can be tested.
References
- Hunter Lightman et al., Let’s Verify Step by Step, 2023 preprint; ICLR 2024 officially released. arXiv v1; Full text of the official meeting. The numbers in this article are checked according to the official version.
- Jonathan Uesato et al., Solving math word problems with process- and outcome-based feedback,2022. arXiv v1; Full text. This time, the official employment is not confirmed, and it is quoted according to the preprint.
- Zhenru Zhang et al., The Lessons of Developing Process Reward Models in Mathematical Reasoning,2025,Findings of ACL 2025. formal page; Published paper; arXiv V2 full text.
- Tianyu Jia et al., The Weakest Link Tells It All: Outcome-Supervised Process Reward Modeling via Learnable Credit Assignment,2026. arXiv v2; Full text. This article adopts the 2026-09-26 version; the author has stated in arXiv that it has been accepted by EMNLP 2026 Oral, and the official meeting recruitment page has not been independently obtained this time, so this description will not be written as an official publication record that has been verified.