
- Date: 2026-10-14
- For graduate students, early-career researchers and research engineers
- Data deadline: 2026-10-14; retrieve related work for the past 12 months, and retain the early research required to explain the problem. This article is for the deep reading of the paper, no model training or reproduction experiments are run, and it does not claim to be fully covered.
Abstract: Like teachers, answering correctly, and migrating, there are three questions
The small model starts to write "Let me check it", and the answer is more accurate. Does this mean that it has learned the inference method of the large model? There are at least three explanations: it has learned useful problem-solving steps; it learns to organize output and extend calculations, to better call existing knowledge; it will also abide by the answer format, and the answer that cannot be scored is finally recognized. Several changes can occur at the same time.
**Distillation can yield measurable task gains without establishing that the student has copied the teacher’s internal reasoning mechanism. Conversely, finding that the structure of the explanation matters does not show that the student can only imitate.** Four lines of evidence help disentangle these possibilities: supervision with auxiliary rationales, distillation from long reasoning traces, interventions on style, and evaluation across tasks.
What exactly did we give to our students?
Distillation here refers to the update of students with training signals generated by the teacher model. This article focuses on the inference distillation at the text level: teachers provide questions, inference trajectories and answers, and students learn these outputs by supervising fine-tuning. Supervising fine-tuning for short SFT is to use the demonstration output to train the next text; it is not equivalent to directly transporting the teacher's parameters, and does not require the teacher's tokens to be read.
The thinking chain or reasoning trajectory is the intermediate text that can be seen before the answer. "Expression habits" can refer to the tone, conjunctions and chapter structure; "problem-solving ability" needs to be defined by successful behavior under specified conditions. The two are not naturally mutually exclusive: a checking step may be both a verbal expression, and it also increases the chance of re-checking for subsequent generation. It is too fast to classify it as decoration, or to classify it as an internal algorithm.
In order for the discussion to be verifiable, this article distinguishes three levels: surface similarity to see if students are more like teachers; mission performance See if the answer to the independent rating is more correct; Migration and intervention stability After looking at the type of question, changing the expression, and changing the key conditions, whether the income is maintained. The third layer of evidence is usually more constrained than the first two layers, but it is not a single pass to prove "understanding".
For example, suppose a student learns to add a check paragraph at the end of each question. It may actually check the operation, or it may just extend the output, or repeat the correct answer in the check section, making the extractor easier to grab. This example is the hypothesis for interpretation, not the result of the paper. Research design requires that these explanations produce different predictions, not just give the output a louder name.
Four kinds of evidence are not the same question
Reasons can be useful during training without having to write them out during tests
"Distilling Step-by-Step!" of Hsieh et al. uses label prediction and reason generation as two tasks, controlling the same student with different prefixes. Teachers are PaLM and students are T5, and the evaluation involves natural language inference, common sense quiz and arithmetic questions. The paper reports the benefits of data efficiency oversight; the label can only be predicted during the test, and there is no need to make a reason. Formal paper §3.2, §4
The importance of this design is to exclude an overly narrow explanation: the benefits may not all come from writing an extra paragraph during the test. Reasons may also provide additional oversight to change how students use problem information. However, this is a training for specific tasks, and cannot be directly equivalent to the migration of general inference methods. The author also pointed out that the reason for supervision increases training expenses, and the impact of reason quality still needs to be studied. Original Limitations
The long trajectory formula is effective, does not mean that the contribution of each ingredient has been separated
DeepSeek-R1's first technical report uses about 800,000 sorted samples Fine-Tuning Qwen and Llama students, and only SFTs are used at the student stage. About 600,000 of them are reasoning-related data, and about 200,000 are non-inference data. The report presents the results of the distillation model in mathematical, scientific question and answer and code evaluation. Technical Report V1 §2.3.3, §2.4, §3.2
The key boundary here is: the training packet changes the content of the problem, the output structure and the instruction behavior at the same time. It supports the feasibility of this set of formulas, but it does not contribute to the separate identification of "process content" through the control of the same question, the students, and only changing the semantics of the trajectory. Teacher training and screening also have costs, and the total cost advantage cannot be declared just by looking at the training stage of students. The research was later published in Nature; the formulation of this paragraph is positioned using the first edition of the clearly ensured that the first edition of the report is not mixed, and the chapter number of the subsequent version is not mixed. Official publication record
"Wrong answers can also learn something" does not mean that the content is not important
In “Style over Substance,” Lippmann and Yang compared R1 traces with GPT-4o traces generated from a structured template, then trained students that had already undergone instruction tuning. The template prescribed how to organize the problem, explore possible solutions, check them, and bring the pieces together. Both kinds of data improved performance on the tasks tested. In a separate ablation, replacing the final answer with an incorrect one still yielded some gain, though less than with the correct answer. Original V3 §3–4, Table 1–2
The easiest to read is the author's own limitations: the wrong ending version still retains the correct reasoning content before, not pure style or all wrong track. So this experiment cannot prove that semantics are useless. This paper further believes that the structured template may also change the problem-solving behavior, and this kind of intervention cannot be directly regarded as only a few mantras; the equal number of samples is not equal to the equal amount of training tokens and the inference usage. Original § 4.2, Appendix C–D
This work does provide support for "style contributes", but the strong wording in the title should not replace the experimental boundary. This time I read the full text of arXiv v3; the author's homepage and arXiv indicate COLM 2025. The direct access to the conference PDF is restricted by the browser verification, and the search excerpt is not regarded as the full text of the meeting.
Cross-task progress is more informative than writing styles, but the source needs to be dismantled
Gautam compares six pairs of original models with R1 distillation models in EACL 2026, and the task is English pronouns with pronouns. The paper reports improvements at the dataset summary level, and analyzes the error type through manual annotation; some improvement involves instruction compliance. The manual evaluation here is carried out by the author of the paper, not the review performed by this article. Formal Papers §3–5
This provides behavioral evidence beyond typical mathematical problems, but cannot attribute all the effects to mathematical reasoning migration: training is mixed with non-inference data, and most of the original models do not have instructions Fine-Tuning . Although the evaluation has the same upper limit of generation, the actual output of the distillation model is longer; the combination of individual models and data sets will also regress. The authors clearly discussed these limitations. Therefore, the "Improvement of Cross Tasks" and "Mechanisms have been isolated" must be separated. Original §4, Appendix B.1 and D
| What the evidence can tell us | Compare conditions | Supported Observation | can not be proved |
|---|---|---|---|
| Assistant Reasons Supervision | Task-Specific Students, Labels and Reason Branches | Reasons are useful in training | General methods have been migrated |
| Long Track Distillation | Screened mixed training package | Recipe brings task benefits | Content is the only source |
| style intervention | Template track with error ending | Structure worthy of separate study | Only the surface style is effective |
| Cross-task evaluation | Published model pairs, new task tests | Benefits of behavior are not limited to style of writing | The causal effect of pure reasoning migration |
Each line corresponds to the above papers, which is neither a budget competition nor a camp that denies each other. Different tasks, students' starting points and indicators are different, and their promotion numbers are arranged horizontally to the list, which will lose the information that this article really cares about.
Why can't the student's capacity and budget be hidden in the footnotes?
When comparing students, "same size" is not equal to "same starting point". Pre-training knowledge, math training and instruction training can all affect how it uses a teacher demonstration. Primary school students cannot reproduce the long solution, which is not enough to show that there is no valid information in it; older students perform better, and they cannot attribute the difference to capacity by simply parameterizing the difference. More credible comparisons should try to fix the model family, training recipes and tasks, and then use the capacity as a clear variable.
The teacher's answer is correct, and there is no guarantee that every step in the middle is correct. The filtering endpoint can keep the path that happens to be the right answer; students may not have enough background to make up for the steps that the teacher has omitted. Therefore, the stronger the teacher does not necessarily mean that all the trajectories it writes are more suitable for a certain student. This is a teaching adaptation hypothesis that needs to be compared, and this article does not use it as a law to prove it together as four studies.
The budget includes at least four parts: teacher generation, data screening, student training and student reasoning. The same number of problems may correspond to different training texts; the same maximum output length may also correspond to completely different actual consumption. If you are concerned about the use value, you should compare how Accuracy changes with the budget; if you care about the transmission of knowledge, you should also check whether the semantic complete trajectory still has additional benefits after controlling the amount.
This also shows that more "checking" vocabulary is not a good independent ability indicator. The frequency of these words can be counted, but it should be regarded as a behavioral description, and recorded separately with Accuracy, error type, length and new question performance. Otherwise, the training objectives and evaluation standards may revolve around the teacher's style of writing, and finally can only get the circular conclusion that "students are more and more like teachers".
A follow-up experiment that can better distinguish interpretation

The figures are shown in this paper, which have not been performed yet. The four channels are trained separately from the same student's checkpoints, and then evaluate new problems according to the same budget caliber. The mechanical tracing of the cover and the new labyrinth are also only problem metaphors, not visual evidence of the internal mechanism.
First select the limited tasks that can automatically verify the answers and intermediate operations, fix a set of public training questions, and retain completely independent new structural test questions. Four materials are made for each training question: only the answer, the complete and effective steps, the style rewrite of the same semantics, and the trajectory of only one key step is destroyed. The last set should keep the final answer and length as consistent as possible, and record the conflicting position, to avoid miss the label noise as the role of process semantics.
The style rewriting itself also needs to be verified: deleting "Let me check" cannot delete the actual inspection at the same time; the replacement word cannot change the operation dependency. Prioritize the program-verifiable step representation, and then generate different language expressions. If manual semantic judgment is required, the labels must be truly collected, and the automatic scoring cannot be used to pretend to be an artificial gold label. All rewriting and culling rules should be determined before the test results are observed.
The training also reports the results of the alignment by question number and the results of tokens according to the effective training of tokens, and explains what the two calibers have changed each. The test terminal compares multiple fixed generation budgets, and records the cut-off rate and actual usage; the answer is standardized and the answer is standardized to reduce the interference of the format to improve the scoring. The new questions are reserved according to the structure or combination, instead of just changing the number of the same template; repeated operations are used to determine whether the difference is stable.
How is expected to be interpreted? If the complete step and semantic equivalent rewriting are better than the answer group, and maintain it on the new structural questions, and the damage to the key steps is obviously damaged, the process semantics and contributions are more supported. If the advantage mainly changes with the output length and is greatly reduced after matching the budget, the calculation allocation explanation is more supported. If the proceeds are mainly derived from the reduction of invalid answers, it should be admitted that the instruction is followed as an important source. They may be established at the same time, and the experiments should report their respective scopes.
Even if the first result is obtained, it can only be said that the transferable information is found in this task family, students and budget, and it is impossible to declare and copy all the inference algorithms of the teacher. In the next step, the problem generation rules, student starting points and teacher expressions should be changed, and then the conclusion can continue to be established.
Summary: Leave "what passed" to the measurement that can be distinguished and interpreted
The actual benefits of distillation do not need to be proved by "full understanding", and mechanism research cannot stop at the evaluation of "just imitation". Existing evidence allows us to acknowledge that there is justification oversight, structured trajectory, and cross-task training behavior progress while retaining their respective causal gaps. When reading the next distillation paper, the most worthy of questioning is: what information does it change and which alternative explanations are excluded?
References
- Cheng-Yu Hsieh、Chun-Liang Li、Chih-kuan Yeh、Hootan Nakhost、Yasuhisa Fujii、Alex Ratner、Ranjay Krishna、Chen-Yu Lee、Tomas Pfister,2023. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. Findings of ACL 2023; use the full text of the official paper.
- DeepSeek-AI,2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Formula positioning uses V1 of 2025-01-22; Nature official paper Published on 2025-09-17, the title is DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning.
- Philip Lippmann、Jie Yang,2025. Style over Substance: Distilled Language Models Reason Via Stylistic Replication. The full text uses the V3 of 2025-07-15; the COLM 2025 state is composed of arXiv and Author Research Homepage Cross-checking, the full text of the meeting is limited directly.
- Vagrant Gautam,2026. Training in Step-by-Step Formal Reasoning Improves Pronominal Reasoning in Language Models. EACL 2026 Short Papers, 121–135 pages; use full texts of formal papers.