
Date:2026-08-06
Who this is for: already read Transformer, Instruction Tuning, RLHF Or preference optimization, want to systematically understand how the "inference model" paper goes from prompting skills to training paradigm, reasoning budget and verifiable search for graduate students, new researcher and engineering background readers.
Abstract
Reasoning Models is not as simple as "let the model output a few more lines of thinking Process". The routes of the past few years have roughly gone through five stages: Chain-of-Thought to make the intermediate steps explicit; Self-Consistency has changed a problem to multi-path sampling and answer aggregation; Verifier and Process Supervision turns "whose Reasoning is more reliable" into a learnable oversight signal; the Tree/Search method regards the inference path as a space for exploration; 2024-2026 O1, DeepSeek-R1, Kimi K1.5, s1, Qwen3, etc., then push the training reinforcement learning, test calculation budget, long CoT, distillation and verifiable rewards to the same research map during training. This article gives a essay line suitable for intensive reading and reproduction: don't just read each paper Scores, but ask whether the task can be verifiable, whether the inference trajectory is supervised, how the Search space is defined, where the additional calculation of the flower is, and whether the failure sample can be positioned in specific steps.
Contents
- Why is this topic important
- Core reading method: split inference into five variables
- A reading guide to the key papers
- A closer look at key papers
- Methods and experiments compared
- How to reproduce the results
- Common misconceptions
- Research questions for graduate students
- Summary
- References
Why is this topic important
"Reasoning" is the most easily abused word in LLM papers. A model can be used in GSM8K, MATH, AIME, GPQA, Codeforces Or getting a high score on the program synthesis task may mean that it really learns a more robust intermediate step, or it may just be more pattern matching, sample more answers, call strong validators, or benchmark has been indirectly overwritten by training data. Therefore, the first principle of reading Reasoning Models is: don't ask "this model is smart or not", first ask "what is Reasoning for this paper".
The contribution of the classic CoT paper is to let the intermediate steps enter the input and output interface: the model does not only generate answers, but generates a readable problem-solving process. The contribution of Self-Consistency is to replace "single path" with "multiple candidate paths", and use answer consistency to reduce accidental errors. Works like GSM8K Verifier and PRM800K continue to advance: If you can score candidate solutions or intermediate steps, reasoning can be selected, trained, and supervised. The Tree of Thoughts further organizes the candidate steps into a search Tree, rather than just taking a path in the Token layer.
After 2025, the focus of the Reasoning Models has changed. DeepSeek-R1 puts reinforcement learning on verifiable tasks to the central position, showing that it can induce self-reflection, verification and policy switching without a lot of manual annotation inference trajectory. Kimi K1.5 emphasizes RL extension, long context and long2short. S1 uses a small, high-quality inference dataset and the minimum path that is extended during testing. The Qwen3 technical report writes Thinking Mode, Non-Thinking Mode and Thinking Budget into the interface of the open source model family. These work together: The Reasoning model has changed from Prompt Engineering to a research direction of training data, reward design, inference control, verifiability and system cost.
Core reading method: split inference into five variables

To read this direction, it is recommended to split each paper into five variables.
First, whether the task is verifiable. Mathematical questions, code questions, formal proofs, and some scientific Q&A can be verified with answers, unit tests, proof inspectors or rules; open Q&A, planning, writing and security judgments are more dependent on human evaluation or model review. The verifiable task is naturally suitable for sampling, RERANK, RL, and search; non-verifiable tasks are more likely to appear in Reward Hacking or a "look reasonable but unprovable" reasoning chain.
Second, where does the reasoning trajectory come from? CoT uses a small number of artificial examples to induce trajectories; STaR, LEAST-TO-MOST, Auto-CoT, etc. try to construct or decompose trajectories automatically; Step-level annotation is required; DeepSeek-R1-Zero is a kind of route that hopes to let the model develop its own long inference model through verifiable rewards. The track to be recorded in the notes is manually written, model generated, screened, or in the RL Process.
Third, choose what level to happen. Self-Consistency votes on the answer layer; Verifier scores on the entire solution layer; Process Reward Model scores on the step layer; Tree of Thoughts Extend and prune in the "Thought" layer; code and proof tasks can be validated in the executor or Proof Checker layer. The finer the hierarchy, the higher the cost of oversight and the ability to mislocate.
Fourth, extra calculating where the flowers are. The inference model often builds performance improvements on more tests: multi-sampling, long CoT, reflection, search, validator scoring, tool execution, and RERAANK. The reproducing experiment must record the samples, temperature, max tokens, thinking budget, search width, search depth, verifier cost and failed retry strategy, otherwise the result will not be comparable.
Fifth, the evaluation of whether the results are correct and the process is reliable. The correct answer does not mean that the intermediate process is credible; the intermediate process looks beautiful and does not guarantee that the answer is correct. The value of ProcessBench is to locate the error to the earliest error step, reminding us whether the PRM can be generalized to more difficult math problems, it is an independent problem rather than a natural establishment.
A reading guide to the key papers
| Stage | Representative work | What to look for as you read |
|---|---|---|
| Explicit intermediate steps | Chain-of-Thought, Least-to-Most | How the Prompt example changes task decomposition, is it only suitable for large models |
| multipath sampling | Self-Consistency, Large Language Monkeys | Coverage, pass@k, consensus, verifier selection relationship |
| result validator | GSM8K verifier, code unit tests | Generate or verify which is harder and validator is overfitting answer pattern |
| Step supervision | Let's Verify Step by Step, PRM800K, ProcessBench | Differences between Outcome Supervision and Process Supervision, step labeling costs |
| Search Reasoning | Tree of Thoughts, MCTS-style reasoning | How to define state, action, thought, value, pruning policy |
| Reinforced Learning Reasoning Model | OpenAI o1 report, DeepSeek-R1, Kimi k1.5 | Train-Time Compute, Test-Time Compute, Verify Reward, Long CoT |
| Open source reasoning interface | s1, Qwen3 | Thinking Budget, Budget Forcing, Distillation, reproducible training script |
This table is not a "must read list ranking", but the coordinate system when reading papers. If you claim to improve Reasoning, you can first determine which column it has mainly changed: hints, sampling, validation, step supervision, search, RL or reasoning budget.
A closer look at key papers
Chain-of-Thought Prompting. Look for both the effect and its limits. The paper shows that, in sufficiently large language models, a few examples with intermediate steps can improve multi-step reasoning. Beyond the phrase “think step by step,” examine the task types, model sizes, how the examples were constructed, and why smaller models did not see the same gains. For a reproduction, fix the prompt, example order, decoding parameters, and answer extraction rules.
Self-Consistency. Self-consistency replaces greedy chain-of-thought decoding with sampling several reasoning paths and aggregating them by their final answer. When several valid paths lead to a unique answer, diverse samples can make the result more robust. When answers are open-ended, answer extraction is unreliable, or the model consistently misreads the question, voting may amplify the same mistake. Sketch the three steps as you read: sample paths, extract answers, and take a majority vote.
Training Verifiers and Let's Verify Step by Step. The GSM8K Verifier work demonstrates the basic paradigm of "generating multiple candidates, then selected by the validator". Let's Verify Step by Step advances the supervision from the final answer to the middle Step, and publishes the PRM800K step-level feedback data. Intensive reading of this type of paper, the core is to compare Outcome supervisor and process Supervision: The former is cheap but the credit distribution is thick, the latter is more diagnosed, but the labeling cost is high, the labeling consistency is difficult, and it may only cover a certain type of question.
Tree of Thoughts. The key to ToT is not the "tree" visual metaphor, but to turn the middle thought generated by the language model into a searchable object. Read this article to be clear: what is the status, how the next candidate is generated, how to evaluate value or vote, how to prune BFS/DFS, and whether the task really needs a global search. It is intuitive on tasks such as Game of 24 and Mini Crosswords, but when migrating to open research questions, the search space and evaluation functions become blurred.
OpenAI O1 report. The official report is a high-level technical signal rather than a complete reproducible paper: it explicitly lists the train-time compute and test-time compute as the inference ability improvement axis, and demonstrates the math, code, GPQA evaluation of tasks. Be especially cautious when reading this model report: you can learn problem definitions, evaluation options and safe observations, but you can't treat undisclosed training details as a reproducible experimental protocol.
DeepSeek-R1. The importance of DeepSeek-R1 is to clarify the "verifiable task + reinforcement learning + inference model emergence" clearly, and to distinguish DeepSeek-R1-Zero and the multi-stage training DeepSeek-R1. When intensive reading, there are four things to ask: where the rewards come from, which tasks can be automatically verified, why do not rely on the artificial manager CoT, the self-reflection can still appear, and whether the distillation ability, format or search strategy is distilled to the small model. Since the arXiv page shows that the work is updated to V2 on 2026-01-04, and the Nature 2025 version is associated, the version should be indicated when referring.
Kimi K1.5, S1 and Qwen3. Kimi K1.5 puts RL extensions, long contexts and long2short together, emphasizing a concise RL framework that does not depend on complex MCTS, value functions and PRM. S1 asks from the opposite direction: with 1000 high-quality inference samples, SFT and Budget Forcing, can you get a simple test extension path. The Qwen3 technical report takes Thinking / Non-Thinking and Thinking Budget as part of the model family capabilities, and exposes the Apache 2.0 model. The three are suitable for reading together: one looks at large-scale training practices, the other looks at small-scale reproduction experiments, and the other looks at how the open source model interface exposes the reasoning budget.
Methods and experiments compared
| Method | main variable | suitable for tasks | Recover risk |
|---|---|---|---|
| CoT prompting | Number of examples, sample quality, decoding parameters | Mathematics, Common Sense, Multi-step Q&A | prompt leak, answer extraction is inconsistent |
| Self-consistency | Sample number, temperature, voting rules | The only answer but diverse questions | High cost, the wrong path may be homogeneous |
| Outcome verifier | Number of candidates, validator training set, RERAANK rules | Math, Code, Multiple Choice | Validator to write an answer or overfitting distribution |
| Process reward model | Step granularity, labeling protocol, error positioning | Mathematical derivation, proof, step-by-step tasks | Labeling is expensive, and the generalization of cross-difficity is unstable |
| Tree/search | State definition, width, depth, value function | Planning, combination search, game questions | Search is meaningless when the evaluation function is fuzzy |
| RL reasoner | Rewards, Policy Optimizer, Exploration Budget | Verify math, code, STEM | Reward hacking, high training costs |
| Thinking budget | MAX tokens, budget mandatory, stop policy | Requires speed/accuracy compromise tasks | Different model interfaces cannot be directly compared |
When reading the experimental table, don't just look at pass@1. Reasoning paper is commonly used pass@k, consensus, rerank, coverage, step error localization, token budget, latency and cost. The most basic reproduction statement should be reported at least simultaneously: single answer, multi-sampling with the same budget, the same token cap, whether to use Verifier, whether to use a tool or test case, and classification of failure cases.
How to reproduce the results
Do not start repeating the full DeepSeek-R1 or Kimi K1.5. A better training camp path is to do a small and diagnostic Reasoning Lab.
- Select the verifiable task. Use GSM8K, MATH subsets, HumanEval/MBPP small samples, or 100-300 you constructed to automatically determine the sub-question. Prioritize the task that can automatically judge Final Answer.
- Establish four baselines. Direct Answer, CoT Greedy, CoT Self-Consistency, CoT + Verifier/Reranker. Each baseline fixed model, prompt, temperature, max tokens and answer draw scripts.
- Record the reasoning budget. The number of samples generated, output token, total AT inference time, number of failure retries, and validator calls for each question record. There is no budget record, and any "reasoning ability improvement" is not complete.
- Do step-level error labeling small samples. Randomly draw 50 failure cases, and manually mark the earliest error steps: reading questions, formula errors, arithmetic errors, search direction errors, answer format errors, and validator misjudgment.
- Reproduce a search toy task. Reproduce the ToT idea with Game of 24 or Mini Crosswords, compare width, depth, value prompt, and vot prompt. The goal is not to chase SOTA, but to understand search variables.
- Join Budget Ablation. Increase the number of samples from 1, 4, 8, 16, or let the Thinking Token Budget be divided, and observe whether the Accuracy, cost and failure types change synchronously.
It is recommended to output a reproduction report: experimental settings, data version, model version, prompt, SEED, budget, metrics, result table, failure case, and the mechanism interpretation you think is the most likely. Note: If a closed-source model or an online API may change, the call date and model name must be indicated in the report; if the version cannot be locked, label "Wait for manual verification".
Common misconceptions
- Treat long answers as long-term reasoning. Long CoT may include repeated, digression and self-rationalization; judging it is useful, you need to look at final answer, step errors and budget benefits.
- Compare Accuracy only, not cost. If a 64-sample consensus method is only a little bit higher than 1-sample, it may be valuable in research, but both engineering and the paper conclusion must be budgeted.
- The validator is valid with an unvalidable task. Without independent labels, actuators, proof checkers, or human expert reviews, the validator score can easily become another model preference.
- Ignore answer extraction. In mathematics and code evaluation, answer formats, units, equivalent expressions, and floating point tolerances will create false failures or false successes.
- Equal Benchmark to general reasoning. GSM8K, MATH, AIME, GPQA, and Codeforces measure different abilities, and should not be mixed into a "reasoning score".
- Make the public model report as a complete paper. o1. Some business model reports provide important clues, but when the training details, data and code are incomplete, they cannot be written as re-realistic evidence.
Research questions for graduate students
- Inference budget router. Given the difficulty of the problem, model Confidence and cost constraints, dynamically select Direct, CoT, Self-Consistency, Verifier, or Search.
- PRM is generalized across difficulty. The process reward model for training or screening with GSM8K/MATH can locate the early errors of the Olympiad-level problems.
- Long CoT compression. Study long2short: which intermediate steps must be preserved and which are just redundant self-tests.
- The validator layer in the code task. Compare the contributions of natural language critic, unit testing, fuzzing, static analysis and execution feedback to candidate program selection.
- The credibility of the process of open tasks. For scientific research questions and answers that cannot be automatically scored, design the Evidence Trail that can be audited by humans, rather than just letting the model self-evaluate.
- Search space design. Compare Token-Level Sampling, Step-Level Search, Tool-Level Planning, and Proof-Level Search for the same task, and analyze which granularity is the most economical budget.
Summary
The main line of Reasoning Models can be summarized as follows: first let the model write out the intermediate process, then let the model generate multiple paths, then train or call the validator to select the path, and finally turn the inference into a system that can be allocated budget, searchable, and can be reinforced learning optimization. When intensively reading this route, it is necessary to avoid being biased by the expression "thinking". What is really worth recording is: how the inference trajectory is generated, verify where the signal comes from, whether the additional calculation is controllable, whether the evaluation can be reproduced, and whether the failure can be positioned to the specific steps.
If you want to start reading today, it is recommended to follow five articles: CoT, Self-Consistency, Let's Verify Step by Step, Tree of Thoughts, DeepSeek-R1. Then use S1 and Qwen3 to observe how the 2025 open source inference model turns Thinking budget into an experimental variable. After reading this, you will not only repeat "the model will think deeply", but can ask a testable question: under what tasks, what budget, and what verification signals, the reasoning has really become reliable?
References
Search date: 2026-08-06.
- Jason Wei et al. “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.” arXiv, 2022; NeurIPS 2022. https://arxiv.org/abs/2201.11903
- Xuezhi Wang et al. “Self-Consistency Improves Chain of Thought Reasoning in Language Models.” arXiv, 2022; ICLR 2023. https://arxiv.org/abs/2203.11171
- Denny Zhou et al. “Least-to-Most Prompting Enables Complex Reasoning in Large Language Models.” arXiv, 2022; ICLR 2023. https://arxiv.org/abs/2205.10625
- Karl Cobbe et al. “Training Verifiers to Solve Math Word Problems.” arXiv, 2021. https://arxiv.org/abs/2110.14168
- Dan Hendrycks et al. “Measuring Mathematical Problem Solving With the MATH Dataset.” arXiv, 2021; NeurIPS 2021. https://arxiv.org/abs/2103.03874
- Hunter Lightman et al. “Let's Verify Step by Step.” arXiv, 2023. https://arxiv.org/abs/2305.20050
- Shunyu Yao et al. “Tree of Thoughts: Deliberate Problem Solving with Large Language Models.” arXiv, 2023; NeurIPS 2023. https://arxiv.org/abs/2305.10601
- Princeton NLP. “tree-of-thought-llm” official repository. https://github.com/princeton-nlp/tree-of-thought-llm
- Shunyu Yao et al. “ReAct: Synergizing Reasoning and Acting in Language Models.” arXiv, 2022; ICLR 2023. https://arxiv.org/abs/2210.03629
- David Rein et al. “GPQA: A Graduate-Level Google-Proof Q&A Benchmark.” arXiv, 2023. https://arxiv.org/abs/2311.12022
- OpenAI. “Learning to reason with LLMs.” Official report, 2024. https://openai.com/index/learning-to-reason-with-llms/
- Bradley Brown et al. “Large Language Monkeys: Scaling Inference Compute with Repeated Sampling.” arXiv, 2024, revised 2024. https://arxiv.org/abs/2407.21787
- Scaling Intelligence. “large_language_monkeys” official repository. https://github.com/ScalingIntelligence/large_language_monkeys
- Chujie Zheng et al. “ProcessBench: Identifying Process Errors in Mathematical Reasoning.” arXiv, 2024, revised 2025; ACL 2025. https://arxiv.org/abs/2412.06559
- DeepSeek-AI et al. “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.” arXiv, 2025, revised 2026; Nature 2025 version linked on arXiv. https://arxiv.org/abs/2501.12948
- Kimi Team et al. “Kimi k1.5: Scaling Reinforcement Learning with LLMs.” arXiv, 2025. https://arxiv.org/abs/2501.12599
- Niklas Muennighoff et al. “s1: Simple test-time scaling.” arXiv, 2025. https://arxiv.org/abs/2501.19393
- SimpleScaling. “s1” official repository. https://github.com/simplescaling/s1
- Zhong-Zhi Li et al. “From System 1 to System 2: A Survey of Reasoning Large Language Models.” arXiv, 2025, revised 2025. https://arxiv.org/abs/2502.17419
- An Yang et al. “Qwen3 Technical Report.” arXiv, 2025. https://arxiv.org/abs/2505.09388