Cover illustration

Date:2026-07-03
Who this is for: Graduate student, LLM/Agent researcher, model application and reasoning system engineer
search date:2026-07-03

Abstract

In the past year, LLM's "ability extension" has not only occurred in the pre-training stage. More and more papers and products are putting additional computing power to AT Inference Time: multi-sampling, search, validator rearrangement, reflection cycle, budget control, and generating invisible Reasoning tokens inside the model. The core problem of Test-Time Scaling is not as simple as "let the model think for a while", but to decide when it is worth thinking about when it is worth thinking about, how to think more, and who can judge whether it is effective under the constraints of delay, cost and reliability.

The research path runs from Self-Consistency, Tree of Thoughts, and process-supervised verifiers to OpenAI o1, DeepSeek-R1, s1 budget forcing, and papers from 2025 that take a systems view. Inference-time compute is becoming a systems engineering problem. Future agent infrastructure will need ways to track reasoning budgets, route tasks by difficulty, evaluate verifier quality, and recover from failure, alongside serving policies that account for cost and latency.

Contents

  1. Research background: Why is at Inference Time computation important
  2. roadmap for the past year
  3. Key papers by research question
  4. Comparing the methods
  5. Technical trends
  6. Implications for deployment
  7. Limitations and open questions
  8. Questions to watch
  9. References

Research background: Why is at Inference Time computation important

Traditional LLM Scaling Law mainly discusses parameters, data and training FLOPs. For users, the inference after the model deployment is mostly regarded as a "cost center": less token, less tuned, and reduced latency. But the complex inference task exposes another fact: the same model, with different decoding strategies, sampling times, search structure and validator help, the output quality may be much worse.

Self-Consistency is an early representative: it no longer greedy to choose a chain-of-thought, but sample multiple inference paths, and then vote for the final answer consistently. This idea is very simple, but it has established the basic form for the later Test-Time Scaling: split once an answer into "candidate generation + selection/aggregation". Tree of Thoughts further organizes the intermediate inference unit into a searchable Tree, allowing the language model to lookahead, self-evaluation and backtracking in at inference time. The validator route tries to train a Reward Model or a Process Reward Model to score candidate answers or intermediate steps.

After 2024, The meaning of The AT Inference Time calculation has been re-enlarged. OpenAI put reinforcement learning, chain-of-thought, bug fixes and multi-sample consensus in The O1 technical description to The foreground; DeepSeek-R1 The paper shows how reinforcement learning stimulates inference patterns such as self-reflection, verification and dynamic policy adjustment. By 2025, S1, Rollout Roulette, The Art of Scaling Test-Time Compute, System Perspectives Test-Time Scaling, etc. have begun to ask: Is it always useful to think about it? When to sample more, when to search, and when to stop?

AT Inference Time Computational Framework Diagram

roadmap for the past year

2024-08:compute-optimal test-time scaling. Snell et al.'s paper expresses the problem very clearly: how to allocate the test with the most effective computing power when given a fixed but non-trivial reasoning budget? The paper compares the method of search based on process validator and adaptive update distribution by Prompt, and points out that the strategy effect strongly depends on the difficulty of the problem. Its importance is not a single point of skill, but in moving test-time scaling from "multi-sampling may be better" to "adaptively allocating budgets on difficulty".

2024-09 to 2025: Reasoning Model is productized. The OpenAI O1 series turns "models generate longer thinking processes before answering" into user-perceivable product capabilities. The OpenAI API documentation also clearly distinguishes between Input/Output Tokens and Reasoning Tokens, and reminds Reasoning Tokens to occupy the context window and the billing output token. For the engineering team, this means that Reasoning is not just a model behavior, but also a resource that can be measured, quota, and triggers Incomplete Response.

2025-01: DeepSeek-R1 and S1. DeepSeek-R1 connects RL, verifiable tasks, small model distillation and open source model ecology; S1 proposes a minimalist problem in the reverse direction: use only 1,000 high-quality reasoning trajectories to supervise fine-tuning, and then use it again. Budget Forcing controls "thinking length", can it reproduce a part of the test-time scaling effect? In this type of work description, the inference ability may come from both large-scale RL, and may also be migrated at lower cost through data screening, distillation and decoding control.

After 2025-02: Search paradigm diversification. Rollout Roulette interprets Inference-Time Scaling as a problem of probability inference, and uses the Particle-based Monte Carlo method to avoid over-reliance on the mode of a single Reward Model optimization. It reminds us that the validator or Reward Model itself will be biased.

The second half of 2025 to before 2026: from algorithm to system. The Art of Scaling Test-Time Compute uses large-scale equivalents to emphasize that "there is no single TTS strategy is generally optimal". The system perspective paper points out that compute-optimal is not necessarily Serving constraints such as delay, cost-per-token, Tensor Parallelism, Specularative decoding, etc. change The optimal strategy. This step is especially critical for real Agent products.

Key papers by research question

1. Multi-sampling and self-consistency: the simplest AT Inference Time extension

The method of Self-Consistency is to sample multiple reasonsing paths first, and then marginalize or vote on the final answer. It is suitable for tasks with clear answers and answer spaces that can be aggregated, such as mathematics, common sense quiz, and multiple-choice questions. The advantage is that the implementation is simple, and the inside of the model is not modified; the disadvantage is also obvious: if the candidate path is highly relevant, the voting is just copying the same error; if the answer is not discrete and comparable objects, such as open design, long text writing or multistep code modification, aggregation will become difficult.

Engineering inspiration is: multi-sampling should be "Fallback with budget", not to expand all requests by default by n times by default. High-value, high uncertainty, and automatic scoring requests can be identified at the task routing layer, and then self-consistency can be triggered. Otherwise, it will directly amplify costs and tail delays.

2. Search and Planning: From Linear CoT to Trees, Graphs, and Particles

Tree of Thoughts treats "thought" as a search node, and lets the model generate candidate intermediate states, and then evaluate, prune, and backtrack. This type of method is suitable for problems that need to be explored and planned, but it changes the number of model calls to a Tree at one time, and the project cost is very high. Follow-up MCTS, Beam Search, particle methods, and search-enhanced searches are essentially answering the same question: how to make the exploration wider, and at the same time avoid reward models or self-evaluation misleading search directions.

The value of Rollout Roulette is to introduce the perspective of probability inference: not only pursue the highest score sample of the Reward Model, but use sampling methods to explore a typical collection. This is important for complex reasoning, because the path that "looks the highest score" is probably the path the validator is most spoofed.

3. Validator and Process Supervision: Let "Select Answers" to be trained

OpenAI's early Training Verifiers to Solve Math Word Problems verfier select answers in multiple candidate solutions; Let's Verify Step by Step compare Outcome Supervision and Process Supervision, and publish the PRM800K Step-Level Feedback dataset. The intuition of process supervision is: if the final answer is wrong, we also want to know which Step is wrong; if each Step can be evaluated, the search has a more fine-grained navigation signal.

But the validator is not a silver bullet. Validator training data, labeling criteria, task distribution, and Adversarial Robustness can affect whether it is reliable. If the weak validator is connected to the strong search, you may get a "more confident error". The real system must record at least the candidates, scores, validator versions, summary reasons summary, final selection and manual review samples, not just the final answer.

4. Reflection and budget control: let the model know when to stop

Reflexion is a type of work that writes the failure feedback back to the language memory, and let the agent correct the strategy in subsequent attempts. DeepSeek-R1 describes self-reflection, verify, dynamic strategy adaptation as a reasoning pattern that emerges in RL. S1's Budget Forcing is more engineered: when the model tries to end, it continues to check by appending "Wait"; forcibly truncated when the budget is exhausted.

The most noteworthy of this line is the "stop strategy". Many demos show how many ideas bring about improvement, but what the production system really lacks is: judge whether the current task is worth thinking about, whether there are still marginal benefits in the next round, whether the tool should be called instead of continuing to think internally, whether it should be transferred to manual or downgraded.

Comparing the methods

manner typical representative Additional Inference Cost suitable for tasks Main risk Engineering proposal
Self-Consistency / best-of-N Wang et al. 2022; OpenAI O1 Multi-sample consensus description Linear increase With clear answers, aggregation Candidate related, high cost Triggered only on high indeterminate requests
Tree / MCTS / search Tree of Thoughts;R2-LLMs;Rollout Roulette Possibly exponential or multi-round linear Planning, Math, Code Search Search is misled by the weak evaluator Plus budget cap and pruning audit
Outcome verifier Training Verifiers Generate n candidates + scores Math, Selection, Code Testing Just look at the final answer, the explanation is weak Combine with unit test/symbol tool
Process verifier / PRM Let’s Verify Step by Step;PRM800K The step-level score is more expensive Multistep derivation Expensive labeling, distribution migration Record step level Telemetry
endogenous reforming model o1;DeepSeek-R1 increasing tokens Common and complex tasks Invisible thinking, cost is difficult to control Monitoring Reasoning Token and Failure Rate
Budget forcing / adaptive compute s1;compute-optimal TTS controllable Validating Reasoning Excessive prolonging low-quality trajectories Routing with a difficulty estimator

Technical trends

First, test-time scaling is becoming a "routing problem". Different tasks, different difficulties, and different model families need different strategies. The large-scale comparison in 2025 has suggested that no TTS method is universally optimal under all conditions. The next stage will be more like an Inference Controller: first estimate the difficulty and verifiability of the task, and then select Greedy, self-consistency, search, tool execution, verifier rerank or manual retraction.

Second, the validator will become the core model in the inference stack. Future Agents need not only the generator, but also Verifier, Critic, Reward Model, Unit-Test Runner, Symbolic Checker, Retrieval Checker. The version management, calibration, and online drift monitoring of the validator will be as important as managing the Embedding Model or reranker today.

Third, the Reasoning Token will become an observable resource. The OpenAI documentation has exposed Reasoning tokens as part of the Usage Object and states that they are invisible but occupancy context and cost. As long as the commercial APIs are developing in this direction, the application layer needs to include the "thinking budget" into the SLA: the maximum recalling tokens, the maximum wall-clock, the maximum number of retries, the acceptable failure rate and hierarchical billing.

Fourth, the optimal system will challenge the optimal algorithm. A method is better under the Token Budget, which does not mean that it is better under the real Serving. Parallel sampling may increase throughput pressure; long thinking will increase P95/P99; the search tree will destroy the cache hit; the validator rearrangement will introduce additional model services. The system paper reminds very directly: compute-optimal is not equal to Latency/Cost Optimal.

Fifth, training and inference can inform each other. DeepSeek-R1 and s1 both show that high-quality reasoning traces can be distilled into smaller models. Failed inference cases can also feed back into post-training. A promising direction is to bring hard cases from production, verification signals, human review, and distillation together in a cycle, so each round of inference can inform the next round of training.

Implications for deployment

To land test-time scaling, the first step is not to go to MCTS, but to define task hierarchy. Low-risk, low-value, open requests are generally generated; high-value and verifiable requests trigger multi-sampling or validators; high-risk tasks must have tool validation, log and manual audits. Don't take "complex Prompt" as the only signal, you should combine user intent, historical failure rate, model confidence proxy metrics, verifiability of answers and cost budgets.

The second step is to establish reasoning Telemetry. At least record: base model, temperature, sample count, number of reasonsing tokens, visual output token Number, total delay, number of candidates, validator score, final selection strategy, whether the trigger tool, and whether to manually review. Without these data, it is impossible to judge whether the thinking is improved or just to make mistakes more expensive.

The third step is to maintain the validator as an independent product surface maintenance. The validator requires offline sets, online sampling, calibration curves, version rollback, and security assessment. Especially in high-risk fields such as code, mathematics, medicine, and law, we give priority to deterministic tools: unit tests, type checks, symbol solvers, and retrieval evidence consistency checks. LLM-as-verifier can be involved in sorting, but should not be exclusive to adjudication.

The fourth step is to design a stop and downgrade strategy. The most easily out of control at Inference Time is "think another round". The system should have a hard budget and a soft stop condition: multiple rounds of candidates are consistent, the margin of validator is small, the tool feedback is clear and failed, the context space is insufficient, or the user chooses a low-latency mode. The product form that is visible to the user can be "fast/deep/verifiable" three gears, instead of all requests being in-depth by default.

The fifth step is to look at TTS and Serving optimization together. The next article will specifically talk about Efficient LLM Inference, but here is a hook: KV cache, Speculative decoding, batching, prefix caching, parallel sampling scheduling, will change The practical feasibility of test-time scaling. The token budget in the research paper must be converted into GPU seconds, queue delay and single request gross profit when it comes to production.

Limitations and open questions

First, Benchmark may overestimate The generalization ability of AT Inference Time calculations. Mathematical competitions, code questions and controllable Puzzle are convenient for automatic scoring, but it is not equal to The open world Agent tasks. The Illusion of Thinking points out that large inference models will collapse when complexity continues to rise, and there is a low-complexity standard model, which is more suitable and medium complexity. The partition phenomenon of model dominance and high complexity failure. Such results should not be interpreted as "inference models are useless", but remind us not to equate long Thinking with reliable reasoning.

Second, the interpretability of the thinking trajectory is still limited. The business model usually does not expose the original chain-of-thought, and only provides summary or invisible Reasoning token statistics. There is also controversy in research: Is it visible that the reasoning text is faithfully reflects internal decisions? If not faithful, what exactly does the process oversight and reflecting logs supervise? These issues directly affect security audits and scientific interpretations.

Third, the validator may cause a new attack surface. As long as the system uses rewards or scores as a search target, the model may learn to cater to the validator. The stronger the search, the more the validator vulnerability will be mined. This is the same type of problem as RLHF Reward Hacking, just happens at Inference Time .

Fourth, cost fairness deserves attention. The deep reasoning model will transfer more costs to the inference stage. For the platform, this may improve the average quality; for users, the price, latency and accessibility of complex problems are layered. Future products need to be clearly explained when additional inferences are used and whether users can control it.

Questions to watch

  1. Will there be a generic difficulty estimator that will be used to predict how much inference budget is required before the request is entered into the model?
  2. Can Verifier form standard evaluation, calibration and deployment specifications like Embedding/Reranker?
  3. Where is the optimal boundary between long thinking model and tool call, retrieval, and code execution?
  4. Can the cost of test-time scaling be significantly reduced through Speculative decoding, parallel sampling scheduling and cache reuse?
  5. Will process oversight data become the core asset of the Post-Training data flywheel?
  6. How to balance between hidden reasoning tokens and auditable interpretation in security and compliance scenarios?

Summary

Test-Time Scaling is one of the most trackable directions for the LLM system from 2025 to 2026, because it connects model capabilities, inference algorithms, validators, Serving costs and product experience. Its research context starts with multi-sampling voting, and enters the Reasoning Model and reasoning budget management stage through tree search, process supervision and reflection mechanisms.

The most practical conclusion for the engineering team is: don't use the test-time scaling as a single prompt, but design it as an observable, route, quota, and backward inference control. plane. The real competitiveness is not just "what the model can think about", but whether the system can know when to think more, how to verify, when to stop, and how to convert the failed sample into the next round of training and product improvement.

References

Search date: 2026-07-03. The following priority lists the papers, official documentation and project homepages; individual Benchmark numbers are all from the original paper or official pages, which are not extended to cross-model SOTA conclusions in this article.

  1. Charlie Snell, Jaehoon Lee, Kelvin Xu, Aviral Kumar. “Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters.” arXiv:2408.03314, 2024-08-06. https://arxiv.org/abs/2408.03314
  2. OpenAI. “Learning to Reason with LLMs.” 2024-09-12. https://openai.com/index/learning-to-reason-with-llms/
  3. OpenAI API Docs. “Reasoning models.” https://developers.openai.com/api/docs/guides/reasoning
  4. DeepSeek-AI et al. “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.” arXiv:2501.12948, v2 revised 2026-01-04. https://arxiv.org/abs/2501.12948
  5. Niklas Muennighoff et al. “s1: Simple test-time scaling.” arXiv:2501.19393, 2025. https://arxiv.org/abs/2501.19393
  6. Xuezhi Wang et al. “Self-Consistency Improves Chain of Thought Reasoning in Language Models.” arXiv:2203.11171, ICLR 2023. https://arxiv.org/abs/2203.11171
  7. Shunyu Yao et al. “Tree of Thoughts: Deliberate Problem Solving with Large Language Models.” arXiv:2305.10601, NeurIPS 2023. https://arxiv.org/abs/2305.10601
  8. Hunter Lightman et al. “Let’s Verify Step by Step.” arXiv:2305.20050, 2023. https://arxiv.org/abs/2305.20050
  9. Karl Cobbe et al. “Training Verifiers to Solve Math Word Problems.” arXiv:2110.14168, 2021. https://arxiv.org/abs/2110.14168
  10. Isha Puri et al. “Rollout Roulette: A Probabilistic Inference Approach to Inference-Time Scaling of LLMs using Particle-Based Monte Carlo Methods.” arXiv:2502.01618, v5 revised 2025-08-14. https://arxiv.org/abs/2502.01618
  11. Parshin Shojaee et al. “The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity.” arXiv:2506.06941, NeurIPS 2025. https://arxiv.org/abs/2506.06941
  12. Youpeng Zhao et al. “Are We Scaling the Right Thing? A System Perspective on Test-Time Scaling.” arXiv:2509.19645, 2025. https://arxiv.org/abs/2509.19645
  13. Aradhye Agarwal, Ayan Sengupta, Tanmoy Chakraborty. “The Art of Scaling Test-Time Compute for Large Language Models.” arXiv:2512.02008, 2025-12-01. https://arxiv.org/abs/2512.02008