Cover illustration

Date:2026-07-11
Who this is for: Graduate student, researcher of mathematics reasoning and formalization method, LLM evaluation researcher, technical reader with engineering background
search date:2026-07-11

Abstract

LLM mathematical reasoning has entered a new stage in 2024-2026: the research focus is no longer just "whether the model can calculate the answer", but "whether the reasoning process can be checked, reproduced, and extended to the real difficult problem". From MATH, GSM8K, Minerva and process supervision, to LeanDojo, ProofNet, miniF2F, DeepSeek-Prover-V2, Kimina-Prover, LEAP, and then to IMO 2025 performance of AlphaGeometry2, Gemini Deep Think and IMO-Bench, MathArena In real-time evaluation, the cutting-edge route is converging in three directions: longer natural language exploration, stronger formal proofer feedback, and stricter decontamination Benchmark.

Mathematical reasoning may be the closest thing LLM research has to a test bed for “verifiable intelligence.” Problems with checkable answers provide inexpensive reward signals, formal proofs allow machines to check correctness, and olympiad and research-level benchmarks keep supplying new problems. Interpreting the results is harder. Drawing many samples does not establish mathematical creativity, an accepted Lean proof does not establish natural-language understanding, and competition scores alone do not demonstrate research-level mathematical ability.

Contents

  • Research background: Why mathematical reasoning suddenly became the main battlefield
  • Core scientific questions: the faults between answers, proof and formalization
  • A reading guide to the past year’s papers
  • Key papers by research question
  • Comparing the methods
  • How to interpret the experiments and benchmarks
  • Reproducibility and data problems
  • Limitations and open questions
  • Research questions for graduate students
  • Summary
  • References

Research background: Why mathematical reasoning suddenly became the main battlefield

Early MATH abilities were often measured by GSM8K, MATH, and AIME subsets. GSM8K proves that primary school application questions also require multiple steps of reasoning; MATH advances the difficulty of the questions to the competition mathematics and provides step-by-step answers; Minerva explains that in scientific papers, LaTeX And continue pre-training on the technical text, which can significantly enhance quantitative reasoning. Subsequently, Self-Consistency, Verifier, Process Reward Model and Test-Time Scaling make "multi-sampling, re-screening" a common paradigm for mathematical tasks.

But there are two structural problems at this stage. First, the correct answer does not mean that the correct answer is correct. The model may use the wrong steps to get the correct value, or it may give a seemingly rigorous but broken derivation. Second, the public Benchmark will soon be contaminated by training data, question bank explanations, video analysis and model distillation. The more popular AIME, MATH and Olympiad questions, the harder it is to distinguish whether the model is reasoning or recalling the question type and solution template.

Therefore, mathematical reasoning research begins to migrate to "verifiable". Proof Assistant, such as Lean, Isabelle, Coq/Rocq, Metamath, etc., provides a small kernel check mechanism: as long as the formal proposition and proof are accepted by the system, the local correctness of the certificate will no longer depend on LLM self-reported. The expansion of Lean 4 and MathLib is especially important because they put formal mathematics, program validation and LLM tools in the same executable environment. Lean's official website emphasizes that Lean is an open source programming language and proof assistant, which can be used for maintainable code and formal proof; Mathlib provides a large-scale mathematical library for cross-algebra, analysis, topology, probability and computer science.

Core scientific questions: the faults between answers, proof and formalization

There are at least five core issues in this route.

first, Is natural language reasoning reliable?. Long-chain CoT, Reflection, Tree Search, and Diversified Sampling can improve the hit rate, but also amplify the illusion. Mathematical text is easy to be "local reasonable, global error", especially combination, number theory, and geometric structure questions.

second, Proof whether it can be checked by machine. Formal proof turns "looks right" into "compiler acceptance". The problem is, from the natural language questions to the Lean proposition, from the human draft to the complete Proof Script, is itself a difficult AutoFormalization task.

third, How to compress the search space. Formal proof is not simply to generate a piece of code. The model needs to choose lemma, expand the definition, call TACtic, handle type errors, read Proof State, and trace back in a large number of failed branches.

fourth, Evaluate whether it is anti-pollution. Fixed public question bank will quickly expire. Real-time competition reviews such as MathArena, and more difficult or deformable, such as FrontierMath, IMO-Bench, VAR-MATH, etc. Benchmark, trying to push the evaluation from the "historical question bank" to "fresh questions, original questions, parameterized questions, and proof questions".

Fifth, Can mathematical skills migrate to research. IMO Gold-Level results are eye-catching, but competition mathematics and research mathematics are different. The competition questions are short, the boundaries are clear, and there are usually elegant solutions; research mathematics may need to define new objects, build theories, deal with long-dependence chains, and multi-year collaboration.

Mathematical Reasoning Research Framework

A reading guide to the past year’s papers

Formal proof models advanced rapidly in 2025–2026. DeepSeek-Prover-V2 brings DeepSeek-V3’s natural-language decomposition into Lean 4 proof generation, using subgoals and reinforcement learning to narrow the gap between informal and formal reasoning. Kimina-Prover Preview starts from Qwen2.5-72B and emphasizes formal reasoning patterns and large-scale RL. It reports strong pass@8192 performance on miniF2F and releases distilled models. Leanabell-Prover-V2 pursues verifier-integrated RL, using feedback from the Lean compiler across multiple turns. LEAP (2026) builds an agentic formal mathematics framework around a general-purpose LLM. It drafts an informal blueprint, then interacts repeatedly with the Lean compiler to break down subproblems and revise the proof state.

The second line is the natural language Olympian reasoning. The official blog released by Google DeepMind in 2025 said that the premium version of Gemini Deep Think reached the gold standard in IMO 2025, and the submission solution was via IMO Confirmation is complete and correct; the same page also indicates that the confirmation of IMO is not equal to the verification system, process, or underlying model. This CAVEAT is very critical: mathematical answers can be scored, and whether the system experimental conditions are equivalent to human competition, whether it can be reproduced, and whether to use undisclosed post-processing still needs to be independently reviewed. IMO-Bench offers AnswerBench, ProofBench, and GradingBench with the goal of evaluating short answers, long proofs, and automatic scoring.

The third line is geometry with Neuro-Symbolic. AlphaGeometry2 reported in the 2025 arXiv paper, and its language coverage has increased from 66% of IMO 2000-2024 geometry to 88%, and the overall resolution rate has increased to 84%. It is not a generic chat model, but a combination of Gemini architecture language model, geometric representation language, symbol engine, synthetic data and search mechanism. This shows that the coupling of neural models and symbol systems is still very competitive in such a structured strong and critical field in geometry.

The fourth line is the Benchmark counterattack. Omni-MATH provides 4428 Olympian math questions; FrontierMath is designed by experts to avoid public pollution in the question bank; MathArena uses the latest competition questions for real-time evaluation; VAR-MATH Transform fixed numerical questions into parametric multi-example questions to test whether the model really masters the structure. The common signal between the papers is that the high score of the old Benchmark is getting less and less enough.

Key papers by research question

1. From answer questions to Verifier: MATH, GSM8K, Minerva, process oversight

MATH and GSM8K are the infrastructure of this route. The value of MATH is to provide competition-level questions and step-by-step solutions; GSM8K makes "training Verifier, choosing from multiple candidate answers" as an early success paradigm of mathematical reasoning. Minerva further demonstrates that the technical corpus and LaTeX structure can help directly with mathematical abilities.

The process supervision paper "Let's Verify Step by Step" moves the supervision signal from the final answer to the intermediate Step, unleashing an important idea: mathematical reasoning training can not only reward the conclusion, but also reward the partial progress that can be checked. Subsequent PRM, ORM, self-consistency, best-of-N, and test-time compute all revolve around this idea. The limitations are also obvious: the human labeling steps are expensive, and LLM Judge is easy to prefer smooth text, and in the end, it is still impossible to replace formal inspections.

2. Autoformalization: Translating Natural Language Mathematics to Lean

ProofNet is an important Benchmark of AutoFormalization research: it contains natural language statements for undergraduate mathematics, proofs of natural language and formal statements of Lean. LeanDojo further provides interactive environment, premise retrieval, proof state data and ReProver, so that researchers can train Lean Prover in open source environments.

In 2024, "Improving Autoformalization using Type Checking" proposed to filter formal results with Type Checking and self-consistency, and pointed out that the evaluation itself also needs to be revised. The key to this direction is not to let LLM write beautiful Lean code, but to judge whether the formal proposition faithfully expresses the original question. When a formal proposition is written incorrectly, even if the proof is passed, it is only proof of another problem.

3. Neural Theorem Proving: Search, Search, RL and Compiler Feedback

DeepSeek-Prover-V1 is trained to prove the model with large-scale Lean 4 data, DeepSeek-Prover-V1.5 joins Proof Assistant feedback, RLPAF and RMaxTS search; DeepSeek-Prover-V2 then further use strong models to decompose recursive sub-targets, synthesize cold-start data, and then use reinforcement learning to improve the proof ability. It reports a very high pass ratio on the miniF2F-test and introduces a ProverBench; these numbers should be understood according to the paper settings, and cannot be directly equivalent to "automatically prove arbitrary math".

The inspiration of Kimina-Prover Preview is that it sees the proof process as an explicit formal reasoning pattern: the model not only outputs Tactic, but plans, trials and errors around the Proof State. Leanabell-Prover-V2 emphasizes the verifier-integrated long CoT, allowing the model to learn self-correction in the Lean error message. The contribution of LEAP is to introduce the Agentic Framework into the formal mathematics: Generally, LLM can write the blueprint first, and then continuously call the Lean compiler feedback, and gradually compress the proof space.

The engineering bottlenecks of this line are very specific: the Lean environment version, the Mathlib commit, the dependency index, the proof state serialization, the Premier Retrieval, the tactic timeout, and the parallel sampling budget, all of which will have a significant impact on the results. The pass@k in the paper is not a single model capability, but a combination indicator of model, searcher, retriever, compiler and calculation budget.

4. Geometric Routes: AlphaGeometry2 and Symbolic Engines Are Still Important

AlphaGeometry and AlphaGeometry2 remind us not to simplify mathematical reasoning to "larger LLM + longer CoT". The key to geometric problems is often to construct auxiliary points, lines, circles, or discover hidden similar relationships. Pure natural language models may know many theorems, but lack the mechanism of system search structure.

AlphaGeometry2 drives coverage and problem-solving rates by extending geometric language, improving symbol engines, knowledge sharing search trees and synthetic data. Its strengths are also the limitations: the system-dependent domain representation language and specialized symbol engines, which need to be redesigned when migrating to algebra, number theory, combination or research mathematics.

5. Osai and Frontier Benchmark: Be more cautious after high scores

Gemini Deep Think's performance statement in IMO 2025, natural language Reasoning Model has been able to give a proof of scoring on extremely difficult short questions. Google DeepMind's official blog states that its approach is based on natural language, not AlphaProof/AlphaGeometry in 2024 Such a formal translation process; at the same time, it also shows that a more valuable system in the future may combine natural language fluency and formal verification.

But the evaluation explanation must be restrained. IMO questions are only 6 times a year, and the statistical samples are very small; public statements usually cannot fully reproduce the experimental conditions; to prove that the scoring requires experts; MathArena's discovery also shows that there is still a clear gap between the top-level model and the performance of the answer question in the Proof-writing task such as USAMO.

Comparing the methods

Route Representative work core mechanism Fit the question Main risk
Answer questions + multi-sampling GSM8K、MATH、Minerva、self-consistency Generate multiple candidates, select by answer or Verifier Arithmetic, Algebra, AIME Short Answers It happens that you answer correctly, the question bank is pollution, and the steps cannot be checked
Process Supervision / PRM Let's Verify Step by Step、PRM800K Feedback to the intermediate steps Long-chain reasoning training, candidate sorting Labeling is expensive, Judge bias, there is no guarantee to prove correct
Autoformalization ProofNet、type-check filtering Natural Language Statement to Lean Propositions Undergraduate mathematics and theorem library construction Propositions are unfaithful, and the equivalent is difficult to judge
Lean prover LeanDojo、DeepSeek-Prover、Kimina-Prover、LEAP Search premise, search Proof state, compiler feedback, RL The machine can check the certificate Depend on version and computing power, pass@k is difficult to compare
Neuro-symbolic geometry AlphaGeometry2 LLM Proposed Construct, Symbolic Engine Proof IMO geometry Field dedicated, high migration costs
Real-time with Frontier Benchmark MathArena、FrontierMath、IMO-Bench、VAR-MATH Fresh questions, original questions, parametric questions, certification scores Anti-pollution evaluation High cost, low sample, difficult to score consistency

How to interpret the experiments and benchmarks

When reading mathematics and pushing theoretical essays, it is recommended to look at the task type first. Short answer questions, proof questions, formal proof questions and research-level open questions cannot be mixed together. AIME may only need an integer answer; USAMO needs complete proof; miniF2F needs proof assistant to accept; FrontierMath tries to approximate the workload of professional mathematicians for several hours to several days.

Second, see if you distinguish pass@1, pass@k, best-of-N and search budget. A model pass@1 is strong, indicating that the quality of a single generation is high; pass@8192 is strong, which may indicate that large-scale sampling and validators are very effective. Both are important, but the research has different meanings.

Third, see if the formal environment is fixed. Lean 4 version, MathLib commit, allowed Tactic, timeout settings, retrieval library and whether to allow sorry will change the result. Without this information, formal proof that experiments are difficult to reproduce.

Fourth, see if the benchmark is fresh. MathArena's ideas are worth paying attention to, as it will be evaluated as soon as the competition topic is released to reduce data pollution. It is also important that the parameterized multi-instance of VAR-MATH is to test generalization of the structure, not to remember a numerical question.

Fifth, look at the relationship between human grading and auto-grading. IMO-Bench's GradingBench attempts to evaluate the consistency of automatic scoring and manual rating, which is a problem that a long proof evaluation must face. An AI answer "looks like a proof" does not mean that you can get full marks under the rules of the competition.

Reproducibility and data problems

The reproducibility of mathematical reasoning is more complex than that of ordinary NLPs. For natural language models, researchers need to publish tips, sample temperature, candidates, tool permissions, whether to use search, and whether to use manual screening. For the Lean Prover, also publish the Lean version, MathLib commit, dependency installation, compilation cache, searcher configuration and hardware budget.

Data pollution is one of the biggest risks. MATH, GSM8K, AIME, IMO historical questions, ProofNet, and miniF2F are all highly disclosed, and many model training sets may have been indirectly included. Even if the title itself does not enter the Training Set, the handouts, forums, parsing video and model-generated data may contain similar solutions. In the future, more credible evaluations will rely on three types of designs: original questions, real-time competition questions, and parametric variants.

Another underestimated problem is "formalized proposition loyalty". If AutoFormalizer translates the natural language problem into a weaker proposition, the Prover is easy to pass; if it is translated into a stronger proposition, the model may be punished unfairly. Therefore, the AutoFormalization evaluation requires manual or symbolic equivalence checking, not just a type-check.

Limitations and open questions

First, the performance of the Olympiad does not mean studying mathematics. IMO questions are difficult, but usually short-range, closed, and scorable problems. To study mathematics, it is necessary to define new concepts, integrate long documents, construct theoretical frameworks and write reusable lemma libraries.

Second, formal proof does not mean mathematical understanding. Lean accepts a Proof Script, indicating that it holds in the given form system; but whether the model understands the key ideas of the proof, whether it can be migrated to the adjacent theorem, and whether it can be explained to the mathematician, still needs additional evaluation.

Third, test-time scaling has a cost problem. A large number of sampling, tree search, compiler interaction, and Verifier Reranking can significantly improve the score, but also bring incomparable computing power, high latency and non-reproducible experimental settings.

Fourth, automatic scoring will be optimized. As long as Benchmark and Judge are fixed, the model may learn to cater to the scorer, rather than improve math abilities. Proof of missions requires independent expert spot checks and adversarial grading.

Fifth, the public code and model are still insufficient. Open routes such as DeepSeek, Kimina, LeanDojo, etc. are valuable, but many of the strongest systems publish blogs or limited technical details. For academic researchers, reproducing systems are more important than headlines.

Research questions for graduate students

  1. Faithful AutoFormalization Review: Construct natural language propositions, Lean propositions and artificial equivalent labels, and study the detection method of "proving pass but wrong".
  2. Lean Proof State diagnostic dataset: Collect failed Proof State, error messages, repair actions and human interpretations, and train a more reliable Verifier-Aware Prover.
  3. Parametric Orsai Benchmark: The AIME/AMC/USAMO style questions are transformed into generateable families to evaluate whether the model has invariants rather than memorized numbers.
  4. low budget Theorem Proving: Study the proof strategy under the conditions of pass@1, pass@8 or single GPU, and avoid relying only on massive sampling.
  5. Natural Language Proofs to Lean Blueprints: Let the model build into a structured Proof Plan, and then fill in Tactic with a dedicated Prover to analyze the relationship between blueprint quality and final pass rate.
  6. Explainable Search of Geometric Structures: Reproduce or transform the AlphaGeometry idea, compare neural structures, WU methods, DDAR and human-assisted structures.
  7. Long certificate automatic score: Research Rubric-Aware Grading, local error positioning, and human-machine consistency around the IMO-Bench/Matharena-style task.

Summary

The frontier of LLM mathematical reasoning is shifting from "can do questions" to "can prove, can verify, and be able to generalize on new questions". Natural language Reasoning Model gives strong exploration capabilities, Lean and Mathlib provide machine checkable correctness, real-time competition and original Benchmark are fixing pollution problems. The most noteworthy system in the future may not be a separate chat model or a separate proofer, but a mathematical research assistant who can switch back and forth between natural language ideas, formal propositions, compiler feedback, search budgets and expert reviews.

For researchers, don't just chase higher AIME or miniF2F scores in the next step. The more valuable question is: Is the original question proved by the model? Why does it choose a lemma? Can I locate the error when I fail? Is it still valid on fresh questions, variants and research-level theorems? These questions determine whether mathematical reasoning can move from competition display to truly reliable scientific research tools.

References

Search date: 2026-07-11. The following links give priority to the use of papers, official blogs, official project pages, or benchmark pages; rapid change model availability, list scores, code repository status, and preprint conclusions without peer review, and manual review before release is recommended.

  1. Dan Hendrycks et al., “Measuring Mathematical Problem Solving With the MATH Dataset”, arXiv, 2021. https://arxiv.org/abs/2103.03874
  2. Karl Cobbe et al., “Training Verifiers to Solve Math Word Problems”, arXiv, 2021. https://arxiv.org/abs/2110.14168
  3. Aitor Lewkowycz et al., “Solving Quantitative Reasoning Problems with Language Models”, arXiv, 2022. https://arxiv.org/abs/2206.14858
  4. Hunter Lightman et al., “Let’s Verify Step by Step”, arXiv, 2023. https://arxiv.org/abs/2305.20050
  5. Kunhao Zheng, Jesse Michael Han, Stanislas Polu, “MiniF2F: a cross-system benchmark for formal Olympiad-level mathematics”, arXiv, 2021. https://arxiv.org/abs/2109.00110
  6. Zhangir Azerbayev et al., “ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematics”, arXiv, 2023. https://arxiv.org/abs/2302.12433
  7. Kaiyu Yang et al., “LeanDojo: Theorem Proving with Retrieval-Augmented Language Models”, arXiv, 2023. https://arxiv.org/abs/2306.15626
  8. Lean official website, “Lean Programming Language”. https://lean-lang.org/
  9. Lean community, “A mathlib overview”. https://leanprover-community.github.io/mathlib-overview.html
  10. Huajian Xin et al., “DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data”, arXiv, 2024. https://arxiv.org/abs/2405.14333
  11. Huajian Xin et al., “DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search”, arXiv, 2024. https://arxiv.org/abs/2408.08152
  12. Z. Z. Ren et al., “DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decomposition”, arXiv, 2025. https://arxiv.org/abs/2504.21801
  13. Haiming Wang et al., “Kimina-Prover Preview: Towards Large Formal Reasoning Models with Reinforcement Learning”, arXiv, 2025. https://arxiv.org/abs/2504.11354
  14. Xingguang Ji et al., “Leanabell-Prover-V2: Verifier-integrated Reasoning for Formal Theorem Proving via Reinforcement Learning”, arXiv, 2025. https://arxiv.org/abs/2507.08649
  15. Po-Nien Kung et al., “LEAP: Supercharging LLMs for Formal Mathematics with Agentic Frameworks”, arXiv, 2026. https://arxiv.org/abs/2606.03303
  16. Yuri Chervonyi et al., “Gold-medalist Performance in Solving Olympiad Geometry with AlphaGeometry2”, arXiv, 2025. https://arxiv.org/abs/2502.03544
  17. Google DeepMind, “AI achieves silver-medal standard solving International Mathematical Olympiad problems”, 2024. https://deepmind.google/blog/ai-solves-imo-problems-at-silver-medal-level/
  18. Google DeepMind, “Advanced version of Gemini with Deep Think officially achieves gold-medal standard at the International Mathematical Olympiad”, 2025. https://deepmind.google/blog/advanced-version-of-gemini-with-deep-think-officially-achieves-gold-medal-standard-at-the-international-mathematical-olympiad/
  19. Thang Luong et al., “Towards Robust Mathematical Reasoning”, arXiv, 2025. https://arxiv.org/abs/2511.01846
  20. Mislav Balunović et al., “MathArena: Evaluating LLMs on Uncontaminated Math Competitions”, arXiv, 2025. https://arxiv.org/abs/2505.23281
  21. Bofei Gao et al., “Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models”, arXiv, 2024. https://arxiv.org/abs/2410.07985
  22. Elliot Glazer et al., “FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI”, arXiv, 2024. https://arxiv.org/abs/2411.04872
  23. Jian Yao, Ran Cheng, Kay Chen Tan, “VAR-MATH: Probing True Mathematical Reasoning in LLMs via Symbolic Multi-Instance Benchmarks”, arXiv, 2025. https://arxiv.org/abs/2507.12885
  24. Auguste Poiroux, Gail Weiss, Viktor Kunčak, Antoine Bosselut, “Improving Autoformalization using Type Checking”, arXiv, 2024. https://arxiv.org/abs/2406.07222