
Abstract
A math problem is handed over to the model, which writes a proof, and Lean marks a double tick next to the statement. What can you use at this time with confidence? If the title has been accurately written as a formal proposition and the proof is complete, and the axioms used are also within the scope of the license, then others can take this certificate and hand it over to the same environment for re-examination. Even if the model forgets how to do it tomorrow, this math result holds. Conversely, if one of the conditions in the question is deleted, and then ask where the original certificate of the model fails, this pair cannot give the answer. The previous question examines the delivery certificate, and the latter question examines whether the creator can recognize the proof dependence on the conditions.
This article narrows the "understanding" into a question that can be tested: Can the model identify the key premise, reuse the argument after the equivalent rewriting, and give a counter-example when the premise is invalid? The following three papers let us see clearly, which step is the meaning of the question, what the validator provides to the search, and whether other proofs can continue to use the generated theorem. The evidence they give can be read together, but the scores cannot be put together into a list of abilities.
First look at which question the inspector received
Imagine a questioner who wrote: "The sum of any two positive integers is greater than each of them." The translator wrote "positive integers" as "integers", and the certifier immediately found that one of the additions could be zero, and the strict inequality would no longer hold. If the translator only allows both numbers to be greater than ten, the proof is easy to pass, but all legal situations involving one to ten have been quietly moved out of the problem. This is an illustrative example. What the two mistranslations have in common is that what the inspector has received is already another question; no matter how strict it is, it is impossible to regain the scope of the correction by just checking this certificate.
Lean is the proof assistant. After the mathematical objects, assumptions and targets are written clearly in formal language, they can call the proven lemma, or they can write "strategy", that is, let the program construct the instruction of a certain step on behalf of the program. The final check is a proof of being able to launch a target in the current environment. Instructions for the double tick in the Lean manual
When reading the report, you should ask whether the submitter only completes the proof below the question, whether it can be rewritten, and whether it is ruled out to leave an unprovable conclusion with placeholders. The latter two settings will change the meaning of "file compilation successfully". A model can be fully proved under a fixed question, which is a useful and verifiable achievement in itself. Proof generation of fixed questions, Figure 1

The left column compares the original question and formal statement, and the middle column indicates that the proof of the given proposition has been accepted, and the right column proposes the migration after the condition is changed. The three icons indicate evidence that needs to be collected separately, and the question mark on the right indicates that it has yet to be tested.
The first gap occurs before writing the proof
What Poirux et al. studied was "automatic formalization of statements", that is, to translate natural language propositions into leans. Even if the certificate is not written here, the quality of the translation can be evaluated. They put together candidate statements and reference statements, and try to push out the other from one with a program called BEq+, and then switch directions. Compared with checking words, this gives the equivalent of rewriting the opportunity to be recognized. For example, moving a hypothetical clause to the left side of the implication, the text seems to have changed, but the proofer can move the same condition back and forth between the two writing methods. Section 3 and Appendix A of paper
On the ProofNetVerif with artificial annotation, the accuracy rate given in Table 1 is 98.0% and the Recall is 48.3%. The former denominator is the candidate judged by the program as equivalent, and the latter denominator is the candidate for label recognition. Looking at the position of the examiner, this means that most of the candidates who get the affirmative judgment can manually judge, but only these candidates are only regarded as all correct translations, and a considerable part of them will be missed. Another interesting result is that on the revised ProofNet#, after summarizing according to the model and method, the automatic score is quite consistent with the sorting of the manual Accuracy. Table 2 compares the sorting of a set of methods, and the reader cannot transfer this result to any one question to ensure that the translation of the question is reliable. Table 1, Table 2
Even "two-way pushable" has boundaries. The author has discussed a mistake in the appendix: if one side itself takes the conclusion as a prerequisite, it proves that the search may easily close the target. In this paper, taking the integer diversion relationship as an example, the original question is concerned with the transition from diversion in the Gaussian integer to the diversion of the ordinary integer, but the candidate has already assumed the divisible in the ordinary integer. The reader lost the most important step of this question. The author also records the repair of such problems. The value of this example is to explain what is used to check the equivalent determination, and it is impossible to infer from this that the current implementation will still misjudgment the example. Appendix A.6
The same certificate may have experienced a very different search
Remove the translation problem and give all systems the same form of proposition. DeepSeek-Prover-V1.5 provides a very clean readout opportunity. The author allows the system to generate the entire certificate at a time, and also allows it to continue to explore from the valid prefix with the intermediate state returned by the Lean. The latter can keep the previous part when a certain step is wrong, and then try different sequels. In addition to acceptance at the end point, the validator can also tell the system "what is left here". The person deploying this search system can reasonably count it as part of a problem-solving tool. Figure 1 and Section 4
First, just look at the same reinforcement learning version in Table 1, the entire generation, and the row of the same miniF2F test set. When the sampling budget of each question increased from 128 to 3200, the average problem solving rate reported increased from 51.6% to 54.9%, and the corresponding standard deviation was 0.5 and 0.7 percentage points. The scoring unit here is the question: as long as there is a certificate of qualification in the budget, the question will be solved. Therefore, even if the model weight is the same, multiple attempts can change the "ability" that the reader sees in the table. The two averages do not tell us whether a certain problem depends on key lemma, common templates, or some accidental combinations after failure. Section 5.1 Table 1
This gives two explanations that can be established at the same time. The model may have mastered how to decompose the conclusion into appropriate sub-objectives, but sometimes the name of the library is wrong; it may also try to familiarize with the strategy of a certain type of questions until there is a branch that just applies. Both can leave a completely correct proof, even the final text may be the same. If you only save the winning trajectory, you can't see how much these trajectories account for in all attempts, and you can't see whether you can choose the right branch again after changing the conditions. In turn, many branches have been tried and cannot be used as evidence that lacks understanding, and there is still a distance between knowing which theorem to use and writing the correct parameters the first time.
What happens when you put the generated theorem back in place?
Kim et al. changed the test method. Take a statement from the real Lean project, let the model make up according to the context, and then put the supplemented content back to the repository. Here the creator can decide the form of the declaration. The author checks whether the candidates themselves can be compiled and whether the subsequent theorems that depend on it can all be compiled. The latter item is called test Accuracy . It is assumed that there is a lemma in the library that applies to all positive integers, but the replacement is additionally required to be input as an even number: to prove that the replacement itself may not be difficult, but a proof of an odd number of old lemma invoking an odd number will immediately encounter an unsatisfactory premise. Task Examples and Section 5.2
Under the complete Test Set, the setting of the natural language certificate and the subsequent theorem context, the independent compilation rate of the same Claude-Sonnet-4.5 is 80.3%, and the test is 38.9% after all the dependencies test. Both columns are denominator of the entire group. The distance between them indicates that some submissions that can be independently established cannot be used for their original purposes after replacing them back to a specific project. The follow-up theorem has already been part of the generation prompt, so this result also has a clear scope of application; if you want to ask the model if it can foresee a new purpose, you need to leave a batch of call scenarios. Table 2, Table 3
Consider what a failure and a success can each establish. A failed downstream call pinpoints a real incompatibility, though even a mathematically equivalent interface can break existing calls if it reorders the arguments. Conversely, a replacement with a narrower domain might slip through if all downstream calls happen to use even numbers. The authors note that sparse dependencies leave gaps in coverage and that a standalone result with no subsequent theorem depending on it cannot be evaluated this way. The natural-language descriptions were also model-generated and may differ from the original authors’ intent. I read this metric as evidence that a replacement works in existing calls. Fidelity to the intended problem still needs an independent check. Section 8
How to design a question that can distinguish explanation
The three studies are different for each fixed. The first item is given a reference translation, and the consistency of the evaluation and judging procedure and the annotation is the consistency; the second item is given a formal problem, and the problem solved under the specific generation and search budget is measured. The starting point of the three models, the dataset, whether to provide subsequent calls, and the allowed attempts for each question are different, and there is no shared training comparison. Arrange the accuracy, problem-solving rate and test Accuracy by numerical values, and mistake three different denominators as three exams for the same batch of students. What they can really constrain together is the wording of the conclusion: it can be said that which work has passed which kind of inspection, and then the behavior after changing the question another question.
An implementable subsequent experiment can start with the above positive integer examples. Write three questions in advance for each motif: the original question, the question that keeps the meaning but exchanges the order of expression, and the question that only relaxes one key condition. The equivalence problem shall be prepared by the questioner in advance for the proof of check, and the counter-example that can be checked for the distortion after relaxation. For example, when a zero is allowed to be added, as long as a positive integer and zero are given, the sum is equal to the original number. In this way, it can not only detect whether the system will repair the certificate, but also leave a way to answer correctly for "this question can no longer be proved". The question should be fixed in advance; let the same generator change the question, answer the question and evaluate the counter-example at the same time, and the translation error will be re-introduced.
Send each variant to the same model independently, hiding the answers of the same family questions. It is required to select a step that may fail before calling the tool, and then allow it to submit a certificate or counterexample. Fixed available lemma, model version and prompts, set the same upper limit of total generation for the search of complete resampling and readable intermediate state, and also record the number of times the checker is called and time-consuming. Only the number of attempts to align only attempts is still unbalanced, as the cost of rewriting the entire proof is different and the last step of the rewrite is different. Timeouts, tool errors, and proactive responses are false and handed over failure certificates should be kept separately, especially the use of search as evidence that the proposition is false.
Keep the results paired by original problem. If a system proves both the original and an equivalent version, then predicts where the reasoning breaks and supplies a counterexample when a premise is removed, it becomes hard to explain success as simply reusing a fixed answer. If the advantage appears only when search restrictions are lifted, or the system gives elegant reasons but repeatedly fails to supply a valid counterexample, narrow the conclusion to finding proofs within budget or generating explanations. Even passing the whole set would provide evidence only for transfer across this kind of premise change. A different mathematical field, library, or more complex quantifier structure calls for new problem families. This is a proposed design. I did not run model experiments for this article.
Summary
The next time you see "Verified by Lean", first find out the statement of the theorem of the verified line, and see which conditions are written into it; then see how many attempts the generator gets, and how to call it with other proofs. This can not only take over a re-examination certificate, but also give the sentence "understood" with a truly discerning question: Can it still point out the reason and do it right to change the condition that determines success or failure?
References
Poirux et al. (2025), Reliable Evaluation and Benchmarks for Statement Autoformalization,EMNLP.
Xin et al. (2025), DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search,ICLR.
Kim、Han、Hwang(2026), Benchmarking Testing in Automated Theorem Proving,ACL Industry Track.
Lean Language Reference, Validating a Lean Proof, the official online manual, viewed in November 2026.
Apps by the author
I also build small tools for everyday life. Here are two of my apps:
- Daily Expense Journal: Track your income and spending, plan a budget and set savings goals. View on the App Store.
- One Small Action: Pick a small action that fits the time and energy you have today, then record how it felt and what you noticed. View on the App Store.
Find more articles and projects on my website: zffjzz.top. I’d love to hear how the apps work for you and what I could improve.