
Large-model research controversy: from the paper conclusion to the boundary of evidence
Abstract
Someone in the group meeting said that the model after fine-tuning finally did an old question. The person sitting next to him wants to see its previous ten answer sheets, another person wants to change the key facts in the question stem, and some people want to anonymously hand over the old and new answers to the same referee. The three people are all asking if this progress is credible, but they may each get a correct and irreplaceable answer.
This is also the most worthy of reservations when looking back at this series. Whether the question has been solved, whether the answer is used for the specified evidence, and whether the judge recognizes a good answer, there are three links that can be wrong. The following four first-hand studies, for each link, choose a question that can be answered in a small-scale comparison. The experimental protocols are all proposals, and the observations indicated in the article come from the papers.
Put the previous controversy back on the same table
Looking back at the few articles of reasoning, distillation and multi-agents, the most likely to be missed is the attempt to fail. What the reader sees is the selected beautiful solution, and the system may also generate many candidates who answer the question. There is one more choice of process supervision and self-reflection: some people in the middle point out which step is wrong, and the person who answers it again knows one more thing than the first time. If only one original material can be saved, I will leave all the attempts of the same question, including the answer that was not selected in the end. They allow later people to differentiate and choose what each contributing.
Knowledge conflicts, long-term memory and visual basis encounter another difficulty. When the input material is exactly the same as common sense, you will get the same word according to the material answer and guess by memory. The success of the tool call also leaves a similar question. The familiar interface name has been paired with the actual function, and it is difficult to see which one the model depends on. It is worth leaving a pair of questions: keep the questions and operational requirements, and only replace the evidence that determines the correct answer. Readers can point out where the two answers should start from.
Let's look at a few articles on simulation migration, formal proof, list reversal, and scientific research discovery. What the judge accepts will determine how strong the conclusion can be written. After the certificate is handed over to the inspector, there is still the question of whether the meaning of the question is translated correctly; after the scientific research system has written a readable paper, there is still a question of whether the product can be tested by others. There are no common fractional units for the three gaps. The following is only a dispute that meets two conditions at the same time: the researcher can take the initiative to change one thing, and at least one possible result will make him take back the original judgment.

Each line is an independent proposed comparison. Envelopes represent opportunities for generation, pages represent given materials, and letter cards represent frozen answers. There is no measured value in the figure, and there is no preset which side wins.
For example, image search that article It is pointed out that new text information may also be added when building a map. If you want to test the effect of the edge, you must let the baseline without the edge also read the same material. Scale Extrapolation It is required to keep the largest several training sessions until the fit is revealed. The former needs to be level input, and the latter needs to isolate the predicted target. These two studies are worth doing, but only readers who have answer sheets, no drawing records or training curves are difficult to make up for key comparisons. This is the choice of the following three questions in this article.
Question 1: Give the starting point model several times, which questions will it recover?
It can be verified that reward reinforcement learning allows the model to update parameters based on feedback such as answer checking. To judge what changes it brings, it is usually necessary to compare the two checkpoints before and after receiving this training. An answer is the opportunity to give the model a question to get the correct answer. You can ask multiple samples, and if you give each question the same multiple opportunities, as long as one of them is passed, how many questions can be covered? This indicator is called pass@, in It is the number of candidates to view the most in each question. A reviewer who knows the standard answer can recognize hits, and real users may not be able to pick it from so many candidates.
Figure 2 of Yue et al. gives a specific reason. On several mathematical benchmarks, the reinforcement learning model is ahead in the small budget. After adding sampling, the starting point model can catch up and overtake. The sampling depth of each question set is not uniform. For example, this analysis of MATH500 generates 128 times per question, and AIME24 generates 1024 times. The right end of the curve therefore represents the range that the researcher actually sees. It may be very difficult to hit a certain question here, and there may be a correct output of extremely low probability. The two situations cannot be distinguished by explaining zero again. Yue et al
Whether the curve really crosses or not requires uncertainty. Chen Yang's preprints make pairing inferences about the pairing of the previous models of the same topic, and at the same time investigate the small budget lead and the big budget backward. In Figure 1, DeepScaleR is compared with its own starting point. The upper limit of a single answer is from 8K to 16K to 32K tokens. The visible crossing and statistically supported cross appear in different settings. Tokens are units that charge and truncate when the model generates text. The proportions of the two sides that are truncated in the same upper limit in the figure are also different, and the number of each question generated in each setting is also different. It reminds us to record unfinished answers; this graph itself cannot specify the appropriate length for each model. Chen Yang
What can be done. Select a pair of public checkpoints with a clear training inheritance relationship, freeze the questions, prompt templates, temperature and scorer, and use the development questions to adjust the answers. First, it is agreed that each question has a fixed upper limit of the number of samples and a single length, and the actual tokens are saved at the same time. Calculate at least one hit in the first and budget on the same batch, and list the questions that are both in the former, only in the former, only in the latter, and missed. The last group will also appear on the table, otherwise the problem will disappear out of thin air on the report.
If the post-training model is more stable for the first time, and the original seemingly unique success has been recovered by the starting point model many times, this batch of questions supports the interpretation of sampling efficiency. If there is a batch of questions that only hit the post-training model repeatedly, and a batch of similar questions reserved in advance is still the same, there is reason to study the newly added coverage under the limited budget. The stop condition is the pre-allocated questions and the generated amount after running. If the difference interval estimated by question pairing is very wide, or a large number of answers on both sides are cut off, the result is written as "this budget is still inseparable". Sampling all the time for the pursuit of crossing will make the end itself also affected by the results.
Question 2 Change the sentence that decides the answer, will the model follow it?
Assuming that the archives are written in the collection in Hall A, and later a temporary transfer order reads "Borrowing the exhibition to Hall B". Ask the collection place and ask the current exhibition place, the correct answer can be different. If you only change the name of the museum, and at the same time change the "collection" in the question to "exhibition", it is impossible to tell which information the answer has changed from. This is a hypothetical example. It shows that the most time-consuming part of material replacement is often writing questions, especially to determine which word controls the standard answer.
Longpre, etc. replace the entity in the question and answer material, and examine the model that the original answer can be answered correctly, it will output the original answer, the replaced answer, or other answers. Figure 3 shows that in cases where the training sample has been seen or closer to the training sample, the generative reader retains the old answer more often. Their "memory ratio" is divided by the number of original answers divided by the number of times the original answer and the number of replacement answers. "Others" are excluded from this specific ratio, but remain in the output distribution. Ignoring this, a system that often answers the question may also appear very decisive between the two named answers. Longpre et al.
What can be done. Select short materials with a clear source and can confirm the answer, and ask the model to "answer only according to this paragraph". Write an original version for each question, a version that replaces similar entities, and a control version that only changes irrelevant details. The three versions are asked in the new session respectively, and the answers when there are no materials are recorded as a reference. Add another cheap diagnosis: give it the evidence with the evidence alone. If you can follow the reading right at this time, and the full text will fail, you can give priority to checking the positioning and interference; if the single sentence is also read incorrectly, the failure on the full text cannot be attributed to the retrieval alone.
There are two distinct "successes" here. The two valid materials are answered correctly, and the original answer is maintained regardless of rewriting, and the model supports the use of designated evidence on such questions. All rewritings make the answer fluctuate greatly, and even the stability of unrelated words is a problem. Pre-contracted to report both the whole questions, the subset of the original correct answers, and abstention and other answers. If the substitute is introduced syntax exception, both answers make sense, first head the title surface defect and report the proportion. One round of independent checks still cannot determine the unique text answer, so disable this set of questions. Even if the model fully follows the fictional material, it can only prove that it executes reading instructions, and the real facts and cross-session knowledge update also require other tests.
Question 3: The two answer sheets only change seats, does the referee still recognize the same one?
Zheng and colleagues used the same GPT-3.5 model to generate two similar answers to each first-turn MT-Bench question, then had the judge compare them in both orders. With the default prompt in Table 2, GPT-4 gave a consistent verdict after the swap 65.0% of the time. Consistency means reaching the same outcome after the answers trade places. Always picking the first answer fails that test. The authors deliberately chose close calls, so 65.0% cannot stand for judging accuracy across all judges or questions. They subsequently took a conservative approach, declaring a winner only when both orders favored the same answer. Zheng et al
What can be done. Select a small group that can be scored independently from the public question, such as requiring two fields to be extracted from the text. Freeze two candidates, one complete and one missing item, and try to match the length; Seal the field answers, the referee only sees the questions, answers and scoring requirements. In addition to exchanging answer sheets, do the independent repeat of the original order, and keep the number of calls equal. This extra comparison is useful, because even if nothing is changed, a random referee may change the sentence. Every time, the location is restored to the identity of the answer, and the outcome, draw and cannot be resolved are recorded respectively.
If the transposition flip exceeds the fluctuation of the original sequence repeated and is always biased to the same position, the sequence can explain a part of the ruling change. If the answer to the same missed item wins steadily in both positions, the grading standard cannot be saved no matter how high the consistency of the transposition is. First report how many questions you have selected in the question that can be independently checked, and then look at how many are hardly divided by hard points in the approximate draw. In actual development, the table will be sealed after the pre-agreed repetitions are used up; if even obvious omissions are continuously judged, the referee should be stopped to discuss subtle style preferences. The two answerers did not write another word from beginning to end, and this experiment is positioned in the measurement link.
Summary The material in hand decides where to start
If all you have is two batches of saved answers, the third question is the easiest place to start. If you can edit the input material, the second turns “it looks like the model used the source” into a prediction you can falsify. The first demands more: you need to know the model’s training starting point and keep failed outputs for each question. The three studies differ in models, training histories, test splits, compute budgets, and scoring units. Curve coverage, entity selection rates, and judge agreement cannot be combined into a capability ranking.
I will ask the students who are ready to open the question first: "If there is any kind of result, I will give up the current explanation?" Even if a wide range is left in the experiment, the next researcher can know whether to add questions and revise materials, or change the scoring method.
References
- Yue et al., NeurIPS 2025. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
- Chen Yang, arXiv Preprint v3, 2026-11-03. When Pass@k Crossovers Are Real—and Why: Paired Inference and the Markov Kernel of RLVR
- Longpre et al., EMNLP 2021. Entity-Based Knowledge Conflicts in Question Answering
- Zheng et al., NeurIPS 2023, Datasets and Benchmarks. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Apps by the author
I also build small tools for everyday life. Here are two of my apps:
- Daily Expense Journal: Track your income and spending, plan a budget and set savings goals. View on the App Store.
- One Small Action: Pick a small action that fits the time and energy you have today, then record how it felt and what you noticed. View on the App Store.
Find more articles and projects on my website: zffjzz.top. I’d love to hear how the apps work for you and what I could improve.