
Deadline: 2026-11-06. In this article, we read three first-hand studies, and did not run model experiments.
Summary · Look at what the two lists are in
If you give a group of assistants to correct the homework, judge the task to be successful according to the expert records in the morning, and hand over the same batch of homework to the automatic referee in the afternoon. The first system in the morning, the second in the afternoon. If the students do not re-answer the question, will you still write the first place to "student regression"? In Huang's analysis of the public agent trajectory, there is a specific answer to this question. The expert label selects Claude, and the GPT-4o-mini referee selects the GPT-4o. At the same time, the author adjusted the scope of the task and the weight of various tasks, and the success rate of several systems was reduced, but the rankings remained in the original order. Huang, Section V-C
The two lists can be calculated correctly, or they can answer a valuable question. Where the disagreement occurs, it depends on whether it can be chased along the same question: what the model sees, what answer is handed, who judges it correctly, and how much weight this question accounts for in the total score. The following three materials are selected multiple-choice questions, cross-task summary and agent logs. They allow us to observe the answering process, summary rules and label sources, and only a portion of the changes can be re-examined with the already saved answers.
The inversion in multiple-choice questions occurs before handing in
MMLU, studied by Alzahrani et al., is a set of interdisciplinary multiple-choice questions. It is enough to look at the two rows of the paper table 2: without giving examples, the default accuracy of Yi-6b and Mistral-7b is 61.12% and 59.56%, respectively. The researchers moved the correct option for each question to C, and the accuracy of the two became 58.59% and 63.84%. For the same pair of models, the former leads by default, fixed to C and the latter leads. The knowledge points of the question test can be kept as it is, but the position and answer letters in the input have changed at the same time. There is a process of re-proposing questions to the model, and this intervention cannot be reproduced by modifying the score table. Alzahrani et al., Table 2, Section 3.1
Why emphasize "simultaneous"? Imagine the model prefers a letter, and even if the correct content is originally selected, it may be attracted by the transposition mark. Another possibility is that it more often selects a certain position in the sequence. Put all the correct answers to C, and these two explanations will make similar predictions. The author has also experimented with disrupting symbols and options, and pointed out that only the words and marks are moved together, and it is still difficult to open the position preference and symbol preference; the further investigation in Figure 6 covers only three disciplines. Table 2 is enough to give examples of ranking flips. To say that one of the models is "lost knowledge", there is still a lack of independent measurement. Alzahrani et al., Section 5.2, Figure 5–6
This example cannot be read upside down as "changing the prompt will definitely change the top of the list". If the same paper replaces the example with an example of other disciplines, or adjusts the end of the instruction, the ordering changes will be much smaller. Fix all the correct answers, suitable for amplification of a certain sensitivity; the position on the natural test paper does not have to be so extreme. If the actual goal is to answer the questions under the usual arrangement, this result reminds us to report the arrangement of the report. If the actual goal is to answer correctly under various arrangements, then the weakness exposed after the transposition is the ability to measure. Alzahrani et al., Section 5.4
The score has been fixed, how to summarize is still a research problem
Look at a transcript, each row represents a task or a discipline, and each column represents a model. After each grid is filled out, the researchers still need to decide how to read this form. Each question has the same power, and the subject with many questions has a great influence; first take the average in the discipline, and then let each subject have the same power, and the influence of the primary discipline will increase. The two weights correspond to "random questions randomly selected from the question bank" and "select the subjects randomly first, and then draw the questions in them". If you want to understand the performance of students' cross-disciplinary responses, and the unintentional number of differences in the collection cost of the questions, you should not quietly decide the research goals for you.
We can use a hypothetical situation to identify the role of weight: A leads the text question, and B leads the graphic question. Gradually increase the share of graphic questions, B may overtake. Conversely, if A is ahead in each category, and both sides use the same group of non-negative weights for each category, the weighted average cannot allow B to overtake. When encountering the latter report, it is worth going back to check whether each model missed different questions, whether the denominator is the same, and whether the average score is calculated or voting by ranking. This inference only uses the definition of weighted average, and does not need to make any assumptions about the internal mechanisms of A and B.
Gordienko and colleagues’ preprint offers another way to read a leaderboard. Starting from actual task scores, the researchers make an assumption deliberately favorable to the challenger: give the target model full marks on every selected task. For each model, how few tasks must change for it to have a shot at first place? Across 24 BIG-Bench Hard tasks and 4,507 model entries, the median minimum was 13 tasks when scores were aggregated by arithmetic mean and 22 under mean win rate. Mean win rate measures how often a model beats other entrants across tasks. The two rules reward improvements on selected tasks differently. Gordienko et al., Sections 4–5
There are two easy-to-mix actions here. The previous subjects weighted to maintain various grades; this paper actively changes some grids in order to measure the sensitivity of the rules to the local increase, and then compares the different summary rules. The author did not really train these models, let alone prove that the contestants on a certain list have seen the test questions. It is difficult to push a rule to the top of the list in this way, and it may still be far from the reader's ability to care. Especially when the reader cares about the serious mistakes on the low-frequency task, how many competitors can be better than to tell him the cost of the mistake. Gordienko et al., Sections 5–6
Let the two referees read the same trajectory
back to the beginning of research. A single "answer" of the agent is a string of operations, and the same basic model may also be matched with different tools, prompts and retry strategies. Huang's AgentRewardBench record contains four configurations and 1302 trajectories after deduplication. Llama lacks VisualWebArena observations. If you fail to fill in the blanks, you will be punished for items that are not participating; if each only averages the items you have run, several systems are answering different test papers. The authors choose the three benchmarks that are jointly covered, so that they are equally entitled when summarizing. At this point, all configurations are down and the order is maintained. This supports a narrowed comparison, and how Llama will still be open on missing projects. Huang, Section V-C, Figure 3
Changing the judge in the same section gives a more revealing comparison. Expert records and automated judgments can be matched to the very same action. With the trajectory fixed, a different success label can still change the rankings. For each disagreement, ask whether the action succeeded but the environment check missed it, or whether the answer looked complete while the action failed. One produces a false negative, the other a false positive. You need the task and its trajectory to tell them apart. The judge’s name alone cannot settle who was wrong. Expert labels serve as the reference here. Applying the method elsewhere requires a way to verify the actual outcome in the new environment. Huang, Section V-C
There is also a result that is easily swallowed by the title: both reports are out of order, but the data is not enough to distinguish the two systems. Huang took out 45 pairs of submissions in the top ten of the SWE-bench original list, and compared the weights according to the repository, using two percentage points as the actual meaningful gap. Under the rule of uncertainty, 5 pairs are stable, 1 is sensitive to the target, and 39 is insufficient for evidence. This judgment is limited to the threshold set by the top ten selected and authors. There may be differences in the two items that are insufficient; to claim that they are close enough, the reasonable range of the gap must be completely within the predefined equivalent interval. Huang, Section V-B, Figure 2
What changes can be explained together?
| operations in research | what is reserved | What can be judged based on this |
|---|---|---|
| MMLU Move Correct Options | Knowledge content and test model of the same batch of questions | The comparison direction is sensitive to presentation protocols and cannot be individually positioned knowledge changes |
| BIG-Bench Hard Simulation of local full marks | The same set of starting scores and candidate models | The number of tasks required to rush to the top of the summary rules changes, and the real training cost is not measured |
| AgentRewardBench Replacement Success Tag | already have a track | The referee chooses the top list that can change the estimation of the point, and still needs to check the label error |
The three studies share no common starting model, training process, data split, or inference budget. The first had the models available at the time answer the questions. The second made counterfactual changes to existing leaderboard scores. The third reexamined published records of system runs. Accuracy on city-knowledge questions, the number of tasks needed to reach the top in a simulation, and tool-operation success rates have no common unit. What we can compare is where the intervention occurred. Mapping what changed and which explanations still fit the observations is a better guide to choosing a control for the next study than putting the three numbers in a column.
A small experiment that can distinguish interpretation

The two branches in the figure are the suggested independent recalculation, and the arrows indicate the analysis order. There are no papers measured in the figure.
First choose a pair of public models with a clear version, and only study one ability, such as the correct selection of a given option in multiple disciplines. Use public evaluation questions that are not selected in the rule selection, keep the title number, answer text, options order, original output and parsing results according to the subject. Both sides use the same prompts, number of examples, upper limit of generation, and number of attempts. It is also necessary to prescribe whether the research is the basic model or the complete system with the answer parser. If you only get the log of the complete system, the conclusion is in the system as a unit; a parser can tolerate the extra description, which is enough to change a part of the "wrong answer" record.
The first round only operates saved answers. Fixed the same question-by-question score, so that each question has the same right and each subject has the same right, and observes how the difference between the two models changes with the task composition. Re-fix the task weights, and the same answer is more strict with the matching rules of letter matching and pre-defined answer extraction. Ambiguity should be preserved for outputs with multiple conflict answers, and do not pick a favorable explanation for one of the models after seeing the top of the list. If the gap is only flipped during re-weighting, you can list the direction of pulling the total score by subject; if you only flip in the stitching, you can directly show which original answers have changed the label. Neither of these diagnostics need to be called again.
The second round changes the input. Move each question option according to the pre-generated replacement, and move the correct answer mark synchronously, and the models on both sides will be re-answered. Record whether the same text is still selected before and after the same model and the same model, and check the order preference and letter preference at the same time. Separate this round from the old answer with heavy judgment, and the reader can distinguish "the model has handed another answer sheet" and "the referee makes another judgment on the old answer sheet". To study the version upgrade, add old and new checkpoints, and run the same protocol in full across each version. Otherwise, the version and the question method are replaced together, and the difference cannot be attributed to any of them.
Finally, the gap is delivered together with the range of uncertainties. Multiple arrangements come from the same question, and the sampling should be based on the original question, and the answers of the two models and the corresponding arrangements should be reserved together; otherwise the repeated variants will appear like many independent evidences. It is worth distinguishing how big the gap is written in advance, and which task distribution is a research goal. If the gap is in the same direction and exceeds the threshold under all scheduled settings, there is a reason to report a solid lead within this range; if the gap under different targets is reliably reversed, the target should be named separately. If the interval still allows two meaningful directions at the same time, stop the research at "undecided for the time being", and point out which disciplines the next batch of independent questions should be supplemented from.
Summary
When reading the list, you can first complete a sentence: "Under these questions, this set of answering conditions, and this scoring rule, how much A is relatively ahead of A?" If the input or model version has also changed, it needs to be re-paired to run. It is not uncommon to choose different winners from the two lists; it is worth explaining which step is enough to change the direction of the comparison, and after changing to a still meaningful goal, the existing evidence can support how strong the conclusion is.
References
- Alzahrani et al. (2024), When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards, ACL long text.
- Gordienko et al. (2026), How Hard is it to Rig a Benchmark? A Social Choice Analysis of Leaderboard Robustness, arXiv v1 preprint.
- Huang(2026), What Does an LLM-Agent Leaderboard Rank Actually Compare?, DSAA short text, this article is based on arXiv v1.
Apps by the author
I also build small tools for everyday life. Here are two of my apps:
- Daily Expense Journal: Track your income and spending, plan a budget and set savings goals. View on the App Store.
- One Small Action: Pick a small action that fits the time and energy you have today, then record how it felt and what you noticed. View on the App Store.
Find more articles and projects on my website: zffjzz.top. I’d love to hear how the apps work for you and what I could improve.