AI Frontier Observation · Interpretation Date: November 16, 2026 · Data Deadline: Beijing Time 20:28:16

The brick chessboard is side-by-side with a matrix board that is being adjusted, indicating the difference between writing from zero and local patching

Original concept illustrations do not represent experimental devices or measurement results.

The rules for making a brick for a language model, let it directly fill in the parameters of another neural network, and do not play a game first. Give it a ready-made network and let it judge: Can the current problem be repaired by minor changes, or is it beyond the authority of this modification? Aayush Marasini, Barsat Khadka and Nick Rahimi of the University of South Mississippi have put these two types of questions in the same set of reviews. The team reports that the top-performing model can write strategies that can be played, but often cannot be sure when encountering a second problem. Both tasks are called "operation weights", and the evidence required is very different. preprint record It shows that the first draft was submitted in the early morning of November 14, Beijing time; the experiment date recorded by the author is November 11 to 13.

Catch the ball and find out where it's broken

Weights are a set of numbers used by the network to turn input into output. Take the bricks as an analogy, a simple strategy is to compare the horizontal position of the ball and the baffle, and then decide which side to move to. If this rule has been considered, the next question is whether "comparison" and "movement" can be written as operations that the specified network can perform. Even if you can't try it before submitting, this question gives the understanding the problem: it knows what to do, and it can start from a blank network to arrange each component.

There is no such freedom for maintenance issues. If a language model keeps repeating in reasoning, it is not difficult to imagine a few words after the change, but the difficulty is that it can still complete the question after jumping out of the loop, without damaging the original normal answer. What's even more difficult is that the topic sometimes requires the recovery of a fault-blocked ability, and sometimes it is required to write something that the model has never learned. Methods of the paper Put the difficult but solveable tasks into the feasibility judgment questions. Refusing to revise and cannot pass the customs; the "revised not good" here is only for the prescribed modification scale and ability reservation requirements, and it is not mathematically proved that no method has any solution.

How to execute "No Trial Run"

When reading the weight, send a set of inputs to the target network to see the result, which is called a forward pass. If the continuous trial input is allowed, the agent can guess what the network is doing from the output, and there is no need to identify the internal parameters. Conversely, it is not appropriate to prohibit all expensive operations: breaking a weight matrix to observe the structure may be more costly than a test run, but did not ask a question to the network.

The authors therefore separate the agent from the final scoring program. Static diagnosis tasks prohibit trial runs of the target network, while some other tasks allow a few probes. The agent can analyze the weights numerically, and the system tracks operations on them to check whether they amount to an unauthorized trial run. It logs probes and anomalies for later audit. This distinction lets readers ask whether a successful run relied on the parameters alone or drew on limited feedback about behavior. Auditing has a cost, too. A flagged run scores zero pending human clearance, so a zero may mean there was no submission eligible for scoring. To distinguish a bad patch submitted on time from a patch that never made the deadline, check the individual run record to see where the points were lost. Public repository and scoring notes

Two denominators in a score sheet

The most striking gap is within a single model. According to Table 1 of the paper, Claude Opus 5.5 averaged 0.829 on the six tasks that required writing weights from scratch, but only 0.492 on the five feasibility judgments. The other two Claude models also scored around 0.5 on feasibility, which the authors liken to a near-chance baseline. The first figure is no measure of how often the model won at Breakout, and the second cannot tell us how many of the five questions it got right. The table gives composite scores based on each task’s payoff, with deductions for collateral damage, edits, and probes where applicable.

The overall score involves another averaging step that is easy to miss. Scores are averaged within each task type, then each of the five types gets one fifth of the weight. Opus ran all 42 tasks for an overall score of 0.535. GPT-6.1-sol scored 0.568 on 38 tasks, omitting four reasoning-loop repair tasks. Subtracting the scores directly changes both the contestant and the test. On the 38 tasks they shared, Opus scored 0.546. Even then, the 0.022 gap describes only these runs. The experiment used one task seed, few model samples per task, and somewhat different tools for running them.

Next time the measurement, which condition should I replace first?

The most worthy of additional experiments are fixed the same tool and the same detection budget, so that the model can handle paired faults: the external performance is similar, one case can be repaired, and the other case cannot be repaired under limited conditions. How many revision tasks are mistakenly rejected, and how many unrepairable tasks are fortified. For the researchers, the consequences of these two errors are different: mistakes will be lost to a valid patch, and strong corrections may destroy the original abilities that were originally available. Change the task seeds to see if the judgment is stable. It is better to answer "Can it be allowed to take care of maintenance" than the small increase in the total score.

When checking the results, you also need to ask which version is running. Author's recalculation description Two things are distinguished: recalculate the original table with the saved successive records, and re-run the model with public software. The public edition will remind the agent to pay attention to the upper limit of the round and submit it as soon as possible. This prompt may change the score. The evaluation was not run this time, and no independent reproduction of this result table was retrieved. The existing evidence is enough to make people take the practice of directly writing weights seriously; as for handing an off-the-shelf model to it for repair, it depends on whether it can repeatedly recognize the one that should not be done.

References

  1. WeightBench preprint: author, abstract and submission history
  2. WeightBench paper full text: experimental design and table 1
  3. Author Code Repository: Tasks and Preliminary Achievements
  4. Author's re-calculation description: experimental date, zero score rule and version difference

Apps by the author

I also build small tools for everyday life. Here are two of my apps:

  • Daily Expense Journal: Track your income and spending, plan a budget and set savings goals. View on the App Store.
  • One Small Action: Pick a small action that fits the time and energy you have today, then record how it felt and what you noticed. View on the App Store.

Find more articles and projects on my website: zffjzz.top. I’d love to hear how the apps work for you and what I could improve.