AI Frontier Observation: People, Breakthroughs and Evidence | Interpretation Date: November 8, 2026 · Data Deadline: Beijing Time 20:30:59

A program that does two-digit addition, after moving from the GPU to the TPU, handed over the results that are enough to pass the test. The problem lies in the test paper: the migrated program is tested with training data. Princeton University Linrong Cai et al. in the preprint XStack This case was recorded. They asked seven programming agents to migrate twenty machine learning code bases, and the best performance passed fifteen. What is more worthy of the ranking than this ranking is: if the program can run and the number is correct, how can a laboratory judge that what he got is still the original experiment?

The paper tape and the inspection template between the two computing devices symbolize the verification before and after the migration.

Schematic diagram of original concept; do not show actual hardware or experimental results.

Start with a code that is already available

Usually doing code evaluation, the starting point is often a mistake, and the end point is to pass the test again. The starting point of XStack is just the opposite: the original program can work, and it is the machine in the hand of the laboratory. The agent gets a pinned version of the PyTorch repository, and it is necessary to hand over the version that can run on TPU with JAX. PyTorch and JAX are both software frameworks that describe numerical calculations; in addition to the framework, the hardware to perform the calculations is also replaced here, and the training environment is erected on a clean machine. Authors’ repository List the source code version corresponding to each task, and how the submission should be installed, trained and exported.

In the original program, there may be teaching examples, quick test runs and the main experiment of the paper. Running through a small configuration can indeed prove that some codes can work, but you will get the same conclusion without explaining the complete training. The author asks for a set of re-running deliverables: switch to another machine that has been emptied, run from the preparation environment to the generation of indicators, and finally put the parameters obtained from the training back into the original model. In this way, the temporary files left on the development machine cannot complete the experiment for the submitter.

First ask if the original version can pass your own line

The most direct way to score the migration version seems to be copying the numbers on the homepage of the original project. The character generation item makemore illustrates the risk of this approach. Press Paper checks the baseline, the loss listed in the original description is about 1.92, the lower the better; the team ran eight times with an unmodified source code on its own GPU, with an average of 1.989 and a standard deviation of 0.017. If you stick to 1.92, a transplant that faithfully retains the original behavior may also be convicted of disqualification. The team therefore set the threshold not higher than 2.023, that is, the mean plus two standard deviations.

Here is the ruler's ruler. Eight runs gives the range of fluctuations under this set of specific versions, but cannot set a general tolerance for all tasks. The addition question at the beginning uses another ruler: the questions left by the five hundred paths, at least answered at least four hundred and ninety-five. Allowing the wrong five can accommodate the training difference, but only if these questions are indeed left outside the training set. A threshold must be explained at the same time, numbers, data and calculation methods. If you break away from any of them, you may judge different experiments as the same thing.

Give a second chance, but freeze the realization that has been handed over

A voice repository can support more than one set of data. If the agent faithfully migrates the training program, but chooses a different experimental configuration from the referee, the direct judgment will count the "migration failure" as "migration failure". For this, Evaluation protocol The information is given in two times: first hide the scoring threshold, and let the agent choose how to run it by itself; after entering the second stage, the experiment to be run is explained.

This restriction gives the second attempt a diagnostic value. If you change the entrance, you can pass the level, indicating that the previous results are at least partly affected by the operation mode; if the model must be rewritten to pass, the problem will be deeper into the realization. It is not enough to check only which files have changed: the entry script may also rebuild the code that should have been frozen at startup. According to the author's statistics, the results of 28 numerical values in the second stage were rejected by the referee, of which 21 were precisely because of this approach. The seemingly cumbersome freezing clause determines what the second pass can be said.

How to read 15 pass repositories

After both stages, Claude Opus 5 passed on fifteen repositories and GPT-5.6-sol on nine. Pair all seven agents with all twenty repositories and 44 of the 140 combinations passed. The first denominator tells us how one agent did on this set. The second pools the outcomes across systems on those same projects. Neither gives the chance of success on an arbitrary new project. The team screened more than 200 candidates for accessible data, source code that could be pinned, and a full training run that would finish within a limited time. Those requirements make the results easier to check, but exclude many of the large projects that labs find hardest to port. Appendix A and Table 4 can be read in comparison.

Once the numbers met the target, the team had a model judge inspect the run logs and code changes to check that the specified experiment had actually produced them. That was how it caught training data being passed off as test data. The judge can make mistakes too. The authors reviewed some rejections, which falls short of an independent audit of every decision. The first version of the preprint was submitted in the early hours of November 6, Beijing time. I did not rerun the evaluation or find an independent replication in this search.

The next most convincing material will be another group of researchers who blindly review the critical case on the new repository: keep the original data division, the training log and the weight of the export. If the submissions that meet the standard but run the wrong experiment can still be identified stably, the laboratory will not only get a model transcript, but also a set of acceptance methods that are worthwhile to use when changing the machine.

References


Apps by the author

I also build small tools for everyday life. Here are two of my apps:

  • Daily Expense Journal: Track your income and spending, plan a budget and set savings goals. View on the App Store.
  • One Small Action: Pick a small action that fits the time and energy you have today, then record how it felt and what you noticed. View on the App Store.

Find more articles and projects on my website: zffjzz.top. I’d love to hear how the apps work for you and what I could improve.