AI Frontier Observation · Interpretation Date: November 5, 2026 · Data Deadline: Beijing time 20:34

A programming agent took about an hour and forty minutes to improve the compression scheme. Small-scale testing supports this change, and it can be put into a complete model, but the old solution is better. The agent finally returned the old version. This detail comes from the submission of Xinrui Zhong, Zhifeng Li et al. from Carnegie Mellon University CompressionBench preprint. At 2:14 a.m. on November 5th, Beijing time, the first edition of the paper was submitted to arXiv. It lets people see one step that is easily covered by the ranking list: when the two experiments give the opposite suggestion, why should the agent decide which version to leave?

Original concept illustration: partial engraving under a magnifying glass, Yamagata synthesized with a multi-layer printing plate

The cover is an original schematic diagram, which is a metaphor for the difference between the local precision and the whole experience.

Eight hours later, hand over a model that can run

Quantization stores weights using a small number of discrete values, so the model takes fewer bits to represent. The difficulty is that the same byte savings can have very different consequences. Some parameters tolerate coarse values, while small changes to others alter the output. The agent has to decide which parameters need the precision and write code that can read the resulting representation and use it to generate answers.

The team arranges six questions for the six programming agents: three source models, and each model has a tight storage limit and two gears. Design, achieve and test for eight hours each time in a prescribed environment. For example, the tight file of Qwen3.8 27B is 8 GiB; the model storage of the submission package on the disk and the model storage of the resident graphics card of the AT Inference Time must respectively meet this upper limit. Therefore, when the file is small and loaded, it is secretly unfolded into a whole high-precision weight, and the volume cannot be handed in. This is also not an upper limit promise for all temporary video memory during operation. Author's Task Description

Which local advantages will disappear on the whole machine?

Try a new set of parameters, and compare it with a matrix first. Compare the weights before and after compression, and the smaller the average error, the more loyal it looks. However, when the actual input passes through this layer, the degree of use in all directions is not the same. A direction that is very small but is often amplified, which may affect the output more than many insensitive directions. If the multiple layers are connected again, there is no reason to keep the ranking of local errors intact.

The paper has made a discriminating control: the researcher changes the weight representation under the similar code size, and then uses the same high-precision to calculate the weight after running the reconstruction, and try to separate the influence of the numerical approximation and the underlying realization. A scheme that selects representative values by clustering is closer to the original weight, but in the whole model test, the confusion has increased. Here the confusion measures the compression model to Subsequent text generated by the original model What a surprise; for the same batch of text, the lower the probability that it allocates to the actual tokens that actually occurs. Section 4.2

The opening example shows the choice in practice. On Qwen3.5 under a tight budget, Opus tried allocating just one bit to some expert networks, then reversed the change after testing the full model. Across 26 runs the team found 79 candidate comparisons it could pair up. Whole-model feedback reversed the local ranking in 17 of them. The denominator includes only recorded, matchable comparisons, so the fraction cannot predict how often an arbitrary local experiment will mislead. Cheap tests can screen ideas, but before discarding an old version it can pay to retest the whole model. Each retest leaves less time for the next revision. A reversal can rescue a candidate and warn the agent that relying on the same local measure may lead it astray again.

What can whole-model perplexity miss?

Even if the entire model has been tested, the next word is predicted above, and the model is still two questions after writing an answer from scratch. The former sees a fixed text at every step, and the latter has to go down the content he just wrote. The first step is to select different words, and what is encountered later may be a path that the original test has never been rated.

The paper gives a concrete example. As Opus compressed Qwen3.8 to 8 GiB, perplexity improved over several training versions, yet the number of math questions answered correctly fell from 44 out of 50 to 36 out of 50. Coding and knowledge question answering improved over the same period. These were versions from one search, so the result cannot establish that compression always hurts math. It does show why acceptance testing needs scores for each task: gains on two tasks can hide the loss a user cares about most. Paper Figure 7 and Section 4.3

After the 29 copies are handed in, what else should I ask?

After the deadline, the team checks the frozen submissions with another container. Arithmetic, code, and factual Q&A each draw 50 questions, and the three average Accuracy must retain at least 80% of the average score of the original BF16 model; this is a relatively threshold, not every 80% of the answer is required. In addition to the storage, execution and compliance checks, the authors report 29 submissions of 36 submissions. The decoding speed is reported separately and does not enter the reward. If the reader intends to choose a compressed version, it also depends on the speed column: the same 12 GiB file in Qwen3.8, Sonnet's scheme outputs 16.7 tokens per second when a single request is decoded, the control released quantitative version is 74.6. If you can install, answer correctly and answer in time, you need evidence respectively. Paper table 1

Authors’ repository The entry for task definitions, acceptance procedures and result tables is given. The program and the search process are separated and cannot replace the third party for reproduction; this article has not run the experiment, and this search has not been found independently. What I want to see most next is an experiment that can write down the criterion in advance: a fixed eight-hour budget, replace some of the local search time with a full-mode re-test, and then compare the final delivery on the issue that is not used in the selection plan. If in the new model and repeated repetitions, we can still submit a few "partially more accurate and actually worse" answers in this way, and we are more confident that we can learn that in addition to compression skills, the judgment of where to spend the experiment time is also judged.

References

  1. CompressionBench: Abstract, Author and First Edition Submission Time
  2. Full text of the paper: task agreement, table 1, local diagnosis and generation test
  3. Infinai Lab: Method Explanation and Research Track Diagram
  4. Author code repository: task, acceptance tool and result table entry

Apps by the author

I also build small tools for everyday life. Here are two of my apps:

  • Daily Expense Journal: Track your income and spending, plan a budget and set savings goals. View on the App Store.
  • One Small Action: Pick a small action that fits the time and energy you have today, then record how it felt and what you noticed. View on the App Store.

Find more articles and projects on my website: zffjzz.top. I’d love to hear how the apps work for you and what I could improve.