
The cover is a concept illustration generated by AI, not a paper architecture, real device or experimental results.
Interpretation date: 2026-11-03
Data deadline: 2026-11-03 20:34:19 (Beijing time, Asia/Shanghai)
At 02:56 in the morning of Beijing time this morning, researchers such as Catherine Li and Jingchu Gai at Carnegie Mellon University submitted a essay version of Muon and AdamW to arXiv. Team report: As pre-training data increases, the pros and cons of the two optimizers may be reversed after fine-tuning. This work is worth reading, because what researchers ultimately need is not the model with the best fraction at the end of training, but a model that can continue to learn without excessively losing existing abilities. Paper and revision history
The new is the paper version, the problem has been publicly discussed
Let's make the time clear first: the submission time of arXiv is 18:56:43 UTC on November 2. The official website of the MOSS seminar has included related topics October 9th Agenda, the two optimizers in the title are different from the current version. Therefore, this article interprets the recent arXiv version, and does not describe the research conclusion as the first public today; the actual date of the experiment cannot be inferred from the submission time. agenda
A browser verification challenge on the linked OpenReview page prevented me from reading the review record. The workshop presentation and the full version may have undergone different kinds of review. The experimental results below are the authors’ reported findings. I did not rerun the experiments.
This time distinction also affects news judgment: it is not necessarily a method that is worthy of a sudden "overturn", but whether the public material has given us the opportunity to re-examine the original selection criteria that were too simplified.
What has the optimizer changed?
An optimizer is the rule that adjusts model parameters during training. The original AdamW work decoupled weight decay from the update driven by the loss, a reminder that a seemingly small change to an update rule can affect how well the resulting model generalizes. AdamW Original Paper
Muon approximately orthogonalizes the updates to hidden-layer weight matrices. Intuitively, it changes the relative contribution of different update directions. Simply increasing every step size would have a different effect. The original introduction suggested that this might strengthen weaker directions, an explanation that still needs testing across settings. Muon Author's Notes
There is also a detail that is easily masked by the name: the official implementation suggestion of Muon, and other parameters such as embedding and output head still use AdamW. Therefore, "Muon vs AdamW" is not necessarily that each parameter uses two completely different algorithms. When reading the comparison experiment, you should first clarify which parameters have changed, and then discuss the outcome. Official implementation note
Put together academic performance and forgetting price
The new paper studies OLMo-style models with roughly 30 million to 600 million parameters, fine-tunes them on four datasets, and compares loss on the new task with loss on the pretraining data. The authors report an advantage for Muon with little fine-tuning data. As the data budget grows, AdamW can offer a better trade-off between learning and forgetting, even while Muon retains its advantage on pretraining loss. Experimental Settings and Results
This comparison can be understood with a hypothetical example: two students have completed the professional course, and one has a higher professional grade, but has forgotten more about the basic course; the other has a slightly lower professional grade, but the basic knowledge is well preserved. Just looking at professional grades, you can't answer who is more suitable to continue to undertake different tasks. This example is only an explanation of the evaluation objectives, not the actual data in the paper.
The same is true for models. Increase the learning rate, which may adapt to new materials faster, and at the same time change the old behavior more significantly. A reasonable comparison should allow both parties to adjust the learning intensity, and then observe how many original performances are retained at the same level of adaptation. Otherwise, a seemingly beautiful "upgrade" may just choose a more radical training point.
Where does the evidence support?
The paper also uses linear associative memory to study the stability of disturbances when disturbed, but its single-step theory compares gradient descent and idealized Muon, which cannot be directly regarded as the theorem of the entire AdamW process in real language models. theoretical settings
There are two boundaries that need to be preserved here. First of all, the loss of pre-training data is an evaluation indicator, not an item-by-item inventory of all the knowledge of the model; the loss is lower and the answer is more reliable. Second, the trends observed within one scale also require larger models and other task tests. Especially when the deployment requirement is long text reasoning, code modification or multiple rounds of interaction, one proxy indicator cannot be used instead of complete ability evaluation.
No confirmation of independent reproduction was found in this search. The abstract of the aforementioned paper belongs to the same study; the official Muon repository provides a background algorithm, and it is not an independent verification of this comparative experiment. Readers should use the current conclusion as an experimentally supported research signal and continue to wait for external inspection.
What should a scientific reader look at next?
The judgment of this paper is that the most valuable inspiration of this research is to move the end point of the optimizer backwards: if the model must be fine-tuning in the future, the adaptation and retaining performance after fine-tuning should be included in the pre-training decision.
Three questions remain open. Can an independent team reproduce the reversal with fixed data and a fixed tuning budget? Does the crossover occur at a similar point on tasks in other domains? Would the practical choice stay the same after accounting for training time and tuning costs alongside final task quality? Failed configurations and checkpoint selection rules also matter. If we see only each method’s best checkpoint and never learn how many attempts it took to find it, we cannot judge how reliably its advantage can be obtained on a limited budget.
The value of small-scale research is that the variables can be disassembled and checked; its external thrust depends on the follow-up evidence. At this stage, there is no need to rank two optimizers for a permanent ranking. A more useful action is to add a column to your own evaluation table: after learning new tasks, which original abilities are still retained?
References
- Li et al.: Paper abstract and arXiv submission record
- Paper V1: Experimental Settings, Results and Theoretical Analysis
- MOSS@COLM2026 official agenda
- Keller Jordan: Design Instructions for Muon, 2024-12-08
- Muon official implementation and parameter group description
- Loshchilov、Hutter:Decoupled Weight Decay Regularization
Apps by the author
The following is the author's own product promotion. In addition to writing a technical blog, I also independently developed two daily gadgets: Daily Expense JournalUsed to record revenue and expenditure, budget and savings objectives;One Small ActionHelp you choose small actions according to the current time and energy, and record how you feel when you are done. Welcome to my own needs, and welcome to visit myPersonal website, read more technical articles and personal works. These products are not technically linked or endorsed by the research in this paper.