
- Date: 2026-10-23
- For graduate students, early-career researchers and research engineers
- Deadline: 2026-10-23. Retrieve the relevant work for nearly twelve months and retain the necessary classic studies. This article is a deep reading of the paper, without executing a model experiment, and does not claim to be fully covered.
Summary: First ask what is saved
After the model learns the new task, the old question is still answered correctly, is it possible to complete the continuous learning? If the answer comes from a readily available notes, how can we describe progress? Conversely, if the old question is suddenly wrongly answered, does it mean that the original knowledge has been erased?
The judgment of this article is: Reducing forgetting is an important result, but it is necessary to report together with new task learning, cross-task migration, and memory access conditions, in order to judge what the system has learned. The callable ability in the parameters and the effective ability of the entire system are worth studying, but the two cannot be replaced by each other. The following five studies provide evidence for playback, behavioral recovery, hidden state memory, adequate sample storage and external empirical reuse.
In the previous article, I paid attention to where it is easier to correct mistakes. This article has changed to the ability of continuous tasks. When the old score is saved, has the new rules learned, and what will happen after the specific historical support is removed?
"Unforgettable" and "Learn to Learn" are two questions
Continuous learning means that the system receives data or tasks in turn, and uses past experience to adapt to subsequent tasks. It needs to be distinguished here Stability and Plasticity: The former is whether the performance of the old task can be maintained, and the latter is whether the new task can continue to be learned well. All the parameters are frozen, and the parameters can be rewritten by the new gradient, but this point cannot be used to prove that the system will absorb new experience.
"Memory" also has different locations. Playback is to reuse old samples during training, or use old task pseudo samples generated by the model; retrieval is to send the stored content back to the model when answering; hidden state memory compresses history into vectors, rather than preserving text as it is. They can be combined, and cannot be judged by the method name only to determine where the learning occurs. LAMOL and M+ The first two training and storage relationships are shown respectively.
A hypothetical example can illustrate the problem: the system first learns a classification rule, and then learns another rule. The scores of the old questions remain unchanged, maybe because the old rules are retained, or because the system has not learned the old rules well, or Retriever has found ready-made examples one by one. Only the first explanation is close to what we want to prove; the latter two still need to be checked separately. This example is not a paper measurement result.
Therefore, this article does not regard "real learning" as an unobservable psychological state, but disassembles it into a verifiable requirement: under the same support conditions, it can maintain its old ability, can use new experience to improve new tasks, and can migrate under pre-specified changes. When it is allowed to check memory, it is necessary to make it clear that it belongs to the system with memory.
Playback can keep performance, but old experience is not without cost
LAMOL makes the same language model serve as a problem solver and sample generator: generate old task pseudo samples before entering the new task, and then mix with new data. Its GPT-2 experiment is displayed in multiple language tasks and task sequences, and the generation of playback is better than the direct order fine-tuning; the task-specific generation flag is used to alleviate the problem of the gradual scarcity of early task samples. Original §3, §4.2, Table 3
The "no extra memory" here should not be understood as "the cost of no longer paying the old experience": both generation and retraining need to be calculated, and the quality of playback also depends on whether the model can generate useful samples. It proves that the maintenance improvement under a specific task agreement cannot be guaranteed for an infinitely long task flow. The performance of old questions is stable, and the ability of new combinations cannot be automatically launched.
ICLR 2026's "Forget Forgetting" replaces the assumptions with the ability to hold more old samples, but is unable to repeat the full amount of retraining. The author proposes a method of combining parameter reset and weight average, and evaluates it in the settings of TRACE continuous instruction tasks. The paper emphasizes that after the old data is sufficient, the adaptation of new tasks is still a problem. Formal Papers §4–5
The memory in this work is mainly Sample library for training playback, not a search tool when answering. The language experiment uses Llama-3.2-1B, and the cost indicator includes the training time for the scaling use of video memory; it cannot directly represent the wall clock time or complete service cost of the large model. The authors also put the access to representative old data and the clear task boundary as a prerequisite. Appendix A.4, A.5, A.7
The two articles do not deny each other: when the old samples are difficult to save, the generation of playback is meaningful; when the old samples can be saved, the stronger real playback should be incorporated into the baseline. Changes in resource premises would have changed the way worth comparing.
If the answer is wrong, it may not be that the knowledge has completely disappeared.
“Spurious Forgetting” investigates whether a drop in performance can reflect a loss of task alignment that makes existing knowledge harder to access. The authors constructed data about people and their attributes to control the relationship between facts learned in pretraining and later question-answering tasks. After learning about new people, the model did worse on the old task. Additional recovery training made much of the earlier knowledge accessible again. Formal Papers §3.1–3.2
The key boundary is: recovery does not happen unconditionally. This experiment is trained with half of the old task quiz data, and the other half is evaluated; Figure 2 also distinguishes the complete answer matching rate of the first word Accuracy of the ordinary task curve and the recovery curve. Therefore, it is not advisable to directly regard the difference between the two curves as a "concealed percentage of knowledge", nor can it be written as "the model that does not move will always answer".
I draw a distinction between knowledge the model can access when it fails a task and knowledge it can recover after an intervention. Recovery weakens the claim that the knowledge disappeared entirely, though it cannot explain every case of forgetting. The paper also acknowledges a limit of freezing lower layers. Freeze too many, and the model may struggle to adapt to the new task. Original Figure 2, Appendix J.1
Memories stay outside, will interference still happen?
M+ puts the hidden state moved from the short-term pool into the long-term storage, and retrieves it back when it is generated. Its knowledge keeps the experiment first written into the material containing the answer, then inserts the interference text, and finally the question and answer; the data is screened with short answers and replies. This is a test of historical information access, not learning a string of new skills and then testing old skills. Original §3.2, §4.3
The authors report improved retention ability, but cross-generation MemoryLLM comparison also changes the starting point and training data of the model; stage ablation also includes additional training. Therefore, similar GPU memory is not equal to the same total storage, training budget or access cost, and all the benefits cannot be attributed to "adding a memory bank". Original §4.2, §4.5
A more direct counter-example comes from the 2026 preprint "When Continual Learning Moves to Memory". The authors fixed the backbone of Qwen-Plus, used BM25 to retrieve the experience, and studied a pair of sequential tasks in each of ALFWorld and BabyAI. Even if the weight remains unchanged, after adding the follow-up experience, the performance of some old tasks will still decline; the effect of abstract experience and complete trajectory also changes with the direction of the task. Original §2–4, Appendix A
The authors interpret this phenomenon as retrieval and interference in the context. Evidence supports "external storage does not guarantee that behavior will never degenerate", but its scope is only two types of environments, one pair of tasks per class and two runs per condition. Each retrieval of about the same amount of experience is not equivalent to a token or other call budget for the whole round; frequent retrieval is especially required for additional costs. This article is still a preprint, and the local phenomenon cannot be generalized into the inevitable law of all memory systems.
The five groups of evidence cannot be ranked in the overall list
| Evidence | key observation conditions | Can support | can not be proved |
|---|---|---|---|
| LAMOL | Pseudo-samples participate in subsequent training | Playback improvement tasks remain | No cost, indefinite learning |
| Surface Forgotten Study | Extra re-test after training | Some knowledge can be called again | Ability is fully available without intervention |
| M+ | Questions and Answers on Materials After Interfering with Text | Improved historical information access | New skills continue to be obtained |
| Adequate Memory Research | Old samples are reserved and used for playback | storage hypothesis change method comparison | It works just as well when there is no old data |
| External Experience Study | Freeze the backbone, update the memory in sequence | Behavioral interference will also occur | All Retriever is like this |
The table summarizes the verification settings of the above articles. Their starting point, source of information, task length and scoring method are different, and the retention rate does not point to the same object. There is currently no reason to give the ranking of "Who is more comprehensive in parameters or external memory" based on these five articles.
In particular, be wary of the final average score: a decline in an old ability may be offset by an increase in another new ability; a task with a low starting point will hardly decline, and it may just lack the ability to lose. When reading the paper, it is best to look at the results of each stage, each task, and the corresponding single-task reference at the same time, instead of directly treating "don't drop" as a good study.
How to do a follow-up experiment that can distinguish interpretation

The solid line in the figure indicates that the same history is provided to different experimental branches, and the parentheses indicate that they have entered the evaluation separately; the dots and lines are only connected to the conditions and test items to be compared, and do not indicate causal improvement. Each branch should be tested separately on the right. The graph is not the paper data, and the following scheme has not yet been implemented.
You can start with the same checkpoint and a fixed empirical stream, set the conditions for pure sequential update, bring playback updates, freeze models and external memory, and update and memory. The old and new tasks should contain questions that need to be applied by the rules, to avoid only asking for the original sentence to be retrieved. Exclude approximate repetitions and test answers from the empirical library, and determine the truncation, retrieval and scoring rules during the development phase.
The first step, open to maintain and increase the income. Every time you complete a stage, the old task, new task, and new combination that are not trained are evaluated. Save the results-by-question results, with the initial model and the model that only learns the current task as a reference. If a scheme retains the old score but does not benefit from the new task, it solves the problem of retention and cannot be described as a complete continuous adaptation.
Step 2, Intervention Support Sources. For the same post-training state, open memory, shield memory, and provide equal-length irrelevant memory respectively; if the structure allows, only remove the records related to the current question. It improves and disappears after being shielded, indicating that the income depends on the access conditions. But closing memory may itself be a distribution change, so this difference cannot be proved by itself "not learned anything".
The third step, check migration and recovery. Replace the question entity, expression and rule combination, and keep the answers cannot be retrieved directly. If the rule transfer still improves, the evidence is stronger than the original answer. If you do short-distance recovery training, you should give all the same data and budget, and ensure that the recovery set is isolated from the evaluation set, to prevent the re-educational answer from being mistakenly called the restoration of old knowledge.
Finally, the training token, the number of playback generation, the number of memorized bytes, the number of searches and the answer budget are booked separately. When all matches cannot be matched, report their respective costs and benefits, and avoid using the "same model" to mask resource differences. The purpose of these comparisons is to allow the competition interpretation to give different predictions, not a pre-selected mechanism to win.
Summary: Let the conclusion of the learning bring the conditions
Forget a little less, it may come from parameter hold, re-alignment, playback exercises, or retrieval compensation; these progress can be useful. The conclusion of the study should be explained: what state has changed, under what access conditions, which old tasks have been maintained, and which new tasks have improved.
What is really worth asking about is not the memory is placed in the model or the outside is learned, but Whether the experience brings repeatable, migrate, cost-representable behavioral improvements, and under which conditions fail. Only by writing this question clearly can we move from a beautiful maintenance score to a verifiable continuous learning proposition.
References
- Fan-Keng Sun、Cheng-Hao Ho、Hung-Yi Lee. LAMOL: LAnguage MOdeling for Lifelong Language Learning. ICLR 2020; this article verifies the full text of arXiv v2 (2019-12-23), and the PDF is annotated with ICLR 2020 published. Full text
- Junhao Zheng、Xidi Cai、Shengjie Qiu、Qianli Ma. Spurious Forgetting in Continual Learning of Language Models. ICLR 2025, official conference edition. Paper
- Yu Wang et al. M+: Extending MemoryLLM with Scalable Long-Term Memory. ICML 2025; the published information is verified by PMLR, the method and experiment are based on arXiv V2 (2025-05-30). formal page|Full text
- Dongkyu Cho、Taesup Moon、Rumi Chunara、Kyunghyun Cho、Sungmin Cha. Forget Forgetting: Continual Learning in a World of Abundant Memory. ICLR 2026, Official Conference Edition. Paper
- Qisheng Hu、Quanyu Long、Wenya Wang. When Continual Learning Moves to Memory: A Study of Experience Reuse in LLM Agents. 2026, arXiv V1 (2026-04-29) preprint; no official employment is confirmed. Full text
Apps by the author
The following is the author's own product promotion. In addition to blogging technology blogs, I have also independently developed some small tools to make everyday life more organized. Welcome to understand:
- Daily Expense Journal: Record the daily income and expenditure, arrange the budget and savings goals, so that the whereabouts of each money is more clear. View on the App Store.
- One Small Action: Select small actions according to the current time and energy, record the feelings and discoveries after completion, and leave a position for daily actions and reviews. View on the App Store.
Find more articles and projects on my website: zffjzz.top. Also welcome to share your experience and help me continue to improve these tools.