
- Date: 2026-10-21
- For graduate students, early-career researchers and research engineers
- Deadline: 2026-10-21. Retrieve the work for nearly 12 months and retain the necessary classic research; this paper is a deep reading of the paper, does not run a model experiment, and does not claim to be fully covered.
Summary: If you can pretend to be history, why do you still "remember" wrongly?
Suppose you say in the chat that the paper group will be changed to Thursday; after several sessions, it will be clearly changed back to Tuesday. Assistant replied "It's fine on Thursday or Tuesday", and it seems that two messages are remembered, but don't know which one still works. If you repost all the records to it, it will finally answer correctly, and you cannot judge whether this information has been saved before. This is an illustrative example, not paper data.
The long context expands the range of information that can be accessed at one time; long-term memory also requires information to be available across sessions and can maintain effective status according to time and conditions. The three can support each other, but cannot replace each other with an input length or total accuracy. In this paper, we compare the four studies to discuss which step the evidence has been measured, without presupposing that "expanding the window is useless" or "external memory will win".
Take the "remember" apart first
The context window is the input and related sequence ranges of a model call that allows processing, usually measured in token; token is the unit of the model to segment the text, not equal to the number of Chinese characters. read The question is: the evidence has entered the current input, can the model be found and used correctly? The data is in the window, which is an accessible condition, not a guarantee of success.
Maintain The question is: after the previous session ends, which information is still saved by the system and can be re-accessed later? Save locations can be original history, abstracts, external databases, or other persistent states. If the system re-splicate the full text from the saved logs every time, it can also provide cross-session memory service; therefore, long context and external storage are not mutually exclusive routes. The question is who has to say who saved, what was saved, and what inputs the new session has regained.
Renew The question is: After the arrival of the new information, does the system distinguish the current state, the past state and the negative statement? "I used to be in School A, but now in School B" does not require the deletion of history; to answer that the current unit should choose School B, and answer that the past experience may still require School A. The update mentioned in this article does not mean rewriting the model weight, nor does it treat "the answer is no longer mentioning the old information" as the underlying data has been completely deleted.

Figure: The three groups of diagnostic tasks proposed in this paper. The arrow indicates the flow of information to be tested, and the dotted line represents the session boundary; the green card indicates the current valid status, and the old card can be left as a history. There is no measurement result in the figure, and it does not claim that an internal mechanism has been proved. The observed results of the paper are shown below.
This distinction brings three competing explanations: the wrong answer may be because the key sentences have not entered the input at all; the input may be complete, but the search or reading is not selected; Just looking at the final answer, all three situations are like "forgot". The repair scheme may be to increase the accessible capacity, improve the selection, and maintain the status version respectively.
Information is right in front of you, or it may not be used
"Lost in the Middle" et al. controls the location of the answer documents in the multi-document question and answer, and changes the number of interference documents; another task tests the key-value retrieval of the key value. In the paper, the obvious position effect was observed in the measured model, and the multi-documented question and answer often showed that the two ends were superior to the Middle. Some models can do a good job of simple key-value retrieval, and the complex question and answer cannot be launched accordingly. Official full text, §2–3
The value of this design lies in separating the "does not given" and "giving it but not working well". Fixed questions and document collections, only moving answer documents can check location sensitivity; increase interference documents, change the length and filter pressure at the same time. The two interventions cannot be mixed with the same length effect.
But it doesn't test what is left after closing the session, and it doesn't ask to revoke outdated preferences. The paper uses the model, prompt and greedy decoding at that time. This article is inferred: the window specifications cannot replace the task evaluation; not all subsequent models will inevitably present the same curve. Writing historical results as a law that cannot be broken through today's model, it exceeds the evidence.
Handing over the entire piece of history to the model, and remembering it one by one, is a different protocol
LongMemEval covers tasks such as cross-session quiz, time reasoning, and knowledge update. In the official version of Figure 3b, when the strategy of taking notes first and then answering the strategy, GPT-4o only reads the accuracy of the relevant evidence session is 0.870, and reads about 115k Token has a complete history of 0.606; the report in the figure has a relative decrease of 30.3%. The Oracle here is a relevant session provided by the review, not a Deployable Retriever . §3.3–3.4, Figure 3b
This comparison first tests the reading and interference pressure, and the gap cannot be counted as "writing failure". The commercial system experiment of the same paper makes people interact with each other and ask questions in a new session; it uses 97 questions and shorter history, which is not the complete benchmark above. The two sets of numbers correspond to different inputs and processes, and cannot be put into a unified ranking list. Commercial system testing occurred in August 2024 and cannot be regarded as today's product evaluation. §3.4 and Footnote 2
For research design, the role of Oracle is to diagnose: if the evidence is still wrong, it is not enough to improve the search alone; if only the relevant session can be answered correctly, it only means that the scope of the problem is narrowed. It also reduces interference, input cost and selection difficulty, and cannot directly prove that a certain memory architecture will get the same benefits.
The version is also an experimental condition. The official repository records the cleanup of historical sessions in September 2025; run the cleanup version today, and the original table of the paper cannot be reproduced by default. This article cited the results of the official paper, no download data or rerun model. Authors’ repository
The key to the update task: what is the right answer?
The Knowledge Update Scoring Tips for LongMemEval Appendix A.4 allows answers to contain previous information, as long as the new answers required are correct. This is suitable for checking if updated answers can be provided, but not a strict "old information misuse rate". Appendix A.4
Memora separates the scoring of whether valid information appears and whether the invalid information is excluded, and then uses the forget-aware memory Accuracy FAMA to merge; it generates problems with simulated state trajectories, and discovers problems such as reuse of old information under more frequent update conditions. The text model group directly reads the history, and the memory system group ingests the history one after another, and the latter uses GPT-4o-mini to generate the answer. §3–5
There is no conflict between papers that must be selected. An answer may satisfy "giving a new answer", but not "no use of the invalidity as the current basis". Similarly, it is a different semantic behavior to mention the old unit as a resume and use the old unit as the incumbent unit. If the evaluation only detects the appearance of old strings, a reasonable historical explanation will also be misjudged; item-by-item semantic criterion is therefore closer to the problem than simple keyword disabling.
Memora's limitations are equally critical: the dialogue is generated by simulation, and the week, month and quarter settings change both historical scale and update pressure; the two types of systems also use different generative models and processes. This article does not interpret the inter-group scores as pure causal effects of external storage, nor does it attribute the decline under longer conditions to the calendar time itself. Its LLM referee has artificial consistency verification, but it is still a measurement tool with errors. §4.1–4.2、Limitations
Reading other people's chats, and experiencing your own chats, may also be different
AMemGym allows the simulated user to actually interact with the tested assistant, and constrains the dialogue with the pre-generated state changes. It compares this on-policy review and the off-policy review: the latter reproduces the existing interactions generated by GPT-4.1. In Table 2, the memory configuration ordering of the same generation backend will change. §3.1–3.3、§4.1–4.2
The authors suggest that what an assistant does during a conversation affects its memory operations. That is plausible, but a change in system rankings cannot identify the mechanism by itself. The questions asked, the information repeated, the output length, and the actual cost can all vary with the interaction. The online protocol better reflects the conversation each system experiences, at the cost of giving up a verbatim shared history across systems.
The normalized memory scores of the paper are referenced by random baseline and the scores when directly providing real user status. This "upper bound" is the empirical reference of a specific model under the input of ideal information, not the upper bound of mathematics, nor is it the amount of internal memory that has isolated all reasoning differences. The simulated user's state is controllable, making mistakes easier to locate; the vagueness, correction and silence of real users still need to be verified separately. §3.2–3.4
How should the four evidence be put together?
| Research and pending inspection propositions | Key Conditions and Observation | cannot be proven separately |
|---|---|---|
| Lost in the Middle: Can the window be read stably | Adjust the position and interference, the performance is affected | Cross-session keeps failing |
| LongMemEval: Historical Visits and Q&A | There is a gap between related sessions and complete history | Real search can reach Oracle |
| Memora: Can the information be used correctly after the update? | Score valid and invalid information separately | External memory must be superior to long context |
| AMemGym: Whether the interaction protocol affects judgment | The actual interaction and the ordering of playback are different | Static evaluation has no value |
These papers do not share all tasks, model starting points, data division and budget. They mainly evaluate the existing models and systems, and cannot be interpreted according to the same starting point training experiments; input, memory writing, retrieval, reasoning and referee calls are not a unified cost caliber. They support the extension of the diagnostic dimensions, rather than a cross-paper performance curve. In particular, it is necessary to distinguish: the recall of relevant materials, the correct answer to the questions, and the correct rule of exclusion of failures are different indicators.
A small study that can distinguish interpretation
Below are the research programs that have not yet been implemented. First fix the same generative model, prompt and a batch of newly constructed character status trajectories, and isolate the ginseng characters from the final test character. The status is generated by the determination rules, which contain invariant facts, explicit updates, cancellations and historical questions; do not call other series of data in this workspace. Semantic ambiguous samples must prescribe the rules of decision in advance, and the self-assessment of the model cannot be disguised as a human gold label.
The first step, when the evidence is fully visible, only change its position and irrelevant text length. The second step is to reset the session, allowing the system to access the state written in advance only, while retaining the control of "re-providing all logs". The third step is to change the number of updates when the total length is approximately fixed, and ask the current state and historical state respectively. Only in this way can we have the opportunity to disassemble the interference, persistence and validity judgment.
For each trajectory, save what was written to memory, what retrieval returned, and what the model actually received. For failed examples, try two separate interventions. Supply the correct evidence in one, and give the current state directly in the other. Check which fixes the failure. Use a length-matched replacement control when supplying evidence, so shortening the input cannot explain the gain. Report these diagnostic interventions separately from normal operating scores. A failure that persists with the correct state explicitly given points to a problem beyond memory retention.
At the same time, report factual coverage, current status accuracy, failure basis usage and history to ask Accuracy; for the shortcut of "the latest sentence of the general election", add a sentence that appears late but describes old events. Delay and total token fees should include writes, summaries, searches, and answers, and compares the performance of the fixed budget with the allowable extra budget, respectively. Pre-specify the number of samples, stop rules and statistical uncertainty reporting methods, and avoid picking several successful cases to announce "learning long-term memory".
If the expansion of the window only improves the first step, the conclusion should be stopped in the read condition; if the cross-session gain depends on saving and retrieval, it will report the system-level capability; if the current state is improving and the history is degraded, it is necessary to check whether the old value is only covered. These criterions make negative results also have research value.
Summary
Long contexts provide a larger workspace for memory, but "accessible" is not "appropriately used", "saved" is not "updated". Existing evidence is enough to require researchers to measure reads, maintain and update separately, but not enough to declare a certain architecture forever.
The next time I see "Long-term Memory Accuracy Improvement", I will first ask: When will the evidence enter the system, what is actually given during the test, and whether the answer is allowed to mix in the invalidation information? Target these three conditions, only to know which level the improvement is on, and only to know what the next experiment should change.
References
- Nelson F. Liu、Kevin Lin、John Hewitt、Ashwin Paranjape、Michele Bevilacqua、Fabio Petroni、Percy Liang. Lost in the Middle: How Language Models Use Long Contexts. TACL,2024,12:157–173. Official pages and full text
- Di Wu、Hongwei Wang、Wenhao Yu、Yuwei Zhang、Kai-Wei Chang、Dong Yu. LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. ICLR, 2025; pre-print first edition 2024. formal page; Code and Data Release Notes.
- Md Nayem Uddin、Kumar Shubham、Eduardo Blanco、Chitta Baral、Gengyu Wang. From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents. Findings of ACL,2026,26814–26841. Official pages and full text
- Cheng Jiayang、Dongyu Ru、Lin Qiu、Yiyang Li、Xuezhi Cao、Yangqiu Song、Xunliang Cai. AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversations. ICLR,2026. Official pages and full text
The methods and results of this paper are subject to the official full text that has been opened; the two papers in 2026 have supplemented the evidence of the last 12 months. Propositions, verification and positioning and search options are saved in matching Evidence.json, Sources.json and research_log.json.