Cover illustration

Date: 2026-06-24
Suitable for readers: NLP / LLM direction graduate students, engineers doing RAG and Agent systems, technical readers who pay attention to model service costs
Search date: 2026-06-24

Abstract

In the past two years, the long-context model has rapidly advanced from 32K and 128K to the order of millions of tokens. The problem also becomes clearer: the nominal length of the context window is not equal to the available long context capability. A model can receive 1M token, which does not mean that it can stably find intermediate evidence, do multi-hop reasoning, understand the code repository, and generate high-quality output of the 4D 4-level, nor does it mean that the reasoning cost is acceptable. The main line of the paper in the past year can be summarized as four: first, the position coding and training strategy continue to solve the extrapolation of "short training and long test"; second, the evaluation is from Needle-in-a-Haystack Turn to deep understanding, non-literal matching and interference retrieval of real tasks closer to real tasks; third, the system side surrounds KV cache, Sparse Attention, Cache Compression, and Retrieval Decoding reduces long-context service costs; fourth, the application side begins to discuss whether long contexts can partially replace RAG, SQL, Retriever and writing planners.

The core judgment of this article is: Long Context LLM is entering the "Effective Context Project" from the "Contest of Context Expansion". The value of the next stage is not to put all the materials into the Prompt, but to combine the Long Context model, the retrieval system, the structured memory, the reasoning budget and the evaluation protocol.

Contents

  1. Research background: Why long contexts are not "big prompts"
  2. roadmap for the past year
  3. Key papers by research question
  4. Comparing the methods
  5. Technical trends
  6. Implications for deployment
  7. Limitations and open questions
  8. Questions to watch
  9. Summary
  10. References

Research background: Why long contexts are not "big prompts"

Early long-context studies are often understood as expanding the sequence length: the Transformer's attention matrix, position coding, training length and video memory budget are pushed together. However, Lost in the Middle in 2023 has reminded the community that the model is very sensitive to information locations, and the relevant evidence tends to perform better at the beginning or end, and the performance will decline when it falls in the Middle. This conclusion was later verified by a large number of Needle-like experiments, long-documented Q&A, and code repository tasks.

After 2024, the context window for business models and open source models has increased rapidly. The Google Gemini 1.5 technical report brings the "million-level context" to the mainstream discussion; the OpenAI developer documentation lists GPT-5.4 / GPT-5.4 Pro has a 1.05M Context window; the Anthropic document lists Claude Fable 5 and Mythos 5 when searched in June 2026. 1M Token, some Claude 4 series also enter the range of 200K to 1M; the official information of Meta Llama 4 Scout and the model card to discuss the 10M Token level context. Supplier specifications change rapidly, and the specific available length, price, rate limit and model alias must be based on the document of the day that is called.

The question of academic research has also turned: If the model can "see" a million tokens, can it "use" a million tokens? If so, what is the cost? If not, the bottleneck comes from position coding, training data length distribution, attention dilution, decoding stage KV cache, or the evaluation task itself is too simple?

Long context research roadmap

roadmap for the past year

Route 1: Location coding and training extension. Basic work such as RoPE, ALiBi, YaRN, LongLoRA, etc., enables the model to be extrapolated from shorter training length to longer inference length. The 2025 LongRoPE2 continues along this line: It proposes that insufficient training in high-frequency RoPE dimension can lead to extrapolation of OOD and use Needle-Driven Perplexity The search rescale and mixed context training attempts to preserve the short context abilities while expanding the context.

Route 2: The long context evaluation is upgraded. LongBench V2, NoLiMa, RULER, LOFT, ICR^2 and other work instructions, simple "needle hiding haystack" is not enough. What is really difficult is multi-document inference, structured tables, code repository understanding, long conversation history, non-literal matching retrieval, and in-context retrieval with strong retriever interference items.

Route 3: Reasoning system and KV cache. As long as the autoregressive decoding is maintained, long input and long output will push the KV cache into the service cost core. RocketKV, FIER, KV cache Clustering, ParisKV, etc. 2025-2026 work from compression, sparse reading, clustering, GPU native retrieval and CPU OFFLOAD Equivalently reduces the memory and bandwidth pressure. The specific speed improvement depends on the model, hardware and task settings, and the project selection cannot only look at the peak numbers in the abstract of the paper.

Route 4: Long output and application paradigm. LongWriter and LongWriter-Zero extend the problem from "reading" to "write long". LOFT and ICR^2 push the problem to "Whether long context can replace the retrieval system, RAG, SQL-like query". The conclusion is more like a complementary rather than a substitution: the model can directly handle large contexts, but external retrieval and structured tools are still necessary in scenarios with strong interference, strong structured constraints, frequent updates, or an auditable retrieval link.

Key papers by research question

1. Position code, training length and extrapolation

RoFormer/RoPE is the basis for the location modeling of many of the current LLMs. It maps the absolute position as a rotation and naturally contains relative position information in the attention, so it has a certain length of extrapolation potential. However, extrapolation is not a free lunch: the frequency dimension, position distribution and long-sequence mode that are not covered in the training stage will be manifested as an increase in confusion in AT inference time, and the local memory is acceptable but the cross-segment reasoning fails.

LongRoPE2 treats context extension as a two-part goal: the model must work at the target length and retain its original short-context performance. It balances the two through a search for rescaling factors and training on a mixture of lengths, going beyond a simple change to the RoPE base. When extending an existing open-source model, supplement needle-in-a-haystack (NIAH) tests with short-context regression tests for general capabilities, instruction following, coding, and question answering. A longer window is of little use if the model becomes less capable at its original length.

2. Evaluation: From Literal Retrieval to Real Reasoning

LongBench v2 is an important node for long-context reviews from the end of 2024 to 2025. It contains 503 multi-selection questions, covering tasks such as single-document, multi-document, long-context ICL, long conversation, code repository, and structured data, and Input length can reach 2M Words. In the paper report, experts have accuracy about 53.7% under the 15-minute limit, the best model for direct answering is about 50.1%, and the O1-preview with longer reasoning is about 57.7%. The significance of these numbers is not "which model is permanent first", but shows that the long context task is not just a retrieval, but also needs to inference budget.

NoLiMa's weaknesses for NIH: If the problem has a lot of literal coincidence with Needle, the model may rely on keyword localization rather than real understanding. NoLiMa designs Needle with low word surface overlap, requiring the model to find evidence through implicit associations. The paper is reported in the July 2025 revision, and 13 models claiming to support at least 128K are significantly degraded under long context; for example at 32K Most models are less than half of the short-context strong baseline. The most worth remembering here is the evaluation idea: not just "string positioning", but "semantic positioning".

RULER then asks "realistic context length" instead of nominal length. It allows reviewers to see when the model begins to collapse through programmatically generated tasks and controls of different lengths. For the R&D team, the Ruler-style review is closer to capacity testing than a single fixed-length benchmark.

3. Can long contexts replace RAG?

LOFT discusses a very engineering temptation: If LLM can read the entire Corpus, do you still need Retriever, RAG, SQL and other tools? The paper extends the task to text retrieval, visual retrieval, audio retrieval, RAG, SQL-like data query and many-shot ICL. The results show that the long context model is competitive in some retrieval and RAG scenarios, but there are still obvious challenges in the sensitivity of combined reasoning, structured queries and prompt strategy.

ICR^2 further points out that tasks such as LOFT may underestimate the real difficulty, because there are usually similar interference paragraphs returned by strong Retriever in real RAG scenarios. ICR^2 uses Confounding Passages to construct a closer evaluation of actual combat, and proposes retrieve-then-generate Fine-tuning, Retrieval-attention-probing, joint training search head and other methods. Engineering inspiration is very direct: the long context is not to cancel the retrieval, but to migrate the retrieval from the external index to the internal attention and reasoning process of the model; whether it is worth it depends on the cost, auditability and error recovery ability.

4. Reasoning system: KV cache is the real bottleneck

Long context reasoning has two stages: prefill reads into long input, and decode generates by token. The cost of prefill is usually affected by the input length and attention implementation, and decode keeps reading the larger and larger KV caches. RocketKV represents a type of training-independent method in 2025: first do coarse-grained cache eviction, and then do fine-grained top-k sparse attention, the goal is to reduce the memory and bandwidth under a small precision loss. Follow-up directions such as FIER, Chelsea, and ParisKV explore fine-grained retrieval, online clustering, anti-drift retrieval and million-level KV access respectively.

Be wary of two things in the project. First, the SpeedUp of the paper report often appears under specific GPU, Batch Size, context length and task combination; you must use your own Prompt distribution test before going online. Second, the KV cache strategy will change the wrong mode: some methods have no sense of ordinary Q&A, but may delete key evidence in multi-hop references, long code positioning, and audited Q&A.

5. Long output: long reading does not mean long writing

LongWriter observes that a model able to read 100K tokens may still struggle to generate more than 2000 words consistently because its SFT data contains few long outputs. The authors use AgentWrite to construct long-form writing data and introduce LongBench-Write. LongWriter-Zero, submitted in June 2025 and revised in April 2026, turns to reinforcement learning. Starting from a base model and using neither human-annotated nor synthetic long-form training examples, it uses a reward model to encourage length control, writing quality, and adherence to the requested structure and format. The paper reports that its trained Qwen2.5-32B model outperforms traditional SFT methods on long-form tasks including WritingBench and Arena-Write. Check the paper version and leaderboard status before citing specific rankings.

This line is important for blogs, reports, code migration, and long document generation: long context applications are not just "reading in", but also "planning, segmentation, global consistency, reference backfill, and post-editing".

Comparing the methods

Direction Representative work Solved the problem Engineering heuristics
location coding extension RoPE、YaRN、LongRoPE2 Short training length test, target length extrapolation, short context maintenance After expanding the window, we must test short tasks and real business tasks backwards.
long context review RULER、LongBench v2、NoLiMa The nominal length is inconsistent with the available length Replace single NIH with length scan, location scan and semantic search test
long context application LOFT、ICR^2 Whether long context is a substitute for RAG/SQL/retriever Most scenarios are more suitable for "retrieval + long context + reference check"
KV cache system RocketKV、FIER、Chelsea、ParisKV Long input/long output inference cost is high Do Latency, Video Memory and Accuracy Joint Evaluation with Business Prompt Distribution
Long output training LongWriter、LongWriter-Zero The model is long but not written Long text generation needs data, rewards, planning and verification closed loop

Technical trends

First, the evaluation will continue to "template". NIAH is still suitable as a smoke test, but not enough. What is more valuable in the future is non-literal evidence positioning, cross-modal long context, code repository-level modification, multi-round session memory, RAG with interference terms and structured data Q&A.

Second, the long context is bound to the Reasoning-Time Compute. LongBench v2 with longer inference models performs better, indicating that the context length and thinking budget are coupled to each other. Only give the model more tokens, but do not allow it to plan, reread, reference and verify, the effect will be limited.

Third, system optimization will become the core difference. The context of a million tokens is not an "input parameter", but a combination of video memory, bandwidth, scheduling, cache lifecycle and billing policies. Context cache, prefix cache, chunked prefill, kv quantization, sparse reads, and CPU/GPU layered storage can become the key capabilities of the service framework.

Fourth, the long context will not eliminate the RAG. Three types of architecture are more likely to occur: small context retrieval, long context rough reading global material, and iteration between the two. For auditing, compliance, finance, medical and scientific retrieval, traceability is still more important than "remembered within the model."

Implications for deployment

  1. Do not use the context window length to select models. At least four sets of tests are to be done: Evidence location scan, interference intensity scan, task length scan, output length scan.
  2. For production RAG, do not expand the top-k without limit. A more stable solution is "Retrieve Recall + Rerank + Group Summary + Key Original Segment + Long Context Review".
  3. For the code repository task, the long context is suitable for the global structure, dependency graph and key file fragments, and it is not suitable for the entire repository to be inserted at one time and look forward to the perfect editing.
  4. For long report generation, the output should be split into a plan, chapter draft, reference verification, global consistency check and final polish; do not generate a final draft with a prompt.
  5. For service costs, it is necessary to record Prefill Latency, Decode Latency, KV cache peak, Cache hit rate, and cost per request token, not just total delay.

Limitations and open questions

First, the long-context experiments of many papers are still mainly English, synthetic tasks or a few open models, and the extrapolation of Chinese long-term documents, cross-language data, table PDFs, and real enterprise knowledge bases is limited. Second, the context window, price and current limit of the business model change rapidly, and the specifications of the model are only used as the versioning record of 2026-06-24 search. Third, long contextual evaluations are easily affected by the Prompt strategy: whether to allow the thinking chain, whether to allow step-by-step search, and whether to provide a file name and directory structure will change the result. Fourth, the "precision lossless" of the KV cache compression method is usually statistical or Benchmark, and cannot be used by default for high-risk Q&A.

Questions to watch

The most worthy of tracking is not who has posted a longer window, but the following question: Can the model maintain multi-hop reasoning stability in the range of 512k to 1m? How is the division of labor between long context and tool calls, code execution, and external databases? Is it possible to establish a stronger auditable reference mechanism, so that each conclusion in the model answer is pointed to the context fragment? Can the service framework automatically compress KV cache without sacrificing key evidence? Can long output training extend from writing to long code migration, long lab reports and complex Agent trajectory reviews?

Summary

The study of Long Context LLM has passed the stage of "can it be installed?" The real problem is: after it is installed, whether the model can stably retrieve, reason, generate, reference and self-check; whether the system can complete these actions at an acceptable cost; whether the evaluation can reveal the true failure mode. For the researchers, the Long Context provides a new model capability boundary problem; for the engineering team, it is more like a set of system design constraints. The outcome of the next stage is not the maximum token number, but in the effective Context utilization.

References

The search date is 2026-06-24.

  1. Ning Shang et al., “LongRoPE2: Near-Lossless LLM Context Window Scaling”, arXiv, 2025-02-27. https://arxiv.org/abs/2502.20082
  2. Yushi Bai et al., “LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks”, arXiv, v2 revised 2025-01-03. https://arxiv.org/abs/2412.15204
  3. Ali Modarressi et al., “NoLiMa: Long-Context Evaluation Beyond Literal Matching”, arXiv, v3 revised 2025-07-09. https://arxiv.org/abs/2502.05167
  4. Payman Behnam et al., “RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression”, arXiv, 2025-02-19. https://arxiv.org/abs/2502.14051
  5. Yuhao Wu et al., “LongWriter-Zero: Mastering Ultra-Long Text Generation via Reinforcement Learning”, arXiv, v3 revised 2026-04-08. https://arxiv.org/abs/2506.18841
  6. Yifu Qiu et al., “Eliciting In-context Retrieval and Reasoning for Long-context Large Language Models”, arXiv, v3 revised 2025-06-09. https://arxiv.org/abs/2501.08248
  7. Jinhyuk Lee et al., “Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?”, arXiv, 2024-06-19. https://arxiv.org/abs/2406.13121
  8. Cheng-Ping Hsieh et al., “RULER: What’s the Real Context Size of Your Long-Context Language Models?”, arXiv, 2024. https://arxiv.org/abs/2404.06654
  9. Nelson F. Liu et al., “Lost in the Middle: How Language Models Use Long Contexts”, arXiv / TACL, 2023. https://arxiv.org/abs/2307.03172
  10. Jianlin Su et al., “RoFormer: Enhanced Transformer with Rotary Position Embedding”, arXiv, 2021; revised 2023. https://arxiv.org/abs/2104.09864
  11. Bowen Peng et al., “YaRN: Efficient Context Window Extension of Large Language Models”, arXiv, 2023. https://arxiv.org/abs/2309.00071
  12. Yukang Chen et al., “LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models”, arXiv, 2023. https://arxiv.org/abs/2309.12307
  13. Google AI for Developers, “Gemini API Models”, last updated 2026-06-15 UTC. https://ai.google.dev/gemini-api/docs/models
  14. Meta Llama, “Llama 4 Model Cards and Prompt Formats”. https://www.llama.com/docs/model-cards-and-prompt-formats/llama4/
  15. Anthropic, “Claude models overview”, accessed 2026-06-24. https://platform.claude.com/docs/en/about-claude/models/overview
  16. OpenAI Developers, “GPT-5.4 Model”, accessed 2026-06-24. https://developers.openai.com/api/docs/models/gpt-5.4