
Abstract
"Support 128K context" only means that the model can receive such a long input, not that it will use each location equally. Liu et al. Lost in the Middle Put the evidence of the same answer to the beginning, the middle and end of the long context in turn, and find that the performance of the multi-type models is often U-shaped: the beginning and the end are better, and the middle is poor. This article is along TACL The paper and the official repository are not reproduced a dynamic model score, but a set of auditable protocols: maintaining questions, answer evidence, and interfering with document sets The combined and decoding parameters remain unchanged, and only the evidence position is moved; at the same time, the context length is scanned to report the position curve, the first and last - middle difference and the worst position. The accompanying standard library scripts validate data construction and statistical code. It uses the artificially set hit probability, not a language model experiment, and cannot be extrapolated as the result of the paper.
Reproducing value and goal: the window length is not equal to the effective length
Long context reviews are most likely to make two mistakes. First, only put the evidence in the last Needle-in-a-Haystack (a haystack) test, and after success, it declares that "the model has a full window"; For paper reading, RAG and code repository Q&A, the location of key evidence is usually determined by search sort, document stitching, and templates. If the same evidence is moved, the answer is not only measured by the system, but also in the position sensitivity.
There are four goals in this time: how to read the official data to ensure that there is just one answer document; establish a two-dimensional scan of length and relative position; report the average accuracy with the worst position and position difference; use a flat comparison to prove that the statistical program itself will not draw u out of thin air. shape. When the model is actually connected, only replace predict(), the rest of the sample identities, arrangement, analysis and aggregation should be kept frozen.
Core idea: turn the position into a controlled variable
Suppose each sample has questions (q_i), an answer document (D_i^+) and (L-1) interfering documents (D_i^-) without answers. At the relative position (p\in[0,1]), put the answer document into the index
(l) is the number of documents, (j) is the gold index starting from 0, (c_{i,l,p}) is the document sequence of length (L). If the length of the token is the main variable, each document should be cut to a fixed token budget first, and then the tokenizer, template and actual input length should be recorded; "20 documents" are not naturally equal to the same token number.
Model (M_ heta) outputs (hat y_{i,l,p}) in the freeze prompt and decoding settings. For (n) fixed samples, the position Accuracy is
where (Y_i) is a collection of acceptable answers, norm It is the pre-registered answer normalization. If a BATCH contains (B) samples at the same time, each context has (L) documents, the input can be understood as a document tensor (X\In\MathBB{N}^{B\Times l\times t}), (t) It is the number of tokens after cropping; the position experiment only changes the arrangement on the (L) axis, and does not change the content of the answer.
Don't just give the macro average (ar A_L=|p|^{-1}\sum_{p\in p}a(l,p)). should also report
(G_L) is the difference between the end-to-end mean relative to the midpoint, and (W_L) is the worst position. The position robust model should have a smaller (G_L) and as (L) increases, (W_L) should not collapse suddenly.

Paper and guide to the official implementation
Start with the task design in the TACL 2024 paper. The authors moved relevant information within multi-document question-answering and key–value retrieval tasks, comparing performance at the beginning, end and middle. Changing the position could substantially degrade performance for the models tested, often with stronger results at the ends than in the middle. The finding is specific to the tested models, prompts and 2023 experimental setup.
Look at the official repository README.md. qa_data/ At the same time, Oracle, 10/20/30 document settings are provided; each ctxs record including hasanswer、original_retrieval_index and isgold. Generate script make_qa_data_from_retrieval_results.py Filter Distractors without labeled answers, and then use --gold-index Specify the answer document location; the 20 document example given by README scans 0, 4, 9, 14, 19. The most worthy of reuse here is not a command, but the principle of "the same retrieval candidate set only changes the gold index".
The prediction and scoring paths are then read. EXPERIMENTS.md generate predictions and evaluate_qa_responses.py Separate; the latter only takes the first row of the model answer, calling best_subspan_em, to avoid the model copying the entire context speculative hit. When reappearing, the original output, the answer after parsing, and the sample-by-sample score should be saved; if only the mean value is left, the model cannot be distinguished from finding evidence, the alias is generated, or the parser is misjudged.
LongBench and RULER extend the evaluation in different directions. LongBench covers real tasks including question answering, summarization, few-shot tasks, and coding. RULER goes beyond single-needle retrieval with multiple needles, variable tracking, and aggregation, with configurable length and complexity. Lost-in-the-Middle therefore tests one aspect of long-context ability. Retrieving evidence does not establish that a model can reason across it, and robustness to evidence position does not establish factual faithfulness.
Minimum experiment: first verify the protocol, do not pretend to reproduce the model
run:
python3 code/minimal_position_sweep.py
python3 code/minimal_position_sweep.py --check-only
Scripts only use the Python standard library, fixing the seed 62. For 20, 40, 80 documents and relative positions 0, 0.25, 0.5, 0.75, 1.0, each generate 800 controlled samples; each context has just one TARGET= document. u-shaped The simulator uses SHA-256 to derive deterministic random numbers, and artificially make the head and tail hit rate high, the middle is low, and the length is slightly reduced; flat-control The probability of each position is similar.
In the CPU smoke test with 20 documents, the U-shaped simulator had accuracies of 0.955, 0.675, 0.574, 0.680, and 0.931 at the five positions, with a macro-average of 0.763 and a position gap of 0.369. With 80 documents, the corresponding accuracies were 0.887, 0.630, 0.465, 0.630, and 0.856, with a macro-average of 0.694 and a gap of 0.407. The flat control’s absolute position gap was below 0.02 at all three lengths. Assertions also checked that exactly one document contained the answer, that indices mapped correctly, that values were in range, and that the U-shaped gap and degradation with length were present.
These results only indicate that the structure and polymerization can detect the position effect of "written into the simulator". They are not the scores of some LLM, NaturalQuestions, LongBench or RULER. To truly reproduce the model weight or API snapshot, tokenizer, chat template, precision, inference engine, decode parameters and sample file hashing, and connect the real output into the same evaluation table.
Evaluation protocol: two-dimensional scanning, pairing comparison, layered report
The first step is to freeze the sample. Fixes a set of interference documents for each problem, and all location conditions share the same example_id, so that you can do pairing and comparison; don't draw another batch of more difficult documents for the middle position. The answer string may accidentally appear in the interference document, and the leaked scan after normalization must be done. If the true retrieval itself contains multiple reasonable answers, it should be listed as a single gold subset, instead of disguising it as a single gold.
The second step is to scan the position and length at the same time. The position covers at least the first, quarter, midpoint, three-quarter, and tail; the length should be divided into barrels according to the actual token number. Each cell reports the number of samples, Accuracy or task metrics and Confidence interval, and gives the (G_L) interval for the paired bootstrap. If different lengths are lost due to truncation of problems, instructions or answers, this is a template failure, not a location effect.
The third step is to add the control. Oracle only gives the answer document, closed-book does not give the context, which is used to distinguish between knowledge and retrieval; randomly disturb the document but keep the relative position of the gold, check the search order semantics; run the problem once before and after the context, and test Instruction Recency; keep the content of the answer unchanged but rewrite the document title, and measure the clues of the format. All controls must be determined before looking at the results.
The fourth step is divided into task report. Exact key-value search is suitable for Exact Match. Multi-document Q&A requires alias collection or subspan metrics, and the abstract and cross-document reasoning should use different rubrics. The overall average score cannot prove that each type of task is robust. Costs are also recorded: the actual input token, the output token, the peak video memory, throughput and the number of failed retries; when the window length is declared, the "effective length" that has reached the preset quality threshold is also given.
From Protocol Simulation to Real Model: Minimum Replacement List
When accessing the open source model, the JSONL is first exported for each condition, not temporarily rearranged in the inference loop. At least save each line example_id, questions, answer collections, document ID order, Gold document ID, Gold document index, GOLD start and end token, Token, template version and sample hash. Inference output save raw_response, to resolve the answers, the reasons for termination, delay and random seeds. In this way, the output of the same sample in five positions can be paired one by one, and the data layer, model layer and grading layer can also be rerun independently.
If using the Hugging Face model, print the final tensor shape produced by the tokenizer first, such as input_ids.shape=[B,S]; Here (b) is the batch size, (s) is the real sequence length after the system instructions, documents and problems. Inspection of S Whether the upper limit of the model configuration or inference engine is exceeded, and the truncation direction is clear. The quantitative model is also fixed for the quantitative configuration, as the accuracy change may interact with length or position. If the API is used, the visible model snapshot, request parameters and response metadata should be recorded; services that cannot be frozen can only be marked as "observation of this date".
The minimum real experiment does not have to start with 128K. You can choose a local runnable model first, take 100 samples in the three gears of 4K, 8K, and 16K, and complete the five positions at one time. The main indicators of pre-registration are (G_L) and (W_L), and the average macro is used as an auxiliary; if the midpoint fails, the same example_id The three complete input and output of the first, middle and tail are case studies. The model mechanism or mitigation method is discussed only after the re-run is consistent, the parsing is correct, and the Confidence interval is stable.
The result directory should also be replayable: save profiles, code commits, environment versions, input hashing, sample-by-sample prediction, and aggregate scripts. Don't just take a screenshot of a U-shaped curve. The curve must be traced back to each sample, and the failure case must be able to restore the complete prompt at that time; when it involves a private document, the desensitized structural equivalent sample and hash are saved, rather than leaking the original text.
Failure analysis: first arrange protocol errors, then talk about attention mechanism
If the curve is abnormally serrated, first check the index: whether the relative position map is rounded up, whether the chat template triggers a truncation in some lengths, and whether the answer document is really in the token interval of the recorded token. The document number is not token position; when the first few articles are longer, "Part 10" may not be at the midpoint. The GOLD token and the total token should be recorded, and the relative token is drawn in depth.
If the end is particularly strong, it may come from Recency, or it may be close to the end of the document because of the problem; it can be distinguished by exchanging the problem location. If the beginning is strong and the end is weak, check whether the maximum length crop is deleted from the tail. If the model returns the correct long answer but Exact Match is zero, spot check normalization, newline truncation and alias collection. If the central degeneration only occurs in the real question and answer, and the key-value search does not appear, the bottleneck may be evidence identification or multistep reasoning, rather than simply "can't see the middle".
Attention heat maps can propose mechanisms assumptions, but cannot prove cause and effect alone. Google Research Found in the Middle A possible next step is to investigate a link between position bias and attention calibration. Testing that explanation on a particular model requires varying position, intervening on attention or activations, and checking downstream performance. A U-shaped performance curve alone cannot identify a particular layer or positional encoding as the cause.
Questions for further research
One is relative token Depth, document serial number and semantic search ranking, which one is more able to predict failure; the second is the distribution of multiple supporting or conflicting evidence At different positions, how the position effect interacts; the third is whether the position randomization training can improve the worst position without damaging the end and end; RAG reranking should optimize the correlation mean, or optimize the actual availability of long context models; fifth, whether the position difference (G_L) can be calibrated, abstension Joint modeling with reference loyalty; sixth, the "effective context length" should be defined according to the worst slice, average slice or task risk. A small and reliable topic is a fixed open source model and multi-task samples, a full-factor experiment on the depth, problem location, and the amount of evidence of the Gold Token, and publicly output sample-by-sample samples.
Summary
The core of Lost-in-the-Middle is not "long context is useless", but "receivable" into "can be used stably". Reliable reproduction is to move only the answer evidence on the same batch, scan the position and length, record the true token depth, and report the macro average, worst position and the end-to-middle difference. Use this script to check the data and statistics, and then connect to the real model; only when the location, task and complexity slices are transparent, the so-called long context ability is a comparable and diagnosable scientific research conclusion.
References
Search date: 2026-09-04. The following are papers, conference pages, research institution pages or official codes; dynamic models, API behaviors and lists are treated manually.
- Nelson F. Liu et al., 2024, Lost in the Middle: How Language Models Use Long Contexts(TACL)
- Nelson F. Liu et al., Lost in the Middle official repository、Experiment order、Location data generation script and Q&A Scoring Script
- Yushi Bai et al., 2024, LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding(ACL) and Official repository
- Cheng-Ping Hsieh et al., 2024, RULER: What's the Real Context Size of Your Long-Context Language Models?(COLM / arXiv)、Official repository and
run.sh - Huiqiang Jiang et al., 2024, Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization(Google Research)