
- Date: 2026-09-29
- For graduate students, early-career researchers and research engineers
- Scope: Real model, freeze candidates, and four preprocessing controls; development set exploration, not a complete benchmark ranking.
Abstract
In the previous article, we used the same batch of candidates to compare the original fusion sorting and cross-encoders, and found that the reranking did not improve the main indicators. The next most common story is: "The abstract is too long, and the important information has been cut off." This explanation sounds reasonable, but it is not evidence. Today, split it into two interventive variables: whether the document is titled, and how many tokens are reserved for pair input. Keeping weights, queries, and candidates constant can check how sensitive the method is to the input protocol.
This article compares the four pre-registration conditions, and saves the actual input length, score and ranking one by one. We are not only concerned about which one is higher, but also whether the same relevant document has been truncated, which queries change direction, and whether the results can support the next assumption. All figures come from this real operation; the interpretation of the location of the evidence is still to be verified and inferred.
Contents
- Reproducing value and boundaries
- Core ideas, formulas and modules
- A guide to the official implementation
- A minimal experiment and evaluation protocol
- Results and failure analysis
- Testable research questions
- Code and instructions for reproducing the results
- Summary and References
1. Reproduction value: first check the input, and then evaluate the method
The original SciFact task involves retrieving evidence for scientific claims, judging whether it supports or refutes them, and locating the rationale. Here we use only the document relevance labels in the BEIR export. Ranking a relevant document higher does not by itself verify a claim. [1] Earlier articles fixed the retrieval outputs for the full corpus of over five thousand documents. We reuse the previous article’s development query subset to study how text is presented at reranking time, holding the retrieval and training methods outside the comparison.
The title can summarize the subject, and may occupy the abstract space; longer inputs may retain key sentences, or may add interference. Neither are necessarily profitable. If you only increase the maximum length and find that the score has risen, you still need to ask: Is the content changed, the input position has changed, or the dominant average of some query? The value of reproduction is to turn these problems into checkable records.
What the paper reports: The working instructions of Nogueira and Cho use BERT to coding the route of the reordering with the candidate paragraphs. [2] What we tested: Perform trained MiniLM cross encoder without reproducing the paper training, and not using its report results as a comparison. Author’s interpretation: Protocol changes may explain some method differences and require the current four groups of experimental tests.
The core idea: put the text structure into the scoring function
The cross encoder lets the query and the document interact in the same Transformer, and outputs the correlation score. can be written as:
Here, is a query, is the original document, Indicates whether the title is spliced, is the field structure, is the upper limit of length tokenization and truncation process, is the fixed model weight. Change the title or length, even if the weights are exactly the same, the inputs that the model sees are different. This output is an original scalar score, which is not interpreted as the relevant probability after calibration. [3]
Tokenizer breaks the word into token, that is, the text unit that the model handles. The upper limit of the length includes queries, documents and three special symbols, which does not allow the summary to occupy the specified length alone. Batch input shape is , is the batch size, is the sequence length after completion; the hidden state is , the classification header output . The official implementation uses the longest sequence to prioritize the truncation, and when the time is overdue, it is preferentially deleted from the longer side. [4]

All four groups receive the same candidate pair and use the same weight. The shared model in the figure represents parameter sharing, and does not mean that four inputs are spelled into a sequence at the same time; the label is only used when pairing and evaluation. The original RRF is a fixed order left by the fusion of the reciprocal ranking. As a reference reservation, the preprocessing of the retrieval stage has not changed.
3. Guide to the official implementation
Read BEIR's first Rerank.rerank: It constructs the query and document pair by candidate ID, splices the title, space and abstract, and then re-correses the predicted score to the document ID. [5] This article retains the original bytes of this file; the summary conditions are realized by passing in an empty title to avoid rewriting the official scoring logic. The title condition maintains the splicing method of the previous article.
Read Sentence Transformers smart_batching_collate_text_only, confirm that the string removes the first and last blanks, dynamically fills, longest_first and max_length. Then go predict See the model switch to evaluation mode, turn off the gradient, and return a single label score. [4] Finally, check the model card and configuration, and confirm that it is actually a six-layer model, a hidden dimension of three hundred and eighty-four, and a label, rather than guessing the structure according to the name. [3]
The full text comparison of the official file, license and installation version after hashing or normalized wrapping is done locally. The source map gives the full submission number, file line number and local entrance. The independently written parts include conditional arrangement, token auditing, matching difference calculation and drawing; these are not the original implementation of the paper, and cannot be mixed with "all official reproductions".
4. Minimum Experiment and Evaluation protocol
Using a fixed 100 queries, each of fifty RRF candidates, a total of 5,000 queries-document pairs. The query is still selected according to the hash rule of the previous article; there is no resampling due to this effect. The complete corpus scale is 5183, and the four groups do not rebuild the index, replace the candidates, and do not train.
We crossed two text formats, title plus abstract and abstract alone, with two token limits, 256 and 512, for four conditions. We fixed the model and tokenizer revisions, used single precision on CPU with four threads and batches of 16, and held the random seed fixed. The seed controls the run state, without resampling the queries. Varying the maximum length changes the actual token count and compute, so this experiment cannot compare model abilities at equal compute.
The main indicator is nDCG@10, that is, the discount gain that considers the relevant document rankings and normalizes according to the ideal sorting. The auxiliary indicators are the first MRR@10 related results, and the top ten and top 50 Recalls. The score is in descending order of the document ID string when the score is the same. Because the candidate collection is fixed, the Recall@50 must be the same; if it changes, it is prioritized to realize the improvement.
Each query retains four sets of scores and the pairing difference of the original sort. Also calculate the amount of descriptive interactions:
It is the average nDCG@10 of the corresponding conditions, the subscript 1, zero means reserved, and the title is removed. Describe whether the title effect changes with length, not a statistical proof of the overall causal interaction. All conditions shall be registered in advance and all reported, and the best conditions for the development set are not directly declared as the final plan.
The first command misses the test run parameter, starts the complete condition by mistake, and then interrupts and saves the exit code 130 and existing logs. After correcting, run five inquiries and multiply the four sets of trials, and only decide whether to expand or not according to time and memory. The official operation and the error start are kept separately, and the Existing Scores does not participate in the parameter selection. Resources, shapes and each stage state are released with the source code.
5. Analysis of actual results and failures
| Condition | nDCG@10 | MRR@10 | Recall@10 | Related pairs of truncations |
|---|---|---|---|---|
| Fixed RRF | 0.670906 | 0.637361 | 0.7775 | Not applicable |
| Title + Summary, 256 | 0.659686 | 0.615258 | 0.8125 | 95 |
| Title + Summary, 512 | 0.669133 | 0.626750 | 0.8175 | 20 |
| Summary only, 256 | 0.650207 | 0.602718 | 0.8100 | 86 |
| Summary only, 512 | 0.638613 | 0.592262 | 0.8000 | 18 |
What we tested: None of the four conditions beat the fixed RRF baseline on the primary metric, so the ordering of the methods did not reverse. With titles included, the 512-token condition exactly reproduced all five thousand scores from the previous article. Recall@50 was 0.905 in every condition. With titles, a longer window raised nDCG@10 by about 0.009447, with twelve wins, seventy-eight ties, and ten losses. With abstracts alone, it lowered nDCG@10 by about 0.011594, with ten wins, seventy-four ties, and sixteen losses. The title effect grew from 0.009479 with the short window to 0.030520 with the long one, giving a descriptive interaction of about 0.021040.
Each group of candidates has 108 label-related queries-document pairs. With the title, the window expansion reduces the number of cuts from 95 to 20; when the summary is only reduced from 86 to 18, there is no mean improvement in exchange. Query tokens are all reserved. After removing the title in the short window, the reserved summary token has increased from 1009770 to 1101794, but the main indicators are still reduced, so "it is better to see more abstracts" does not hold here.
Query 308 selected by the pre-registration rule, whose related documents are escalated from the sixth to the first under the condition of the title; the query 1050 and 1166 are both dropped from the first to the second. These are recalculable ranking changes, not semantic errors that have been explained clearly. One query improvement also cannot provide reasons for the other two backward queries.
Four groups of official inferences total 394.74 seconds, the entire operation is 418.88 seconds, the peak process of the process is 1040351232 bytes; the measured Python 3.12.14, macOS, CPU single precision. The timing includes the scope of the log, which does not include download and historical retrieval, and the GPU video memory is not measured. Fixed running sequence also means that the time-consuming is only for resource records, not strict performance conclusions.

The figure is generated by the matching drawing script to read the saved summary, the main indicators on the left and the fixed RRF reference are displayed, and the right side is the truncated related query - the number of documents, not the number of independent documents, nor the number of lost evidence sentences. There are no independent samplings, so no error bars are plotted.
Failure investigations are conducted at the level of evidence. First, check the candidate IDs of each group completely, excluding "changing a batch of easier candidates". Second, check the paired input length instead of the character length; the actual record of the query, title, abstract and special symbols, and the input id hashes is compared with the model entrance sampling. Third, the independent recalculation of the query indicators and the total value one by one, to avoid formatting or aggregation errors disguised as a model change.
Fourth, distinguish "relevant documents are truncated" and "sentences that support the judgment are truncated". Relevance labels only point to the documentation and cannot tell us whether the remaining prefix is sufficient. There is no artificial gold label, and there is no claim to manually confirm the cause of the error. Fifth, you can't just pick improvement cases; choose cases according to the absolute length effect of pre-registration, keep negative changes and all unchanged queries.
6. Rewrite intuition into a testable problem
One hypothesis is that the longer window improves ranking by retaining evidence near the end. Another is that the title or opening terms already suffice, so extra text mainly changes which distractors are visible. Both can account for some of the change in the mean, but they make different predictions. A follow-up could annotate evidence locations in the development sample and compare cases where evidence becomes visible with cases where it was visible all along. This requires independent human annotation, which has yet to be done.
Another differentiation experiment is to fix the actual summary token number, and then change whether the title exists or where it is. If the benefits only appear when the text budget is increased, it should not be attributed to the heading semantics itself. This design also needs to check the position change, truncated boundary and label validity, and should not directly package the title comparison into pure semantic causal experiments.
All these are follow-up questions, not conclusions that have been drawn. Currently, it is only responsible for one freezing model and a batch of used development queries. There is no audit of the model training corpus pollution, and the relevant documents that are not marked may be regarded as irrelevant; there is no cross-domain result, the overall Confidence Interval or "long-text understanding ability" conclusion. The next migration experiment needs to clarify the freezing scheme before changing the domain, and truthfully record the solution from development and exploration.
7. Source code and reproduction entrance
Fixed versions that have been released and anonymously verified: Full source code directory、README and run instructions、SOURCE_MAP official source map and Unified operation entrance. Fifty-three remote files are consistent with the local archives one by one.
After installing a fixed dependency and pressing README to obtain a locked resource, the minimum running command is python run.py --cache ./cache --out smoke-new --smoke. It performs four sets of real model inference, and outputs the score-wise, actual token counts, query metrics, summary and stage logs. Delete the test run switch in the full range, using the new output directory. This time, the actual completion of 100 inquiries and 50 candidates multiplied by four conditions; only running offline auditing cannot replace model reasoning.
The data version is identified by publicly archived and file hashed, and the model is fixed with the tokenizer. Cache identities include fields, length, model, data, hint settings, decoding methods and code hashing; each time the entry is re-inferenced, the old scores are not secretly reused. The package contains all self-writing modules, licenses and small verification records, and large resources are obtained according to the download steps of the verification. The local compressed package is separately accepted and archived, and the reader's entrance is subject to the public source code.
8. Summary
Preprocessing is part of the scoring function, and the method name cannot replace the input protocol. Reliable reproduction requires concurrent delivery of condition definitions, real input audits, query-by-query changes, and failure records. Even if a higher score setting is found, it can only indicate that the freezing system is sensitive to the settings; only by putting a competitive explanation and designing the next experiment that can distinguish them, will it start to approach the verifiable scientific research problem.
References
Retrieve with actual access date: 2026-09-29. See the source map for fixed version and file hashing.
- David Wadden et al., 2020, Fact or Fiction: Verifying Scientific Claims,EMNLP.
- Rodrigo Nogueira, Kyunghyun Cho, 2019 first edition, 2020 V5, Passage Re-ranking with BERT.
- Sentence Transformers, MS MARCO MiniLM-L6-v2 Fixed Model Card.
- Sentence Transformers 3.4.1, CrossEncoder fixed source code.
- BEIR, Rerank fixed implementation.