
- Date: 2026-10-22
- For graduate students, early-career researchers and research engineers
- Experiment Scope: Freeze candidates generated by the real SciFact corpus, 100 have been observed and developed queries, each 50 candidates; 300 confirmation queries are not evaluated.
Summary: A reasonable explanation, how to become a hypothesis that can be lost
The reranking model did not increase the mean, and the easiest explanation was that "the scientific text is too long and the evidence is truncated". This explanation sounds reasonable, but it may just be a story to make up after knowing the results. This article re-runs the real cross-encoder, defines the failure first as a candidate, ranking and score computable status, and then checks how many regressions can be explained in the truncation.
Of the 18 queries whose rankings worsened, 17 had no truncation in any retrieved relevant document. This leaves room for truncation to matter, but rules it out as the explanation for every regression. This article documents the failures and registers the next stage of the study. It makes no claim that a new method has won.
Reproducing value: store phenomena and causes separately
The previous retrieval experiments have established terms search, dense search, countdown ranking fusion and cross-coded reranked assets. Countdown ranking is integrated, that is, RRF, which only uses the results of the ranking combination of the document in each list; the cross-encoder reads the query and the document together, and outputs a score for each pair of inputs. In this paper, the original first 50 candidates for RRF are fixed, and the same minim reranking model continues to be used.
The paper report must be separated from this result. BEIR paper compares multi-domain search methods and points out that different methods have generalization and computational cost differences; it does not guarantee that each reranking will benefit. [1] The original task of SciFact also includes support, refutation and reasoning, and only document relevance is evaluated here, and retrieval indicators cannot be interpreted as scientific fact judgment ability. [2]
Compared with the mechanism and evidence discussion of the previous article, this article selects the search asset as the only main line of research for the next six articles. Compared with the previous reranking experiment, the model and candidates remain unchanged, the new variables are diagnostic layering, and the new products are recalculated classification and research registration. All 809 development queries have a history of observation, of which 100 cannot be repackaged into independent test sets.
Core Ideas and Formulas: Which Floor to Locate the Loss First
Make for query a collection of document-related documents, for fifty candidates, It is the first ten sets after the rearrangement. Judge first Whether it is empty, then judge Is it empty. When there is no relevant candidate, a model that only rearranges an existing candidate cannot be saved.
The main indicator nDCG@10 is the cumulative gain of the top ten normalized losses: the higher the relevant documents, the greater the contribution, and then normalize with the ideal sorting. Here the correlation is a binary value, and the query indicator is written as:
Among is the -th ranked document. The indicator equals one when the document is relevant. For each query, subtracting the original RRF value from the reranked value gives ; Positive numbers are improvement, and negative numbers are regression. The same points are processed in descending order of the string document numbering, and the auditor and the main evaluator cannot be used differently.
Candidate oracle is the ideal score for moving all relevant candidates to the previous one . It reads the tags and can only be diagnosed and cannot participate in actual retrieval or model selection. Thus, there are:
On the left is a gap from the perfect sort; the right side represents the gap that cannot be recovered from the current candidate pool, and the sorting gap that can still be recovered within the current candidate. It is an algebraic decomposition under fixed annotation, not a percentage distribution of the cause, nor the causal contribution of the two modules.
Guide to the official implementation: track along a pair of inputs
Read BEIR's first Rerank.rerank: It selects candidates, forms text according to the title, a space, and an abstract, and then returns the score to the document number. [3] Then read the batch function of Sentence Transformers to check how the query and document are truncated; finally read predict Evaluation mode, forward pass, and output conversion. [4] This time, these official paths were directly executed, and the neural network was not rewritten.
The fixed model is cross-encoder/ms-marco-MiniLM-L6-v2, the weights and tokenizer versions are both 233902d25c440f23af6f7d6e94d2946bac0bee0a. The model card indicates that it is used for MS MARCO paragraph sorting. [5] This provides the motivation to examine the difference in the task, but it cannot be asserted that it must misread the scientific concept just in terms of training area. The source map record actually downloads the verification submission, the function line number and the license.
Minimum experiment: re-run after freezing
Inherit the 100 query numbers, 50 candidates, and relevance labels in the publicly pinned commit. Candidates are historical retrieval from the complete 5183 corpus, without shrinking the corpus to retake points. There is no part of the Git revision number available for the download resource, and is identified with the real public address and archived SHA-256; cache is reused after file verification.
Experimental fixed title plus summary, joint length upper limit 512, original output score, CPU single precision, four-thread, batch 16, and random seeds are 106. First execute 5 queries multiply 10 candidates to run, and then run all 5000 pairs. The estimated reasoning for the test run is 128.35 seconds, which is lower than the budgeted 900-second budget, and the process peak memory is also less than 4 GiB, so the range is not reduced according to the effect.
Formal operation with import, resource checking, model loading and evaluation for a total of 133.41 seconds, the process peak RSS is 893075456 bytes, about 0.83 GiB; this time there is no training, an answer is generated, the whole library is re-coded, or the GPU is used. Shape check record input , the last layer hidden state and fraction , respectively represent two pairs of input, joint sequence length, hidden dimension and single sorted score. The score-by-pair score is exactly the same as the previous freeze record.
Actual verification: five types of ledgers, do not affix semantic labels first
Classifications are mutually exclusive in the order of tables. Only by excluding the missing candidates and the top ten misses can the remaining query judge gain, regression or equality; therefore, there may be regression in the "top ten misses".
| recalculation status | number of queries | Allowed judgments |
|---|---|---|
| Candidate No relevant documents | 9 | The current candidate internal reranking is unrecoverable |
| There are related candidates, the top ten missed | 8 | There is a sorted recovery space |
| Top 10 hits, but nDCG drops | 16 | Hit doesn't mean the order is good enough |
| Top 10 hits with improved scores | 20 | Local improvement needs to be retained |
| Top 10 hits with flat scores | 47 | Main indicator has not changed |
The average nDCG of the original RRF and reranking is 0.670906 and 0.669133, respectively, and the candidate Oracle is 0.906131. The average candidate gap is 0.093869 and the sorting gap is 0.236999. Overall 20 wins, 62 draws, and 18 losses; the number of flats in the category table is small, because the first two categories have given priority to a part of the query.
In order to prevent only the most dramatic stories, each non-empty category is the smallest of the SHA-256 after the query number is fixed with the fixed prefix. The first relevant document of the query 456 was reduced from the sixth to the twelfth, and the related candidates were not truncated; there are still two related documents in the top ten of the query 963, but the nDCG has been reduced from 0.831555 to 0.491149. The example only illustrates the meaning of the classification, does not represent the overall category, and is not manually labeled as "the term understanding failed".
Competitive Interpretation: Let Intuition Face a Counter-Example

The first explanation is input truncation : the key related text does not enter the model, so the sorting fails. The second explanation is sorting coverage: even if the relevant text is complete, the cross-encoder may still cover useful ranking information in the original search. The latter is not equivalent to the domain mismatch, and may also involve interference candidates, label coverage, and differences in score targets.
Only 91 queries with relevant candidates are compared: 75 have no relevant candidate truncations, of which 17 have regressed. The corresponding ratios are 22.67% and 6.25%. Nine candidate missing queries do not enter this denominator, otherwise it will be incorrectly registered as "no truncation and no regression".
The observations do not bear out the simple prediction that queries in the truncation group would be more likely to regress. Group assignment was nonrandom, and the groups differed in size and text difficulty. Removing tokens may leave all the evidence needed for retrieval intact. Conversely, an intact relevant document tells us nothing about truncation in irrelevant candidates. The result rules out an overly strong claim about a necessary condition. It does not estimate the causal effect of truncation.
Evaluation protocol and Research Registration: Decide how to admit defeat in the future
The main assumption in the next step is: a fixed 50 candidates, only a cross-encoder is still scored once, and the original RRF sequence is preserved by the 60 constant-to-reverse ranking of constants, which can reduce the regression caused by the coverage of useful rankings. The main indicator of the predetermined confirmation set is the average nDCG difference of the plan relative to the pure reranking, and the actual gain threshold is 0.01; at the same time, the difference between the relative pure RRF must be reported, and a weaker control cannot be defeated only.
This is an assumption, not the result of this article. It is inspired by the results of the development, and the registration time is earlier than this replay and future intervention, but it is later observed and cannot be called a pre-registration that has never seen results. Fixed registration documents and their hashes are reserved for public reservations, and subsequent modifications must be accompanied by dates and reasons.
Subsequent design evidence retention and matching control, the text transformation must be checked by the real person for whether the label is maintained; then compare pure RRF, pure reranking and a fixed fusion scheme, control the number of candidates, counting the number of times and report the token cost. Only the two-factor experiment of context processing multiplication and fusion is done if conditions permit. There is no actual manual reply, and the "semantic unchanged" judged by the model cannot be used as a review.
Each article can multiply 100×50 pairs, CPU reasoning for 1200 seconds, and peak memory of 4 GiB, and do not train new models. Keep the logs over budget and stop, and do not delete the query or append the model by winning or losing. After the final freezing plan, 300 pieces of confirmation query are used to confirm the query; if the threshold is not reached, the report does not support the predetermined actual income, and the reference is not continued on the confirmation set. Query may share evidence documents, and do not treat the number of querys directly as independent sample sizes.
Troubleshooting and limits of the evidence
This time, the old shared environment was stagnant and the operating environment was stagnant; when the reinstallation value was automatically upgraded, the compatible version was fixed before the inference. The first time the leaked test run parameter is started, the model score is interrupted during the dependency import stage, and the model score is not generated; the failure log is reserved, and then the correct test run and the official operation are both exited to zero.
The audit sequence is to enter identities, candidate coverage, score by score, query indicators by query, and classify the denominator, and finally look at the explanation. Independent standard library scripts recalculate the complete results and correlation tables, and all eight damage injections are rejected, including modification of classification, difference value, forged truncation, and claiming to use acknowledgment sets. It is automatic arithmetic verification, not artificial semantic gold label.
True relevant documents that are not covered by the label may still exist; model pretrained pollution is not audited; a deterministic replay does not provide cross-seed or cross-domain stability evidence. We keep these limitations in order to allow the next experiment to target a clear problem, rather than chasing a story that always explains any result.
Code and instructions for reproducing the results
The full source code has been made public and verified by pinned commit: Source directory、README and run instructions、SOURCE_MAP source map. Contains dependency locks, configuration, research registration, pairwise results and independent auditors; public copies match 69 files on a local archive.
After installing dependencies, running data downloads and hash verification according to README, the minimum real command is:
HF_HUB_OFFLINE=1 TOKENIZERS_PARALLELISM=false python run.py --cache ./cache --out replay-smoke --smoke
It is expected to get the scores, logs, shapes and indicators of five queries multiplied by ten candidates; delete the test run parameters to run the complete range of this article. Run again discover.py and check_discovery.py Generate and check the classification independently. Audits with saved results do not require model inference, but cannot replace real experiments.
Summary
From failure to research, the key is to specify which observations support the interpretation and which observations will force themselves to revise the interpretation. The progress of this article is not to write a small negative result more attractive, but to implement the 18th step backwards as a positionable, recalculable, and rebuttable research starting point, and leave a valid confirmation boundary.
References
Search and fixed source code verification date: 2026-10-22. The paper is used for the background of the task and method, and the numbers in this paper are from the actual log saved with the code.
- Nandan Thakur et al., 2021, BEIR, NeurIPS Datasets and Benchmarks; Paper V4.
- David Wadden et al., 2020, Fact or Fiction: Verifying Scientific Claims, EMNLP; Conference page.
- BEIR author, The official rearranging implementation of fixed submissions.
- Sentence Transformers maintainers, Fixed CrossEncoder source code.
- Sentence Transformers, This weight corresponds to the model card.