
- Date: 2026-10-23
- For graduate students, early-career researchers and research engineers
- Scope: Four items have been observed and developed, 50 frozen candidates, and four input conditions; eight text transformations have been completed on October 24, 2026, and the semantic and label maintenance checks have been completed.
Summary: Ask what the result will make you change the explanation
"Put the evidence to the front, and reranking will be better" is an easy-to-accept intuition. In the previous article, it was found that among the 18 regression inquiries, the recalled relevant documents of Article 17 were not cut off. It's not a few more examples that fit the story, but to arrange a comparison that might fail.
We ran eight hundred model scoring operations on four probes, two long and two short. For each, we compared the original text, an identical reconstruction, and versions with the evidence moved earlier or later. Moving it earlier lowered ranks in both groups and yielded no improvement in the ranking metrics. Retaining every sentence need not preserve meaning after reordering. All eight transformations subsequently passed a review of meaning and label preservation, allowing us to reuse the original relevance labels. That resolves the outstanding label check. The sample is still small, and several factors changed together.
Reproducing value: Distinguishing comes from forecasting, not from experimental scale
The so-called differentiation experiment is to let different interpretations give different expectations for the same observation. If the regression is mainly because the relevant evidence has not entered the model, the visibility of the recovery evidence should bring selective improvement; if the order change can still change the score when the text is complete, it is necessary to consider the position, the organization and the modeling preference of the model. Two phenomena may also exist at the same time, and there is no need to force the only reason to choose.
The tasks of the SciFact paper report include the support or rebuttal of scientific propositions, as well as the positioning of evidence sentences; this article follows its public annotation and only evaluates document relevance. [1] Documents that support and refute claims may be related documents. The retrieval score therefore cannot be interpreted as a scientific judgment of Accuracy, let alone a reranking score as the model's confidence probability of a claim.
This article continues the same search research project, and the newly added controlled variable is the sentence position in the relevant document. Models, queries, titles, and candidate lists are all frozen. The four probes come from a hundred development queries that have been observed before, neither randomly representing samples, nor new confirmation sets. 300 confirmed inquiries will continue to be retained, and the fusion plan will still be left for subsequent comparisons.
Core Ideas and Formulas: Simultaneous Observation with Cutoff and No Cutoff
Make For documentation The original labeled evidence sentences are indexed and merged, for the rest of the sentences. Move forward inputs are spliced according to "evidence block, other blocks", and the backward input is reversed; each block is kept in the original order, and all sentences are kept once. Consolidation is used for operational positioning, which does not mean that multiple reason sets must appear together to support judgment.
The cross-encoder reads the query and document together, and outputs the scalar score:
Among is a query, is the input condition, is a sentence sequence transformation, It is the frozen weight; tokenizer limits a pair of input to five hundred and twelve tokens. Token is a sub-word unit read by the model, and the joint upper limit also includes queries and special symbols. It cannot be mistakenly written as an abstract of 512 words.
The main indicator nDCG@10 measures the sorting quality of the top ten related documents. There is only one label-related document for each query in this query; it is located in the first At the time of the name, the top ten indicators are , more than the top ten is zero. The pre-registered pairing amount is:
Indicates the nDCG of the query, the two horizontal lines are the mean of the two queries. It's just that the descriptive group is poor: the two groups of themes are not randomly matched with the difficulty of the document, and they cannot be regarded as a truncated independent causal effect, let alone the four samples can be used as the overall significant claim.
Guide to the official implementation: Labels, text and truncation are aligned layer by layer
Read the data format description of SciFact first, distinguishing the document number, the collection of evidence, and the period index from scratch. [2] This time, the original training annotation and clause corpus are used to locate the evidence into the actual sentence; the language model is not allowed to regenerate the evidence of "looks reasonable". After the original summary removes the first and last blanks sentence by sentence, the splicing result must be consistent with the BEIR text.
Read BEIR's Rerank.rerank, to confirm the spelling of the title, space, and abstract, and how the candidate score corresponds to the document number. [3] Finally read Sentence Transformers' joint tokenization and predict: This time the official model path is executed, only the text view of the relevant document is replaced externally, and the network is not modified. [4] Whether the end of the evidence sentence of the character offset audit record falls in the reserved input, the complete sentence can be seen and only partially appears separately.
The model is fixed to cross-encoder/ms-marco-MiniLM-L6-v2, the model and tokenizer revisions are 233902d25c440f23af6f7d6e94d2946bac0bee0a. The model card describes its paragraph-oriented rearrangement; this is not a domain capability guarantee. [5] The official source code submission, line number, license and actual byte check record are shown in the source code map.
Minimum experiment: first select materials, then look at model scores
Select the rules to freeze before reasoning in this article. First, it is required that the query has annotated related documents, and is in the fifty candidates; the original token number before and after the transformation is the same. The long text requires that all the evidence is visible after the forward shift, and at least one sentence after shifting back is incomplete; the short text requires that all conditions should not exceed the upper limit of the length and the evidence is all visible. Each group is sorted by the hash of the query number with the fixed prefix and the first two items are selected, and the selection is not according to the model winner or loser.
There are ten long texts and thirty-seven short texts in the qualified pool, and finally the long group 454 and 30 are finally selected, and the short groups are 280 and 155. The original length of the target document joint is 608, 728, 376, and 226, respectively. Only the annotation-related documents of each query are modified, and the remaining forty-nine candidates are completely preserved; therefore, this is a diagnostic probe with the help of labels, which cannot be improved as a real search without a label.
The four conditions are re-score all candidates. The original reconstruction is to achieve negative control, which is used to grasp splicing and mapping errors; short texts are truncated controls without evidence, which is used to observe the sensitivity of the order itself. The evidence visibility check of the long text is a positive control of the operation, only to check whether the transformation has done a predetermined thing, and the score of the model must not be preset.
Run fixed CPU single precision, four threads, batch 16, random seeds 107, no training. First, use the first two candidates of the same query plus relevant documents to retry the run. This time, each one is actually two candidates, with a total of 32 points; it is estimated that the official reasoning is about 16.7 seconds, which is lower than the 1,200-second budget. 800 pairs of official operation, 20.86 seconds of reasoning, the total time is 24.60 seconds, and the peak process memory is 786579456 bytes, about 0.73 GiB.
Practical verification: the evidence is complete, and the ranking is not guaranteed to improve
The following table follows the original correlation label that has been checked, and the value is consistent with the calculations saved before the review. The identity control is exactly the same as the original text, so it is not repeated.
| Query and Group | Original related document ranking | Move up | move backwards | Move forward, reduce back, move nDCG |
|---|---|---|---|---|
| 454, long text | 1 | 1 | 1 | 0.000000 |
| 30, long text | 1 | 2 | 1 | −0.369070 |
| 280, short text | 1 | 1 | 1 | 0.000000 |
| 155, short text | 2 | 4 | 2 | −0.200253 |
The long-group mean was −0.184535 and the short-group mean was −0.100127, a difference of −0.084409. For query 30, moving the evidence earlier brought all four evidence sentences fully into view, up from just one. Yet the target score fell from 4.870255 with late placement to 3.922231 with early placement. Query 155 had no evidence truncation, and early placement still lowered its rank. Both are local counterexamples that passed semantic review. Neither isolates the causal effect of truncation.
You can't just watch the rankings. The score after the forward shift of 280 rose from 5.811522 to 7.137649, which is still the first, so the nDCG remains unchanged; the query of the 454 score drop has not dropped out of the first. The ranking is discrete and ceiling, and the zero indicator change does not mean that the model is not responding. Three of the four original texts are already first, and further limit the observable room for improvement.
The actual effective input of each condition is 75683 token, and the candidate Recall is all one. Shape check is input , the last floor representation , fraction , respectively represent batch, joint length, hidden dimension and single-point output. The score of 200 original texts is exactly the same as the previous article, and the identity control error is also zero.
Evaluation Protocol: Which Interpretations Still Mixed

The prefix window in the figure shows long text; the whole content of the short text is in the window. The four conditions share the material audit, the semantic check is separated from the score calculation, and the audit package does not display the model results. However, the main text and chat have disclosed the results, so it cannot be claimed that this time is an independent blind review.
Move forward at the same time to change the location of the evidence, the connection between the front and the back, and what non-evidence sentences are left after the cut is cut off. Even if the total amount of tokens is the same, the context seen by the model is not equivalent. Therefore, the current author can only infer that the complete evidence of the evidence is not a sufficient condition for the improvement of the order, and the chapter or sequence effect is worth continuing to distinguish. It doesn't prove that truncation is never related, and it doesn't prove that another "covering useful ranking" explanation has won.
This experiment did not specifically extract the eighteen regressive samples of the previous article, but test a stronger and more general recovery intuition; it cannot estimate how many of those regressions are caused by truncation. In the future, you should continue to abide by the registered fixed ranking integration comparison. If you can't see the advance is not good, you can temporarily change the model, pick new samples, or try the confirmation set in advance to find good results.
Failure Troubleshooting and Semantic Check Boundaries
The first material build stops due to the failure of the text consistency assertion: the original clause has a blank at the end, and the direct space connection is inconsistent with the BEIR export. Fixed to remove the first and last blanks, the full corpus of 5,183 can be reconstructed; this repair occurs before the sample selection freeze and any model before this article. The source code reading failure caused by the network restriction also keeps the log, and the subsequent anonymous retry verification is passed.
The independent script does not call the indicator function of the main running script, but reorders from the saved score, calculates the loss and pairing difference, and checks all 800 pairs, candidate sets, sentence arrangement and input identities. Seven types of damage tests were rejected, including score change, missing candidates, token number and forged review. It proves that the deterministic ledger is consistent and does not prove that the text is legally legal.
Semantic check covers eight transformations of the forward and backward movement of the four queries: whether the reference is still clear, whether the relationship between the negative and time has changed, whether the description of the control group has been dismantled, and whether the complete document still maintains the original correlation. The eight inspections were passed on October 24, 2026, and the relevant records and material hashing have been filed separately. The original blank reply table and runtime pending logs are preserved as historical evidence, reflecting the current status in the follow-up records with dates. The inspection did not provide itemized reasons, so it could not claim that fine-grained misinterpretations were obtained. The original data set evidence annotation, script recalculation and semantic judgment answer different questions respectively, and cannot replace each other.
Code and instructions for reproducing the results
This technical package has been disclosed and verified: Source directory、README and run instructions、SOURCE_MAP source map. The fifty-six files of the public version are matched with local and ZIP one by one, and the running entrance is the same as the previous version, which follows the real test run record of the public copy of the previous version. Contains configuration, material, dependency lock, pair-by-pair score, auditor and no-score review page; large resources are obtained by fixed version and hash. The newly added semantic check records and updated documentation have been exposed with the fixed version; the experimental code, input and values remain the same.
After installing dependencies and executing data preparation by README, the minimum real command is:
HF_HUB_OFFLINE=1 TOKENIZERS_PARALLELISM=false python run.py --cache ./cache --out replay-smoke --smoke
It is expected to generate 32 pairs of scores, input hashes, stage logs, shapes and indicators calculated according to the original label; delete the test run parameters and specify a new directory to run 800 pairs. Run again python audit.py --out replay-smoke Independent recalculation. Original results, failures, and blank audit tables are preserved at the same time. The re-run program still outputs the pending status when it is frozen; when explaining the confirmed material this time, it is necessary to jointly read the follow-up inspection record with hash, and the program output cannot be regarded as a new review.
Summary
A useful minimal experiment doesn't have to make the new method win, but it has to make your explanation face the possibility of losing. This article puts the visibility, sequential change and realization errors of the evidence into the control, and also exposes the limitations of small-sample ceilings and semantic legitimacy. The semantics and labels keep checks have been completed, and the next step will take these boundaries into a strong baseline comparison, and do not promise in advance that new methods will win because of the establishment of the counter-example.
References
Search and fixed source code verification date: 2026-10-23. The paper supports the background of the task, and the values of this paper come from the actual saved logs; the semantic and label keeping check was passed on October 24, 2026.
- David Wadden、Shanchuan Lin、Kyle Lo、Lucy Lu Wang、Madeleine van Zuylen、Arman Cohan、Hannaneh Hajishirzi,2020, Fact or Fiction: Verifying Scientific Claims,EMNLP, Conference page.
- SciFact author, Description of the data format of the fixed version.
- BEIR author, Fixed version Rerank implementation.
- Sentence Transformers maintainers, Pinned version of CrossEncoder.
- Sentence Transformers, The model card of this weight version.
Apps by the author
The following is the author's own product promotion. In addition to writing a technical blog, I am also doing some small tools that make everyday life more organized. Welcome to the two apps I independently developed:
- Daily Expense Journal: Track your income and spending, plan a budget and set savings goals. View on the App Store.
- One Small Action: Pick a small action that fits the time and energy you have today, then record how it felt and what you noticed. View on the App Store.
Find more articles and projects on my website: zffjzz.top. I’d love to hear how the apps work for you and what I could improve.