
- Date: 2026-10-25
- For graduate students, early-career researchers and research engineers
- Actual range: 4 SciFact development probes that have been observed, each fixed 50 candidates; the real MiniM has scored 400 pairs; the confirmation set is not used.
Abstract: Orthogonal interaction does not mean that both modules are valid
Suppose a set of methods includes two changes: "Make the evidence more advanced" and "reserve the original search ranking". Open the two together, the result is higher than the original model, can it be determined that both modules are valid? Can't: one of them can be harmful, and the other just eases the regression it causes.
In this complete operation of four combinations, we get a counter-example worthy of preservation: the average interaction effect is positive, but the evidence forward moves to reduce the average score in the two sorting backgrounds; the two modules are used at the same time, and there is no more than only ranking fusion. This conclusion only describes four diagnostic materials, not overall performance judgment. The value of reproduction is to disassemble "which module is useful" into a verifiable condition effect, rather than drawing another module diagram that looks reasonable.
1. Research goals: connect to existing problems, not start a new kitchen
In the previous article, the pure cross-encoder was integrated with the fixed ranking on the 100 development queries. This article follows the same main line of search research, and takes back the registered "whether the integration of context retention and ranking affect each other". The new variable is only enabled for two operations; the model, candidate, label, length limit, and fusion constant are kept frozen.
The cross encoder, referred to as CE, inputs the query and document together to the model, and outputs the correlation score for this pairing. The countdown ranking is integrated, referred to as RRF, which changes the ranking given by each system into the countdown and adds it together. RRF original paper Multiple retrieval system fusion experiments are reported; it does not promise that all text rewrites will cooperate with fusion. This article examines your own combination, and cannot borrow the conclusion of the paper as a result.
SciFact original paper At the same time, scientific assertions, evidence documents and evidence sentences are provided. BEIR Provides a search and evaluation framework. We evaluate the location of the relevant documents, do not assess whether the assertion is true, and do not claim to reproduce the complete scientific fact-verification system.
2. Core idea: turn one deletion into four grids
Ablation study is to remove or close an operation, and observe the changes when the remaining conditions are fixed. If only one module is deleted from the complete system, only the contribution of "the other module has been turned on" is measured. To know if it will depend on another module, at least four grids need to be filled.
Make express the original text, Indicates that the evidence sentences marked with the original dataset are moved to the front of the abstract; let Indicates sorted by CE, Indicates the fusion CE and original candidate rankings. The forward shift here preserves the title, all sentences and the internal content of each sentence, but changes the order between sentences. It uses information related to known answers, yes oracle diagnostics: In order to understand the invalidation mechanism, the gold standard information is borrowed and cannot be deployed as a real retrieval scheme.

All four conditions in the figure use the same candidate set, and metrics are paired by query. The icons illustrate the procedure only. We reused the four earlier probes, two long and two short, keeping the original texts, the versions with evidence moved earlier, and the review records fixed byte for byte. We neither created new rewrites nor selected examples with conspicuous effects. We had already seen the scores for the original, earlier, and later placements, so this was no blind first test. The long and short groups retain the earlier rules for evidence visibility under early versus late placement. Moving evidence earlier may therefore expose nothing new relative to the original. The condition names describe what we did to the input. To interpret a change, first check what evidence was actually visible in that example.
3. Formulas: first explain which background is effective
Remember for query in operation combination nDCG@10 below. It divides the discount proceeds of the top ten related documents by the ideal sorting income. The larger the value, the higher the relevant documents. Each query in this group has exactly one relevant document, if the ranking is , then , otherwise it is zero.
The amount of interaction is defined as:
Square brackets are the gains that enable fusion in the background of the forward moving and the original text background respectively. The roles of two factors can also be exchanged to obtain the difference in the same difference.
Fusion uses the formula that has been frozen:
is the document of the same candidate set, is the original RRF candidate ranking, It is the CE ranking under the current text condition, both from the beginning. The constant is 60 and the equal weight is not adjusted; the same points are sorted in descending order of the document identification string. The fusion is "original fusion ranking and rear ranking", not retraining the model, nor did it expand the candidate set.
Contributions across another factor can also be computed, but cannot be masked: e.g. The average effect of The arithmetic average of the forward-shifting effect in the background. In this paper, the condition effect, average effect and each interaction are also saved, not only a total number is reported.
4. Guide to the official implementation and minimum experiment
Read BEIR's first Rerank.rerank: How it splices the title and the summary, how to put the batch score back to the query and document identification. Read Sentence Transformers smart_batching_collate_text_only and predict: The joint input adopts the longest side priority cut, and the maximum total length is 512 tokens; token is the input unit after the tokenizer is processed. Finally check how the BEIR review code is called pytrec_eval. Fixed commits, specific line numbers and local corresponding paths are all listed in the source map.
This actual use ms-marco-MiniLM-L6-v2, freezing the model and tokenizer revision as 233902d25c440f23af6f7d6e94d2946bac0bee0a. The operating environment is Python 3.12, PyTorch 2.6.0, Sentence Transformers 3.4.1, CPU, single precision, four threads, batch sixteen. The original logit is an un-normalized scalar for the model output and is not interpreted as the correct probability. The model card indicates that the training task is MS MARCO, which is not retrained in this article, and there is no application of hardware speed in the card.
First use four to query the original top two plus related documents to test the run. After this merger, there are 16 pairs of scores; and then run the two text conditions of 200 pairs each. Four grids only require two rounds of real model scores, as each round score can be used for both pure CE and fusion. There are also four pairs of shape cores to be forward, and the accounts are separately kept. Spot check input shape is , the last hidden layer is , the output is : The three represent batch and length, batch and length and hidden dimension, and batch and single correlation scores.
V. Evaluation protocol and the actual results of this time
The four queries are from the observed development data; the relevant documents are already in a fixed fifty candidates, so the Recall@50 of all grids is one. It is not possible to claim that the Recall problem has been resolved, this is only the result of probe selection. All the official results are given below, and the two numbers in the column name correspond to and .
| Query/Group | Original CE (00) | Move CE (10) | Original Fusion (01) | Movement Fusion (11) | Mutual |
|---|---|---|---|---|---|
| 454/Long | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 0.000000 |
| 30/long | 1.000000 | 0.630930 | 1.000000 | 1.000000 | 0.369070 |
| 280/short | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 0.000000 |
| 155/short | 0.630930 | 0.430677 | 1.000000 | 0.630930 | -0.168817 |
| four mean values | 0.907732 | 0.765402 | 1.000000 | 0.907732 | 0.050063 |
What we tested: Moving the evidence earlier reduced the mean by 0.142331 with CE alone and by 0.092268 with fusion. Fusion improved the mean by 0.092268 on the original text and by 0.142331 after the move. The difference between those gains is the positive interaction of 0.050063. The “both on” and “both off” conditions have the same mean, so comparing only those two hides both the loss and the recovery.
Query 30 is the easiest to understand: the relevant document is the first in the original CE, the second is moved forward, and the fusion is restored to the first. Orthogonal interactions come from mitigation to regression. One hundred and fifty-five inquiries are different: the CE ranking has dropped from the second to the fourth, and the fusion gives the first and second respectively; the fusion benefits are greater under the background of the original text, so the interaction is negative. The two constant queries must also be retained; the ranking index has steps and upper limits, the score change may not exceed the ranking, and zero interaction does not mean that the internal calculation is completely unchanged.
6. Failure investigation: what to check in addition to the correct arithmetic
Check the input identity first. The four materials are consistent with the hash of the existing audit package, and the token sequence of non-target documents is exactly the same as the score across text. The 400 scores this time are the same as the previous corresponding scoring points; this is the comparison after the completion of the new reasoning, not the cache replay is packaged as a new operation. If the original text reconstruction, sentence sequence, candidate list or identification is not matched, the process should be repaired first, and the indicators cannot be explained first.
Each text condition used 75683 non-padding tokens, and the lengths matched within every pair. What the model could see, and where it appeared, still differed. Inference took 10.56 seconds in total, the full run took about 13.22 seconds, and peak resident memory was 775897088 bytes. These timings come from one run on this machine and exclude a fresh measurement of initial retrieval. Fusion required no extra model calls, so the four conditions did not amount to eight hundred model inferences.
Finally, re-calculated independently. The separate script reconstructs the ranking from the original pairing score, uses the standard library to calculate the indicators and the interaction, checks the countdown score with rational numbers, and then compares it with the official evaluation results; Commands, exit codes, and input hashes are saved with the package. The first audit command was not executed due to the working directory error, and then passed in the correct directory; the first network failure of the source code download also keeps the record. Automatic recalculation is not called manual semantic judgment.
7. Research questions: Where can this ablation evidence support
Author’s interpretation: The results of this group do not support "evidence advancement will inevitably improve order", and it is not enough to interpret orthogonal interaction as general synergy. Moving forward at the same time changes the sequence of discourse, the position of evidence, and the context left after truncation; even if the complete document label is maintained, it cannot prove that the model only responds to evidence visibility. The short material will still regress without truncation, which is a counter-example that needs to be reserved.
The question that can be verified in the future is: Does the fusion gain mainly come from the restoration of the high-level documents of the original search, rather than "understanding more evidence"? At present, there are only four non-random probes, two in two groups, and the result is a ceiling. The significance cannot be calculated with four independent samples, and the duplication of the same deterministic reasoning cannot be used as a new sample. The expansion experiment needs to fix the selection, transformation effectiveness and grouping rules of more materials, and then observe the results.
This article does not replace the registered confirmation plan with diagnostic results. The next article summarizes the conditions and failures of this main line of research; the final report still retains the original CE, fixed fusion and original ranking comparison, and then uses 300 unevaluated items to confirm the query. The advance operation of the Gold Standard Assist does not enter the candidate method, nor does it adjust the constant after seeing the confirmation result.
Code and instructions for reproducing the results
Fixed versions that have been released and anonymously verified: Full source code directory、README and run instructions、SOURCE_MAP source map. It includes precise dependencies, locked resources, protocols, all pairing scores and independent audit scripts; the remote and local archives are consistent with each file.
After preparing the environment and hash-verified cache according to README, the minimum real running command is:
python run.py --cache ./cache --out replay-smoke --smoke
The expected products are two groups of model scores, four grid sorting, query conditional effects, interaction, token and resource accounts. Remove the last parameter to run the official range of 400 pairs of this article; the small candidate set for the test run cannot be used to replace the formal indicator. Only review and save results are executable python audit.py --out results, there is no need to re-download the model. The package already contains the required self-write module, and the complete data and weight are obtained through a fixed source.
Summary
The contribution of a module must be stated "under the conditions that have been opened". The value of the four grid control is to separate the effect of decline, recovery and combination. The most useful discovery this time is not a positive sign, but a query in the opposite direction of the two orthogonal interactions, and the boundary of "full opening is not the best".
References
Search and fixed source code verification date: 2026-10-25. The conclusion of the paper and model description and this experiment are respectively used, and the official benchmark numbers are not regarded as the results of this time.
- Wadden、Lin、Lo、Wang、van Zuylen、Cohan、Hajishirzi,2020, Fact or Fiction: Verifying Scientific Claims,EMNLP.
- Thakur、Reimers、Rücklé、Srivastava、Gurevych,2021, BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models, NeurIPS dataset and benchmark track.
- Cormack、Clarke、Büttcher,2009, Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods,SIGIR.
- Sentence Transformers, MiniLM cross encoder fixed model card. The link uses the official pinned version of the official fixed version that has been successfully read and verified.
- BEIR, Fixed version Rerank implementation;Sentence Transformers, Pinned version of CrossEncoder.
Apps by the author
I also build small tools for everyday life. Here are two of my apps:
- Daily Expense Journal: Track your income and spending, plan a budget and set savings goals. View on the App Store.
- One Small Action: Pick a small action that fits the time and energy you have today, then record how it felt and what you noticed. View on the App Store.
Find more articles and projects on my website: zffjzz.top. I’d love to hear how the apps work for you and what I could improve.