
- Date: 2026-10-26
- For graduate students, early-career researchers and research engineers
- Experimental scope: 100 observations and development inquiries; fixed 50 candidates; 300 confirmation inquiries continue to be reserved.
Abstract
Adding the old rankings to the cross encoder rankings, the average score has indeed increased. However, the relevant evidence has not improved the four probes, and the fusion has also damaged some of the original queries. The easiest story to write at this time is "overall effective, few exceptions"; the most worthwhile work is to give the boundaries of uncertainty, counter-examples and the next test. This article re-runs the real model, saves 5,000 scores in a complete manner, calculates the pairing interval according to the shared evidence document, and freezes the final confirmation analysis. There are local gains, so the title is no longer strong and says "the new method is not winning"; but the development set gain cannot be cashed in advance to confirm the success of the assembly.
Reproducing value: from one mean to a problem
SciFact original paper studies the relationship between scientific statement and evidence abstract, and provides support, refutation and evidence sentence annotation. [1] This project uses the retrieved data exported by BEIR, and only evaluates whether the relevant documents are in the front, and the rise of nDCG cannot be said to be accurate and accurate. nDCG is a normalized loss accumulation gain: the higher the relevant document, the higher the reward, and then divide it by the ideal sorted reward. [2]
The first chapter has completed the four grid comparison of "evidence forward × ranking fusion". The new question in this article is: How sensitive is the gain of the same fixed method to the resampling unit and individual evidence groups? What changes are statistical units, and the freeze is query, corpus, candidates, models, text stitching and fusion constants. We are still working on the development data that has been repeatedly observed; this is neither a new Test Set nor a re-ranking of the full SciFact benchmark.
Core ideas and formulas: pair first, then maintain the relevant structure
The cross encoder sends the query and the document together into the model, and outputs a correlation score, referred to as CE. This time, the MiniLM six-layer model trained by MS MARCO; the model card describes the training tasks and call methods, and its list results and GPU throughput are not this measurement. [3] The original order of the candidates comes from the fusion RRF of the existing inverse ranking. Fixed integration gives two rankings and other rights to vote: [4]
here is a query, is one of its fifty candidates, and They are the ranks from the beginning of the original RRF and CE respectively. The two rankings were from the same candidate collection; there was no extended recall, and no second CE reasoning was added. When the score is the same, it is arranged in descending order of the docstring number in descending order, and the actual floating point sum rule is retained.
We are concerned with the pairing difference of the same query, not the difference between the two irrelevant means:
Indicates the nDCG@10 of the query, . No matter how many candidates a query has, only one difference is contributed. Treating 5,000 documents as 5,000 independent samples will miscalculate the amount of model calculation as statistical evidence.
Statements that share the same relevant document may be interdependent. We use positive correlation labels to establish a query diagram: two queries share at least one relevant document, and the complete connected component is taken along the connection, and ninety-four groups are obtained, of which two groups of six groups and one of eighty-eight groups. Participate in groups even if the relevant documents are not candidates. This rule only uses the label number and does not look at the method score; it is still only a proxy that depends on the structure, and there is no guarantee that the different groups are completely independent.
is the -th list obtained by drawing 94 times with replacement from the 94 groups. A group drawn more than once is counted each time, and is the number of queries in the group. The denominator changes with the sampling, and the estimated object is still the query weighted average, and cannot be changed to "average of the mean of each group". Execute 10,000 times, take 2.5% and 97.5 fractions, and use linear interpolation. In addition, query resampling is done as a sensitivity control. The SciPy document describes the pairing and percentile method; this grouping algorithm is an independent teaching implementation, and its default BCa method is not copied. [5]

The picture shows the concept flow, and the measurement value is not drawn; the confirmation set is disconnected from the current calculation. The actual sampling index, 60,000 differential copies and intervals are saved in the source code result file.
A guide to the official implementation
Read the pinned version of BEIR first Rerank.rerank: How to select candidates, splice titles and abstracts, construct model input, and then write the score back to the original document number. Then read Sentence Transformers' smart_batching_collate_text_only and predict, check joint truncation, batch and activation functions. Finally, look at BEIR's evaluation portal, and confirm the calculation and sorting agreement of the indicators. [6]
Local official_rerank.py Consistent with the official file bytes, the installed CrossEncoder and fixed source code are consistent with the normalization of the line break. Input tensor random check as , the last hidden state is , the output is . The three dimensions correspond to the batch, sequence position and hidden width respectively, and the last one represents one pair of scalars for each pair. Keep the original logit, tokens and corresponding numbers one by one, and you can't just check the final mean "look similar".
A minimal experiment and evaluation protocol
Using 5,183 complete corpus, there are 100 development inquiries and 50 frozen candidates each. The model and tokenizer are fixed to 233902d25c440f23af6f7d6e94d2946bac0bee0a, the joint length is 512, the batch is 16, the CPU is four-threaded, single-precision, and random seeds are 110. There is no fictional Git version of the data download package, which is the download address, file version information and SHA-256 identification; see the lock file for all identities.
First do five inquiries and multi-ten candidate test runs, and then perform the official 5,000 pairs of reasoning. The formal model reasoning is 131.878 seconds, the basic pipeline is 136.153 seconds, and the base process peak resident memory is about 892 MiB; the statistical report takes about 0.961 seconds. The peaks are recorded before the statistical stage, so the peak of the statistical stage is not false. Four pairs of forward pass inputs for auxiliary shape check input are also recorded, and do not mix in the budget of 5,000 pairs. The cache hash verification has passed, and the download time and original recall cost have not been retested this time.
All 5,000 new CE scores are exactly the same as the previous save results. The three methods share the candidates, and the CE and fusion share a model score; pure RRF does not bear the new CE cost, and the three cannot be said to be equal to the total cost. The grouping rules write protocols before this reasoning and interval calculation, but the development results have been seen before, and it is still exploratory analysis.
Result: The interval cannot replace the confirmation evaluation
The average nDCG@10 of the three methods is RRF 0.670906, CE 0.669133, and fusion 0.702521. The following is the actual re-calculation, and the interval only describes the resampling sensitivity of the current development sample.
| Pairing comparison | average difference | Evidence group percentile 95% interval | Query sensitive intervals |
|---|---|---|---|
| Convergence minus CE | 0.033388 | [0.000883, 0.066319] | [0.001312, 0.068048] |
| Fusion minus RRF | 0.031614 | [0.002692, 0.062390] | [0.004802, 0.059444] |
| CE minus RRF | -0.001774 | [-0.049319, 0.044971] | [-0.049320, 0.043383] |
The Lord compares 20 wins, 66 draws and 14 losses. After deleting any group of evidence, the mean falls at 0.027500 to 0.037854, which is an impact diagnosis, not another Confidence interval. The two resampling sections are close, which does not mean that the correlation has been solved; most of the groups here are a single query.
The lower bound of the fusion range is slightly higher than zero, but it still covers the pre-registered practical threshold of 0.01. More importantly, these queries have been used for error analysis and proposed schemes: resampling only re-combines the samples that have been seen, and does not clear the method selection bias. It is not possible to write "development intervals not cross-zero" as "proven generalization is effective", nor can it be a stable victory from the lower bound of nearly zero. The original registration is required to check the average gain threshold in the confirmation set, and this article does not change it into a new significance rule without authorization.
There are also two repetitions: the fixed model re-inference, mainly to check whether the operation is repeatable; the sampling recalculation interval, mainly check how the current sample composition affects the mean. The model is not retrained this time, and there is no independent new data duplication, so it is impossible to estimate training randomness or cross-domain changes. The three comparisons in the table use the same batch of sampling indexes, maintaining the pairing relationship between the methods; they are not three independent evidences. We did not pick the most favorable column as the main conclusion.
All experimental ledgers and failure investigation
Of the eighteen queries that did worse under the cross-encoder, seventeen had no relevant candidate truncated. That weakens the explanation that lost relevant text accounted for most regressions. Across four evidence-placement probes, moving evidence earlier rather than later yielded a mean difference of -0.184535 in the long group and -0.100127 in the short group. The identity control gave identical scores. In the four-condition experiment, moving evidence earlier had a negative effect both with and without fusion. The positive interaction of 0.050063 does not make the combined method the best. Review of the original materials is complete. Older result files preserve their historical “pending review” labels, and confirmation records are kept separately.
The equal-budget comparison revealed a trade-off. Fusion improved seventeen of the eighteen queries on which the cross-encoder lost ground, but reduced its gains on thirteen of the twenty queries it had improved. Under the prespecified rule selecting the three lowest fusion-minus-CE differences, query 242’s first relevant document fell from first under CE to seventh under fusion, and nDCG dropped by 0.408765. For queries 1263 and 481, the first relevant document fell from first to second, and nDCG dropped by 0.369070 in each case. Query 242 has multiple relevant documents, so its first relevant rank alone cannot determine its nDCG. These rank changes tell us nothing by themselves about which medical statements the model understood or misunderstood.
Check the input hash, query document mapping, simultaneous rules and tokens audit in sequence, and then check the statistical grouping. The independent standard library script recalculation indicators, all copies and interval endpoints; seven categories of ranking damage and five types of statistical damage are rejected. The first anonymous data download is restricted by the network sandbox, and then authorized the read-only retry to succeed, and the failure record is kept. There is no deletion of queries, replacing seeds, or picking another sampling method for the more beautiful intervals.
Checkable Research Questions and Stop Rules
The question that is still open is: Can the fixed fusion achieve an average gain of at least 0.01 on the unrated 300 confirmation inquiries, and at the same time, what is the cost of pure RRF? In the next article, the model, text, candidate budget, constant and sorting rules are used; the grouping still generates a group list before the fraction of the model is merged according to the connection relationship of the shared positive documents. 10,000 resampling, confirm that the analysis seeds are fixed to 111, and report the query sensitivity interval at the same time.
If the mean gain on the confirmation set falls short of the threshold, we will report that the results do not support the practical hypothesis. Whatever the outcome, we will report whether the interval includes zero and whether it includes the threshold, without retuning the weights and testing on the same set again. Moving evidence earlier uses gold-standard evidence locations, so it serves only as a diagnostic upper-bound probe. Differences in model domain, missing annotations, and unobserved dependencies among topics may still affect the results. No experiment has isolated their individual effects.
Code and instructions for reproducing the results
The full source code of this article has been published: Directory at the pinned commit、Running Instructions README、Source Map SOURCE_MAP. The version has been anonymously downloaded, the 100 files are checked and the local archive is consistent, and the public copy is tested and independent recalculation through the real model.
For environment installation and cache preparation, see the package README. The minimum command after preparation is python run.py --cache ./cache --out fresh-smoke --smoke, will execute the real 50 model scores and produce pairwise results, matching indicators, grouping and interval. Remove the test run parameters to execute a complete 100 development queries. SOURCE_MAP.md Corresponding formulas, official functions, fixed submissions, licenses and local changes; independent audit entry does not depend on model weights. The archive contains all the necessary self-write modules, and the external large resources are downloaded through a fixed identity, and do not depend on the author's machine path.
Summary
This group of experiments supports continuing to test the fixed fusion, and also gives a clear counter-example: protecting the old ranking will sacrifice some of the new ranking advantages, and the evidence forward has not improved the probe according to intuition. The negative results that can be reviewed help us narrow the scope of interpretation without automatically demonstrating another mechanism. The next most valuable action has been determined: freeze the plan, complete a confirmation evaluation, and accept the optimistic judgment that it may overturn the development stage.
References
Retrieve and actual access date: 2026-10-26. The official source code submission and the document hashing can be seen in the source code map; the background of the paper and method is not replace the running evidence of this article.
- David Wadden et al., 2020, EMNLP: Fact or Fiction: Verifying Scientific Claims.
- Nandan Thakur、Nils Reimers、Andreas Rücklé、Abhishek Srivastava、Iryna Gurevych,2021,arXiv v4: BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models.
- Sentence Transformers: MiniLM-L6-v2 fixed version model card.
- Gordon V. Cormack、Charles L. A. Clarke、Stefan Büttcher,2009,SIGIR: Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods.
- SciPy developers: 1.13.1 Bootstrap pairing and percentile method documentation.
- BEIR Author: Fixed version rearrangement implementation; Sentence Transformers Author: Pinned version of CrossEncoder.
Apps by the author
The following is the author's own product promotion. In addition to scientific research writing, I also independently develop small tools that make daily life more organized. Welcome to understand according to your own needs:
- Daily Expense Journal: Record the daily income and expenditure, arrange the budget and savings goals, and leave a clear record for each money. View on the App Store.
- One Small Action: Choose small actions according to the current time and energy, record the feelings after completion, and leave a little space for daily actions and reviews. View on the App Store.
More technical articles and personal works, welcome to visit Author personal website. These two works are independent of the scientific research experiment of this article and do not represent the recommendation of the author of the paper or research institution.