
- Date: 2026-10-24
- For graduate students, early-career researchers and research engineers
- The experiment covers one hundred previously examined development queries with fifty fixed candidates per query. It includes no training and no evaluation on the three hundred confirmation queries.
Summary: Let the improvement accept the same bill first
If a search improvement is more documented, the model runs out, and the final score is higher, do we verify the method or budget? This article narrows the problem to an experiment that can clearly keep accounts: keep the scoring of candidates and cross-encoders unchanged, and only change the final sorting rule.
In this actual verification, the nDCG@10 of the fixed ranking is 0.702521, the pure cross-code reranking is 0.669133, and the original mixed search is 0.670906. The fusion did not increase the number of neural model scoring, but it is not without cost: relative cross-encoder, 20 inquiries were improved, 66 were flat, and 14 fell. This is a development set result worthy of further testing, not a research success statement on the confirmation set.
Reproducing value: Strong baselines must be able to refute their own explanations
The previous failure analysis found that the reranking sometimes covered the original useful sequence; the evidence shift probe also reminded that the complete view of the evidence does not guarantee an improvement in the ranking. Therefore, this article does not continue to revise the text, but tests a simple repair: keep the original ranking information, and then integrate with the model ranking. New variables are only sorted decisions, and all queries, document content, weights, and candidate collections are all frozen.
A strong baseline here is a directly relevant comparator that already runs reliably. We make no claim to cover every leading system. The original RRF ranking already combines lexical and dense retrieval, making it a substantive competitor. A pure cross-encoder ranking leaves the order entirely to the reranker. The proposed fusion must be compared with both. If it beats the cross-encoder alone but trails the cheaper original ranking, the extra model layer has yet to earn its cost.
SciFact paper introduces the task of scientific proposition verification and evidence positioning; this project uses its public data to evaluate document relevance. [1] Evidence documents that support or refute claims may be relevant, and the search indicators are not scientifically judged Accuracy. The hundreds of development queries that are still used have already participated in the pre-order analysis, and cannot be re-named as unseen test set just because the experimental number is changed.
Core Ideas and Formulas: Change the Frame Scale to Ranking
RRF is the fusion of the countdown ranking, converting the rankings given by different systems into add-on scores. Cormack, Clarke and Büttcher gave this method in the 2009 paper and reported that the constant 60 was kept fixed after the pilot experiment. [2] The existing collection results of the paper cannot be guaranteed to be valid; this time, only the formula is borrowed, and all its systems or training experiments will not be repeated.
On query fixed candidate documents , this fusion score is:
is the ranking in the RRF sorting, It is the ranking in the CE sorting, all from the beginning, and the values are taken in the same fifty document collection. CE is the abbreviation of the cross encoder, it reads the query and document together, and outputs the scalars used for sorting. The fusion here is "The Existing Mixed Search Rankings" and "Rear Ranking", not directly mixing BM25 with dense vector scores again.
The weight of the two items is added together, and the weight is not trained, and there is no search constant in this article. In this way, the problem that the original search score is different from the CE output scale is avoided, but the score interval is lost: the two rankings are only one by one, and it is impossible to tell whether the fusion device model is hesitant or determined. It is a compromise with a clear preference, not an automatic error corrector.
An intuitive example is: a document ranks tenth in both tables, often better than a document that ranks first and the other table is 50th. The integration preferences are jointly recognized, so it can suppress the radical improvement of a single system, and it may also pull back new discoveries that are truly valuable. This example only explains the behavior of the formula, not the experimental sample selected this time. The research question should therefore be written as "whether this compromise is cost-effective in a fixed range", rather than "whether the fusion is always smarter than a single model".
Sorting in descending order of fusion scores, and the exact same scores are descended in descending order of the string document number. This detail also belongs to the protocol: the inverse summation will generate a juxtaposition. If different scripts use the input order at will, the query result may change. This time, the operation sequence is defined in terms of actual floating-point summation and operation, and the values are independently reviewed with rational numbers, so as not to quietly change the juxtaposition rules.
Guide to the official implementation: from candidate number to output
Read the BEIR pinned commit first Rerank.rerank: It selects candidates, splicing titles and abstracts, and calls the model, and then puts each score back into the corresponding document number. [3] This document is retained in this article, and the external orchestration code is reused in the fixed public version of the previous experiment, to avoid rewriting the input pipeline at the same time for new ideas.
Read the joint tokenization of Sentence Transformers and CrossEncoder.predict. [4] The freeze model is cross-encoder/ms-marco-MiniLM-L6-v2, the model and tokenizer are revised to 233902d25c440f23af6f7d6e94d2946bac0bee0a. The model card shows that it is trained on the MS MARCO paragraph sorting task; this does not constitute a guarantee of the effect of the field of scientific literature. [5] The original scalar output is used this time, and it is not interpreted as relevant probability.
last read blend_report and Independent Auditor: The former only reads the output of this model and constructs two ranking tables, and the latter reconstructs the order, damage and budget from CSV. The source code map lists the official function line number, pinned commit, license and local correspondence; the official file has been re-downloaded and checked, and the installed CrossEncoder source code is completely consistent after the normalization of the line break.
Minimum experiment: what to freeze, what to add
The corpus is still 5183 documents, using the original 100 development queries and its top 50 candidates for RRF. The input is a title, a space and a summary; the maximum length of the joint length is 512 tokens, including queries and special symbols. Token is a sub-word unit of the model processing, and this upper limit cannot be understood as an abstract of 512 words.
The environment is Python 3.12.14, PyTorch 2.6.0, Sentence Transformers 3.4.1, CPU single precision, four threads, batch 16, and random seeds 108. First run five inquiries and multiply ten candidates for real test runs, and then expand to 5,000 pairs according to the predetermined rules. There is no training, candidate expansion, text rewriting or new annotation, and the fusion parameters are not adjusted with the results.
Formal scoring is performed only once, CE and fusion share these outputs. Enter the actual record of 1,761,679 unfilled tokens; both are the same, and the fusion extra model is scored into zeros. Pure RRF does not need this CE forward, so "waiting budget" only refers to the comparison between CE and the fusion neural scoring, and cannot claim that the total cost of the three methods is consistent. The historical calculation cost of BM25, dense retrieval and initial RRF has not been retested in this article.
The formal CE reasoning time is 129.90 seconds, the basic operation includes import, loading and evaluation for a total of 134.72 seconds, and the peak process memory is 855687168 bytes, about 0.80 GiB. The integrated ranking table structure, summation, sorting and trajectory generation are also calculated for about 2.96 milliseconds, excluding indicator calculation and writing files; a single micrometer cannot extrapolate service throughput. The shape check pair performs four additional query document pairs, independent of the official 5,000 reconciliation book.

The bar in the figure is the evaluation icon and does not represent the measured value. The three evaluation paths share candidates, and the confirmation set is disconnected from this process. The actual input, final representation and output shape are respectively 、、, corresponding to batch, joint length, hidden dimension and single score; the model maintains an evaluation mode.
Actual verification: the average value is increased, and the local loss is still there
nDCG@10 measures how well relevant documents are ranked within the top ten, discounting by rank and normalizing against an ideal ranking. Recall@50 is the fraction of relevant documents covered by the top fifty candidates, averaged here across queries with equal weight. “No top-ten hit” counts queries with no relevant document anywhere in the top ten. That count measures something different from recall.
| Method | nDCG@10 | Recall@50 | Inquiry in the top ten |
|---|---|---|---|
| Raw RRF | 0.670906 | 0.905000 | 21 |
| Pure ce | 0.669133 | 0.905000 | 17 |
| Fixed Rank Fusion | 0.702521 | 0.905000 | 14 |
The average increase in fusion relative to CE is 0.033388, and the increase of 0.031614 compared to RRF. The latter comparison is 18, 72, and 10. Candidate coverage remains unchanged, so these changes come from sorting rather than finding more documents; but the missing evidence in the candidates cannot be fused out of thin air.
Putting the results back to the preorder failure, you can see a more specific exchange. In the original 18 articles of CE's regression relative to RRF, seventeen articles have improved after fusion, but only nine have returned to the RRF level or better. Among the 20 articles that originally benefited from CE, the benefits of thirteen articles were weakened; in the original inquiry that did not regress, there was also a new loss relative to the RRF. Only showing that "repair seventeen articles" will hide a considerable part of the cost.
Consider the extreme per-query differences between fusion and CE. For query 309 the relevant document rose from sixth under CE to first under fusion, increasing nDCG by 0.643793. For query 242 the first relevant document fell from first to seventh, decreasing nDCG by 0.408765. Query 242 has several relevant documents, so the rank of its first relevant document does not determine its nDCG. These are illustrative cases selected after the experiment.
Evaluation protocol and troubleshooting
The main comparison is fixed in advance to the fusion-by-query fusion subtraction mean, and the fusion minus RRF is reported at the same time. The 0.01 threshold in the research registration is left for the final confirmation evaluation; the development average of this article exceeds it, and the hypothesis cannot be announced in advance. The unit of comparison here is a query, not a pair of 5,000 query documents. Treating each candidate as an independent success will break up the relevant order constraints in the same query, and exaggerate the amount of evidence. The same mean can also be made up of a few major improvements or a lot of minor improvements, so the full pairing table is just as important as the average. Evidence documents may be shared between queries. This time, 100 pieces are not regarded as independent samples to calculate the significance. In the future, it is necessary to clarify the grouping and uncertainty estimation schemes.
The first run failed when importing PyTorch, the old temporary environment lacks dependencies, and has not yet entered the reasoning. After restoring the full fixed version environment, the real test run and the official run pass; the failure log is reserved. If the web page of the model card fails to read, it will continue to verify through the anonymous fixed version download, and the source is not used to complete the guess.
An independent check reconstructed the ranks and fusion scores for all five thousand pairs, the per-query metrics, the means, the win/loss counts, and the budget. The raw cross-encoder scores matched the earlier records exactly. Seven corruption tests caught altered scores, missing candidates, altered means, extra scoring, misuse of the confirmation set, altered rank histories, and incorrect category counts. The check establishes that the results can be recomputed. Selection bias in the historical development data and competing explanations for the gain remain open issues.
Research questions and summary
The partial observation of this support is: under the same candidate and CE score, retaining the original ranking can improve the average result of this batch of development queries, and at the same time sacrifice some CE's unique benefits. The author infers that the two sorting information are complementary, but the mean cannot assert that the model is mismatched, let alone a mature fusion formula to be named a new algorithm.
The next step is to split the independent contribution and interaction of the contextual processing and ranking fusion, instead of continuing to adjust the constant until the result is more beautiful. The 300 confirmed inquiries that were originally reserved have not been evaluated, and will not be used until the plan and report are frozen. A credible improvement needs to explain at the same time how much has been improved, who has hurt, what budget has been used, and which questions have not been answered this time.
Code and instructions for reproducing the results
This complete supporting source code has been publicly released, a total of 74 files: Source tree at the pinned revision、README and run instructions、SOURCE_MAP corresponds to the official implementation. The submission has been anonymously downloaded, all files are consistent with the local archive item by item, and the real model test run is re-executed from the downloaded version.
After installing a fixed dependency according to README and performing data preparation, the minimum real command is:
HF_HUB_OFFLINE=1 TOKENIZERS_PARALLELISM=false python run.py --cache ./cache --out replay-smoke --smoke
It is expected to output 50 pairs of scores and three-plan query indicators, fusion trajectory and budget ledger; remove the test run parameters to run 100 inquiries and multiply 50 candidates. Then use python audit.py --out replay-smoke Independent recalculation. This time, the test run and the formal scope are actually implemented. The supporting source code includes all self-writing modules; large resources are obtained according to fixed revisions and file hashes, and do not depend on the author's machine path.
References
Search and fixed source code verification date: 2026-10-24. The paper report, this actual measurement and the author's inference have been distinguished in the corresponding paragraph.
- David Wadden et al., 2020, Fact or Fiction: Verifying Scientific Claims,EMNLP, Conference page.
- G. V. Cormack、C. L. A. Clarke、Stefan Büttcher,2009, Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods,SIGIR, The author's paper.
- BEIR author, Fixed Submitted Rerank Implementation.
- Sentence Transformers maintainers, CrossEncoder implementation of fixed commit.
- Sentence Transformers, This weight version model card.
Apps by the author
The following is the author's own product promotion. In addition to writing a technical blog, I am also doing some small tools that make everyday life more organized. Welcome to the two apps I independently developed:
- Daily Expense Journal: Track your income and spending, plan a budget and set savings goals. View on the App Store.
- One Small Action: Pick a small action that fits the time and energy you have today, then record how it felt and what you noticed. View on the App Store.
Find more articles and projects on my website: zffjzz.top. I’d love to hear how the apps work for you and what I could improve.