
Actual Scope: Reuse the real search results generated on the complete 5183 corpus, and run the official fusion and independent audit of 809 development queries; this article does not re-execute model reasoning, and do not use the final confirmation set.
1. Reproduction value: Is the average weaker road still useful?
In 083, the frozen MiniLM dense search average is lower than BM25, but wins on some queries. This gives a natural question: Can the two advantages be combined? BM25 depends on terms matching, dense search encodes the text as a vector, and then compares the similarity of the vector. The two have different error modes, but "different" does not automatically equal "complementary and effective".
Fusion is the only new variable. The corpus, queries, labels, preprocessing, models, and model outputs are fixed. For each relevant document we want to trace which retriever originally found it, where it ranked after fusion, and whether any gain came from the larger candidate pool or the ordering rule. SciFact is used here for literature retrieval. Its retrieval score does not measure the ability to judge scientific facts. SciFact original paper
2. Core thought: change the score to the rank contribution
Countdown ranking Fusion (RRF) does not directly add BM25 to vector similarity, but adds the countdown to each channel and adds the inverse:
Among them, the query is , the document is , the two are used separately express; is the pre-reserved candidate, Counting from the beginning, the contribution that is not listed is zero. Control the speed at which the head ranking advantage is attenuated, not the number of candidates reserved. The main configuration of this article is fixed , and then take the first 100 documents after fusion.
Paper report: Cormack, Clarke and Buettcher proposed and evaluated the RRF in 2009, and the paper uses pilot experiments to determine the constant 60; the advantage of the report comes from its specific experimental set, and cannot be regarded as any retriever Guaranteed of the combination. Original paper
The author's inference: Documents that can be found in both ways can get two contributions, so they integrate preference consensus; this may also lower the correct answer that is only ranked high in the same way. This explanation must go back to the proof-of-candidate record verification, and cannot make up all the failures with the formula alone.

3. Guide to the official implementation: there is an agreement in addition to the scoring
We actually verify and run a fixed commit of Castorini official Pyserini 527c917120ff154f84b09ff2b69dc8946085aae4. Read first reciprocal_rank_fusion, catch up again TrecRun.rescore, finally read TrecRun.merge: Entry cloning input, re-creating by ranking, truncating candidates, and summing according to the query and document number. For specific files and line numbers, see the source map at the end of the article.
The two official modules that are saved as they are running here are not the author's code of the paper, and the entire pyserin search service is not started. The input is six-column record: query number, occupancy column, document number, ranking, original score, and running label; there are 80827 and 80900 rows respectively. A BM25 query is less than 100, so it cannot be assumed that the matrix is always neat "the number of queries is multiplied by 100".
The official default time is in ascending order by the document number. In order to continue the evaluation protocol in the previous article, the adapter layer first takes a complete merge and merges, and then uniformly arranges and truncates in descending order of scores and descending order of string numbers. We clearly ensure that this change is kept and the impact of ascending order is additionally measured. Only the formula and not the same division rule can still be obtained, and different results may still be obtained. official implementation
4. Minimum experiment: freeze the selection rules first, and then run
This time, BEIR's SciFact Train division is used as a development set, and all 809 queries are entered for the evaluation; 300 test queries continue to be sealed. Following the two top 100 rankings and relevance labels generated by the complete corpus search in the previous article, all three inputs are locked byte hash and public submission. The revisions of MiniLM and tokenizer are both 1110a243fdf4706b3f48f1d95db1a4f5529b4d41, the source code uses title plus space plus abstract, maximum length 256, 384-dimensional vectors. There is no immutable revision of the original packet, use the saved SHA-256 identification, see the resource lock for details. BEIR、Model card
Write it down before running PROTOCOL.md: The main indicator is nDCG@10, that is, the top ten related documents are discounted and normalized; the auxiliary indicator is Recall@100, that is, the proportion of relevant documents covered by the top 100, and MRR@10, which is the countdown to the first relevant result ranking. The constant 60 is fixed in advance, and ten and one hundred are only used for development sensitivity analysis; the main configuration is not replaced because this result is better. The seeds are eighty-four, but the fusion itself is a deterministic calculation.
The smallest entrance takes five real development queries, and the ranking left by the complete corpus is still searched. Full operation and preservation of eight sets of output, query-by-query indicators, 136650 candidate contributions, three cases of maximum benefit and loss, as well as stage status, exit code, environment and time-consuming. The input preparation is the test hash, and the reasoning stage is the reuse existing real results, and the fusion and evaluation are actually executed this time; these states cannot be combined as "rerun all models".
5. Controlled Evaluation: Candidate coverage and ranking should be separated
Each of the main experiments has 100 items, and the average is about 168.9 items, and the final output is 100 items. It changes the number of candidate sources and candidates at the same time with the comparison of a single channel of 100 items, and cannot attribute all the improvements to the RRF.
Therefore, join the simple alternate merge: take turns to fetch the documents according to the two rankings, BM25 first, repeat the skip, and finally keep 100 items. Which way to take first is also a freezing rule. It reads the same union as RRF reads, but uses different sorting rules. Plus the control of 50 items per road: deduplication and union can be at most 100 items, and the two rules retain the exact same set of documents, which is more suitable for isolating the top ten ordering differences. This budget control refers to the number of rank items reserved, and does not mean that the upstream retrieval calculation is equal.
| Method | Each input depth | nDCG@10 | Recall@100 |
|---|---|---|---|
| BM25 | 100 | 0.69427 | 0.93506 |
| MiniLM | 100 | 0.66020 | 0.93243 |
| RRF, main configuration | 100+100 | 0.71222 | 0.95035 |
| alternately merge | 100+100 | 0.71311 | 0.94850 |
| RRF, the same set control | 50+50 | 0.71294 | 0.94413 |
| Alternate merge, same set control | 50+50 | 0.71311 | 0.94413 |
The actual verification of this time: RRF main configuration is about 0.01795 higher than that of BM25, but about 0.00089 lower than the alternate merge. The same set control also does not show the top ten sorting advantages of RRF. Therefore, this support "two-way combination can improve average retrieval performance", and does not support "RRF formula is better than simple merge". The difference is very small, and this article does not claim that statistically significant or cross-domain is established.

The picture above plot.py Read the saved summary and query JSON by query, and display all the queries by difference value sorting on the right side, and no visual subset is selected. The development score of the constant tens is 0.72098, which is higher than 60; this observation is reserved as a follow-up hypothesis, and the main configuration cannot be renamed the pre-selected winner.
6. Failure analysis: what rankings have been changed?
Compared with BM25, the main configuration has increased on 162 queries, 546 is flat, and 101 has decreased; the relative alternate merge is 137 wins, 562 draws, and 110 losses. The latter wins more inquiries, but the average score is lower, because the increase and loss are asymmetrical. The number of winners and losses cannot replace the sum of the differences by query.
Query 618 related documents according to the pre-specified extreme value selection rules 6836086 From BM25 25th, dense third, rising to fusion first. Query 317 related documents 4506414 Then from BM25 first and dense 52nd, it will drop to the eleventh of fusion, and fall out of the main indicator window. The related documents of Query 1307 only appear in the BM25, and also drop from the first to the eleventh. They show that consensus preferences come at a price, but it is not yet justify the cause of a linguistic phenomenon.
The contribution table also shows: the query-document pair of the top ten in the fusion, the 7935 pair is jointly recalled by the two channels, and only 155 pairs come from the single channel. Compared with BM25, 44 related pairs entered the top ten, and 23 left. This is the "pair" that is repeatedly counted into different queries, not the number of independent documents. There are also cases where different queries share the same relevant document in extreme value cases; we do not fabricate artificial semantic classification, and do not assume that these queries are completely independent.
The average recall of the complete union is 0.97054, and there are still 22 queries that have no relevant candidates at all; This corresponds to "no candidates can be arranged" and "there are candidates but discharged", which are the two types of upper limits that need to be dismantled in the next reranking experiment.
7. Failure investigation and trusted boundaries
First, check the number, starting point of ranking and deduplication, and then check the same score. Only change the numbering rule back to ascending order, the main configuration nDCG@10 becomes 0.71025, and the indicators of 60 queries have changed; this change is even greater than the mean value gap between RRF and alternate merges. This article retains the descending order agreement, and does not use this to pick points.
Next, check the value and program invariants. Independent auditing uses rational number recalculation and fusion contribution, check 310733 save scores, the maximum error is about ; There are differences between rational numbers and floating-point sorting in the seven configuration-query combinations, but none of the three indicators have changed. Check all contribution records line by line, verify that the eight sets of indicators, the original input has not changed, and copy the same path will not change the original ranking.
The final full operation is completed on macOS ARM64, Python 3.12.14, and CPU, which takes 4.86 seconds, and the process peak is about 377.7 MiB; 0.37 seconds. They do not contain the previous model coding and retrieval costs. A review of the virtual environment interpreter missed the soft link and lacks evaluation dependencies, the failure log is reserved, and the entry in the environment is passed after passing; the environment record when the dependency is lacking has also been fixed.
Also check the identity of the experiment. The cache key contains two hashes of input and annotation, model and tokenizer version, configuration, dependency and all self-written script hashes; in this experiment, there is no prompt and decoding, and the record is a clear null value. Even if the fusion does not load the model, the upstream identity cannot be lost, otherwise the encoding length or model weight will be changed later, and the old ranking may still be mistaken for the same experiment.
For readers who have just come into contact with recurrence, the most practical reading sequence is to first look at the contribution line of a failed query, and then return to the summary table. Don't rely on the total score guessing mechanism: the same recall ratio may come from different document collections, and the same top ten indicators may also cover up a large number of exchanges after ten. Engineering readers should make these verifications a part of the entrance; graduate students need to ask further, what two competitive interpretations are the current evidence.
8. Turn observation into a testable research problem
The first hypothesis is: a single strong hit, another missing query is a systemic weakness of consensus fusion. The next step should be in the development set pre-defined "single-channel front, another unrecall" group, the statistics group is degraded, and a simple protection rule is frozen; the final confirmation set is only used for one comparison. If the degradation is not concentrated in this group, the assumption is weakened.
The second assumption is: reranked gain is mainly limited by candidate coverage rather than fusion constants. The next article maintains the same candidate set, introduces a real cross encoder, and reports the ranking improvement and unrelated candidates' query ratio of recalled related documents, and measured the inference cost at the same time. It cannot be exchanged for more candidates after the promotion is still called pure sorting improvement.
9. Source code and reproduction entrance
Public fixed version: Source directory、README、SOURCE_MAP. Dependencies, configuration, input hash, license, full log, and query results are all in this version, and there is no need to access the author machine.
After installing a fixed dependency in README, the minimum command is python run.py --out results/my_smoke --smoke, it is expected to generate the ranking, indicators, stage status and identity records of five real queries. Completely recreate and remove --smoke, run again python audit.py --out results/my_run. The output path must be the same as the running command. In this case, all 809 development queries were actually completed. The local archives, decompression and public submissions have all been re-run for the full experiment, and the key result files are the same byte.
10. Summary
Hybrid retrieval lets us trace each change to the ranked list. Which candidates were added? Which relevant documents rose? Which fell because the two lists favored other documents in common? Here RRF improved on the single-retriever means but failed to beat simple interleaving. Preserving that negative result alongside the protocol and candidate limit gives the next experiment a firmer starting point than declaring a fusion method the winner.
References
Retrieve and actual access date: 2026-09-27. For fixed source code links and license verification, please refer to SOURCE_MAP.
- Cormack, Clarke, Buettcher. Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods, SIGIR 2009: Paper、publishing information.
- Castorini. Pyserini, this article is fixed: Fusion source code、Ranking Tools.
- Wadden et al. Fact or Fiction: Verifying Scientific Claims, EMNLP 2020: ACL page.
- Thakur et al. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models 2021, this article references arXiv v4: Paper.
- Sentence Transformers. all-MiniLM-L6-v2: official model card.