Cover illustration

  • Date: 2026-10-27
  • For graduate students, early-career researchers and research engineers
  • Actual scope: complete 5183 literature, 300 confirmation inquiries, 50 candidates each; freeze retrieval and reranking, no training.

Summary: Let the conclusion be tested with a real test

An experimental package becomes a short paper, not a more formal wording. The key change is that the reader can follow a clear question, when to trace when, what inputs are produced, and what evidence will make the authors take back their conclusions. The problem of this project is very small: when the cross-encoder is reordered, the information of the original search ranking is retained. Can the search quality be improved under the same candidate and model scoring budget?

Previously, the fusion of equal rights rankings on 100 observed development queries increased by about 0.03339 on average nDCG@10. However, the development results have participated in the formation of research intuition, and the same batch of data can no longer be used to prove that intuition is reliable. In this article, according to the pre-registered plan, the confirmation evaluation of 300 reserved inquiries is completed for the first time. The average gain of the fusion relative to the pure reranking is 0.01754, reaching the preset 0.01, but the pairing grouping interval spans zero, and the fusion does not exceed the cheaper original RRF. This report delivers verifiable boundaries of evidence and does not pre-commit new methods or papers.

1. Reproducing value: writing research questions to fail

Beginners can understand the search as "finding relevant literature for a scientific proposition". Word search BM25 looks at the matching of words, dense search maps the text into vectors, and the cross-encoder inputs the query and candidate literature into the model and outputs a fraction. They only solve the retrieval link; finding the relevant paper does not mean that the judgment is supported or refuted.

Wadden et al. SciFact's original paper research scientific proposition verification, including evidence literature, labels and reasons; BEIR et al. This is Task background of the paper report. This time, the literature correlation mapping of BEIR is used, and the complete fact-verification system is not reproduced. The development division of the original SciFact is called test in BEIR: different names do not mean that the data is added, and it does not mean that it has never been seen by any model or other researcher.

The main assumption we register is: under the same 50 candidates and the same model score, the equal weight fusion is at least 0.01 higher than the average nDCG@10 of the pure cross encoder, and the difference between the original mixed retrieval is reported. The 0.01 here is a preset practical gain threshold, not a significant boundary that is picked out after seeing the results. A complete inspection should allow the average value to be substandard, the interval span is zero, and even the fusion is not as good as the cheaper baseline.

2. Relevant work and existing evidence: do not delete failures from the story

Cormack, Clarke, and Büttcher proposed the Countdown Ranking in 2009, referred to as RRF: merging multiple systems by the countdown contribution of the ranking. This project only uses this existing idea to the combination of "original search ranking and rear ranking", and does not claim to invent the fusion method. The cross encoder comes from the public MS MARCO sorting model; the model card describes its training task, which does not mean that it must be better in scientific literature.

Two explanations emerged during development. Truncation might remove relevant text and mislead the reranker, or reranking might discard useful information in the original ranking. Of the 18 queries that did worse after reranking, 17 had no relevant candidate truncated. That weakens the first explanation without establishing the second. Moving the evidence produced no consistent improvement across four reviewed probes. In the four-way test of context placement and rank fusion, moving the context earlier had a negative average effect both with and without fusion.

These are Developed evidence of previous actual verification, not a confirmation result. Evidence shifting uses the correlation and reason information to construct, which is a diagnostic probe after knowing the answer, and cannot be a result of a deployable method. The final candidate only retains the original ranking fusion that does not require tags to participate in reasoning. The original negative results, audit records and fixed versions of each article are kept in the experimental ledger, and readers do not have to believe a successful narrative that has been rearranged too smoothly.

3. Core ideas and formulas: only change the sorting rules

Freeze evaluation and research report process

First run BM25 and dense search on the same full document library, each retain the first 100, and form a prior ranking through the RRF with a fixed constant of 60, and then select the first 50 candidates. The cross encoder scores these 50 original titles and abstracts. The second use of fusion:

s(q,d)=160+rR(q,d)+160+rC(q,d).s(q,d)=\frac{1}{60+r_R(q,d)}+\frac{1}{60+r_C(q,d)}.

Among qq is a query, dd is the literature of the same candidate set; rRr_R is the prior RRF ranking, rCr_C It is the cross encoder ranking, both starting from 1. The two weights are fixed to 1; if the scores are the same, the character string literature number is in descending order. The formula only reads the ranking, does not call the model, and does not add the original score of the different model directly.

The main comparison is:

Δ^=1N∑i=1N(Miblend−MiCE).\widehat{\Delta}=\frac{1}{N}\sum_{i=1}^{N}\left(M_i^{\mathrm{blend}}-M_i^{\mathrm{CE}}\right).

N=300N=300 is to confirm the number of queries, MiM_i is the ii -th query’s nDCG@10. The metric discounts relevant documents by rank in the top ten, then normalizes by the ideal ranking for that query. Higher is better. We compute the difference for each query before averaging, so both methods are compared on the same questions. The candidates are fixed, so Recall@50 should be identical. It serves as a protocol check.

4. Evaluation protocol: it is not just the model name that is frozen

Compared with the previous article, the new variables in this article are data division: from 100 development queries that have been repeatedly observed to all 300 reserved queries. Models, text processing, candidate budgets, fusion constants, and statistical rules remain frozen. Using all 5183 literatures, the corpus is easier to make with easier results; the sample is not screened and confirmed according to the query length, positive and negative results, or candidates.

The fixed revision of the dense model all-MiniLM-L6-v2 is 1110a243fdf4706b3f48f1d95db1a4f5529b4d41, up to 256 tokens, output 384-dimensional vectors. Rearrangement model ms-marco-MiniLM-L6-v2 is fixed to 233902d25c440f23af6f7d6e94d2946bac0bee0a, the maximum query and literature are 512 tokens, using the original scalar score. The tokenizers of the two are consistent with their own model versions; there is no part of the data available for Git revision to download the address and file hash identification.

Inquire about the shared evidence literature, which cannot be easily regarded as 300 independent coins toss. Before we see any confirmation score, we get 247 groups according to the shared positive literature connection and take the connected component; including the relevant literature that did not enter the candidate set. There are 10,000 samples by the group, and the seeds are fixed to 111. Each time the sum of the query differences in the drawn group is divided by the total number of queries, the weighted average of the query is still estimated, rather than the weight of the group of different sizes.

The report uses the 95% range of the percentile method and gives additional query-by-query sampling sensitivity. Grouping can only be partially dependent on proxy, and topic association and training pollution may still exist, and intervals cannot be used as a guarantee. Whether the practical threshold is reached, whether the interval is covered with zero, and whether the interval is covered by 0.01 are three questions answered separately. Confirmation results are no longer used to modify weights or select new models.

5. Guide to the official implementation and minimum experiment

Reading source code can start from data flow. First look at BEIR's BM25Search.search and ElasticSearch.lexical_multisearch: Check the literature number of the English analyzer, title and abstract fields, single shard and batch returned. Look again DenseRetrievalExactSearch.search: Confirm the full document library coding, cosine similarity and candidate depth, instead of just looking at whether the model is loaded successfully.

Then read Pyserini's reciprocal_rank_fusion、TrecRun.rescore and merge, track how each countdown contribution is added by the literature number. Finally, along BEIR Rerank.rerank Pairing tokenization to Sentence Transformers and CrossEncoder.predict: Check the title and abstract splicing, joint truncation, and output score how to return to the original query and literature. SOURCE_MAP.md Give the verified pinned commit, the function line number, the license and the local corresponding path, and distinguish the original upstream file and the independent teaching implementation.

The resource test run only uses five observed development queries. The new environment initially lacks retrieval dependencies, and then encounters PyArrow that is not compatible with NumPy versions; these failures occur before confirmation and logging. In the compatibility check, the BM25 score is exactly the same as the old record; the maximum error of the dense score is about 3.3×10−73.3\times10^{-7}, the candidate collection and sorting are unchanged. The final full library search is about 51.58 seconds, the cross encoder is officially scored about 391.04 seconds, and 5323656 unfilled tokens are reserved; the peak resident memory of this process is about 1.08 GB. Single batch shape check is input [2,512][2,512], the last floor representation [2,512,384][2,512,384], output [2,1][2,1], corresponding to two pairs of text, sequence length, hidden dimension and single sorting score; these four auxiliary forward input pairs are not included in the official 15000 pairs. The model is not trained; the download time and the Elasticsearch service peak memory are not measured and cannot be mixed into the measured cost.

6. Main result: give a complete comparison first, and then explain the difference

Confirmation set method nDCG@10 Recall@50 Number of queries for the top ten unrelated literatures
Raw RRF 0.711338 0.940333 40
Pure Cross Encoder 0.692765 0.940333 49
Equivalent rankings merge 0.710305 0.940333 38

What we tested: fusion minus reranking alone averaged 0.017540, with 65 wins, 197 ties and 38 losses across queries. The grouped percentile interval was [-0.002345, 0.038275], which includes both zero and 0.01. The query-resampling sensitivity interval, [-0.001477, 0.036632], also includes zero. The observed mean met the preregistered threshold. The evidence does not establish a stable positive effect or a population gain of at least 0.01.

The mean value of the fusion minus original RRF is -0.001033, the win/d draw/loss is 46/213/41, and the grouping interval is [-0.019943, 0.016832]. The pure re-reduction of RRF is -0.018573, and the interval is [-0.052325, 0.013459]. The complete comparison shows that the fusion repaired a portion of the reranking loss, but did not exceed the cheapest reference method on this batch of confirmation data. The development gain contracts to the confirmation gain, and it cannot be attributed to the selection deviation alone, because the query composition changes at the same time.

The cost comparison applies to a specific pair of methods. Pure reranking and fusion use the same cross-encoder outputs, so fusion adds only arithmetic on the ranks. The original RRF ranking needs no cross-encoder pass, which means total costs differ across the three methods. Read the score differences alongside the model call counts. Sharing a candidate set does not equalize compute.

7. Failure analysis and investigation: the report cannot have only one mean

There are 17 queries in the candidate stage, and there is no relevant literature at all, and the three rerankings cannot be saved; even if the ideal sorting of the relevant tags is used, the upper limit is only about 0.941169. It is the upper limit of the candidate diagnosis, not the actual method score.

The three maximum fusion regression queries automatically listed according to the predetermined rules are 690, 1368, 1024. Taking 690 as an example, the first related literature in RRF is 33 in RRF and 1st in pure reranking, and it has dropped to 9th after fusion, and the nDCG@10 has dropped from 1 to about 0.30103. This is a direct counter-example of "reserving the priors always reduces losses": the prior also pulls down a correct reranking. It only supports the description at the ranking level and does not constitute an unaudited semantic misattribution.

The investigation sequence should first deal with the protocol, and then explain the semantics: check the missing candidates, mismatch between the query and the literature number, the truncation length, the parallel score processing and cache identities; then check whether the difference value is indeed generated by the same query and pairing. The independent script reconstructs the ranking from the original score, the correct correlation map and all the indicators, and uses another set of graph traversal and quantile to achieve the recalculation and sampling results. After intentional damage to scores, candidates, budgets or groups, the audit must refuse to pass; it is not enough to make the original script self-reported successfully.

8. Research problems, limitations and summary

A reviewable short report can make a limited contribution only: provide a comparison that can be repeatedly checked within a fixed task, model and budget. There is no new training, no causal relationship between the two wrong mechanisms, and no promotion of the four probes into a full data law. To develop and confirm the shared literature library, the inquiry is not equal to the literature or the subject reserve; one scientific literature evaluation is not equal to the validity of cross-fields.

What is worth testing in the future is: Does the fusion benefit mainly come from specific prior and reranking differences, or only appear in some candidate quality ranges? These layering rules must be defined on the newly developed material and then verified with the new validation material. The current 300 queries have been consumed, and cannot be re-called unseen data after changing the description. Author’s interpretation these directions can be proposed, What we tested Only frozen comparisons in tables and their limitations are supported.

From the experimental package to the short paper, what really needs to be tightened is the distance between claims and evidence. Problems, protocols, results, failures and running entrances are put together, and even if a preset threshold is not reached, work can leave a starting point for the next study to test.

Code and instructions for reproducing the results

this article has been published Source tree at the pinned commit、README and full operating instructions、SOURCE_MAP official source map Together to form a recurring entrance.

After installing the lock dependency and pressing README to get the hash check cache, a minimum real run command is:

python run.py --cache ./cache --out fresh-smoke --smoke

It performs five development queries, each ten candidates, and generates scores, rankings, summary, shape, time-consuming and audit inputs. The official scope of this article is the 15,000 pairs of all 300 confirmed inquiries, as well as the previous full library dual-channel search; the minimum test run is not a complete score. See README for complete operation, independent recalculation, rebuild retrieval and version acquisition steps. All necessary self-writing modules are provided with the source code; the weight of the model and the complete corpus are separately obtained by a fixed identity. The public fixed versions, archive lists, and validation records jointly determine the objects reproduced by the reader.

References

Retrieval and access date: 2026-10-27. The conclusion of the paper, model description and this measurement are respectively identified, and the promotion link does not belong to the source of scientific research.

  1. David Wadden、Shanchuan Lin、Kyle Lo、Lucy Lu Wang、Madeleine van Zuylen、Arman Cohan、Hannaneh Hajishirzi,2020: Fact or Fiction: Verifying Scientific Claims, EMNLP, tasks and data sources.
  2. Nandan Thakur、Nils Reimers、Andreas Rücklé、Abhishek Srivastava、Iryna Gurevych,2021: BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models,v4; Fixed version rearrangement implementation.
  3. G. V. Cormack、C. L. A. Clarke、Stefan Büttcher,2009: Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods, SIGIR, the countdown ranking fusion formula.
  4. Sentence Transformers: all-MiniLM-L6-v2 Fixed Model Card、ms-marco-MiniLM-L6-v2 Fixed Model Card, corresponding to the encoding and rearrangement implementation respectively.
  5. SciPy developers: 1.13.1 Bootstrap documentation, the background of the pairing and percentile interval; the grouping algorithm in this paper is implemented independently and does not classify it as the official scheme of SciPy.

Apps by the author

The following is the author's own product promotion. In addition to scientific research writing, I also independently develop some daily gadgets, welcome to understand according to your needs:

  • Daily Expense Journal: Record personal income and expenditure, arrange budget and savings goals, and make daily accounts more organized. View on the App Store.
  • One Small Action: Choose small actions according to the current time and energy, record the feelings after completion, and leave a little clue for daily actions and reviews. View on the App Store.

More technical articles and personal works, welcome to visit Author personal website. These are the introduction of individual works independently of the conclusion of this article, and do not represent the recommendation of the author of the paper or research institution.