Fixed Candidates and reranking Experimental Devices

  • Date: 2026-09-28
  • For graduate students, early-career researchers and research engineers
  • Scope: The existing search results of the complete 5,183 corpus; 100 development queries were drawn, and each was reranked with 50 candidates; it did not represent a complete benchmark.

Abstract: Ask where the document is first, and then ask whether the model is strong

The last article observes how the Countdown Ranking Fusion (RRF) changes the results of terms and vector searches. This article only adds one variable: hand over the frozen candidate to the cross encoder to score again. After the actual operation, the main indicators did not improve, but we obtained a more specific failure map: which queries lose the relevant documents before entering the model, and which queries still have space for sorting.

This diagnostic replication makes the question “Why didn’t the reranker win?” something we can investigate. The results use real public data, a frozen model, and the official inference module. The ideal ordering is a diagnostic reference calculated with relevance labels. Explanations proposed from the comparison still need testing.

1. Reproducing value: split a total into two problems

reranker (reranker) receives the document that Retriever has selected and changes its order. If the relevant papers do not enter the candidate set, no matter how strong the sorting function is, it cannot be replaced out of thin air. If it has entered the candidate and only ranked 13th, the model has the opportunity to push it to the top ten. Two types of situations require different experiments, and mixing in one mean will mislead the direction of improvement.

[Paper report] Nogueira and Cho's "Passage Re-ranking with BERT" combine the query and candidate paragraphs into BERT, and then sort them according to the correlation score; the paper studies the second stage after searching. The original text uses BERT-LARGE and trains. This article uses public minim checkpoint reasoning, and does not claim to reproduce the original paper training or grades. [1]

[This actual verification] Follows the version and division of SciFact and BEIR: 809 BEIR train queries as development resources, and 300 BEIR test queries continue to be reserved. 082 The latter has been checked for the original SciFact development division; this is a confirmation resource that this project is not used for parameter adjustment, and is not an external seal blind test. Retrieval relevance does not equal support, rebuttal, or fact-checking Accuracy . [2][3]

2. Core idea: read a pair of texts together, rather than compare two vectors

The dual encoder calculates the query and the document vector respectively; the cross encoder puts two paragraphs of text into the same Forward Pass, so that the words in the query can interact with the candidate body. The scoring here can be abbreviated as:

sθ(q,d)=fθ(tokens⁡(q,title(d)∥text(d)))∈R.s_\theta(q,d)=f_\theta(\operatorname{tokens}(q,\mathrm{title}(d)\Vert\mathrm{text}(d)))\in\mathbb R.

where, qq is a query, dd is the candidate document, ∥\Vert Indicates splicing with a single space, fθf_\theta Includes frozen encoders and classification heads. The output is a real number logit, that is, the model score of the model that is not normalized for probability; the value is larger in the front, and it is not regarded as a "relevant probability". The fixed configuration of this model is activated using Identity, and the original score is retained directly. [4]

This shape check actually gets the input number tensor [2,512], the last layer hidden state [2,512,384], classification header output [2,1], the official prediction interface will be re-opened [2]. The maximum difference between the direct forward logit and the interface output of the two samples is zero. The two here represent two pairs of query documents, and five hundred and twelve is the upper limit of the joint length with special symbols; it does not allow five hundred and twelve tokens to each query and abstract respectively.

reranking protocol for freezing candidates isolated from labels

3. Guide to the official implementation: the most important thing to check is the boundary

Read BEIR's first from the source map Rerank.rerank: It traverses the candidates, constructs the query and the "Title Space Summary" text pair, and saves the query and document number at the same time; after calling the prediction interface, put the score back to the corresponding number. If this step is misplaced, the indicator may be output normally, but the experiment has failed. [5]

Then read Sentence Transformers' smart_batching_collate_text_only, confirm to remove the head and tail blanks, dynamic filling and longest_first cut off; read again predict, check off training mode, off gradient, activation function and batch output sequence. The tensor check of the text corresponds to this actual link, not another similar network to replace the official implementation. [6]

The source code package retains the BEIR rearrange module and Apache 2.0 license as it is, and has not changed its scoring or number mapping. Model loading uses the installed official dependencies; local new additions are protocol, hash verification, index recalculation and error diagnosis. The source code map gives the actual verification submission, file and line number, and distinguishes the copy code from the independent teaching implementation.

4. Minimum experiment: only make the sorting change

Registration rules before running: 809 development query numbers are pressed SHA256("85:" + query_id) Sort, take the first 100; each retain the first 50 candidates obtained by the previous fixed RRF parameter 60. Samples are not selected based on related labels, reranked scores or failure cases. The original search covers the complete corpus, and this reduction is the number of queries and the candidate budget, which cannot be directly compared with the full average of the previous article.

The freeze model is cross-encoder/ms-marco-MiniLM-L6-v2, the weight and tokenizer revision are both 233902d25c440f23af6f7d6e94d2946bac0bee0a. There is no independent Git revision for data archives, and the validated download address, archive hash and file hash are used as the version identity. CPU, four threads, batch 16, joint length 512, single precision, random seed 85; no prompt, training or decoding process. [4]

First perform five inquiries and test runs for each ten candidates. Fifty pairs of real text reasoning took 1.30 seconds, the peak process memory was about 0.90 GB, and the linear estimate was lower than the pre-registered 900-second budget, and only entered the 5,000 pair of main experiments. The test run score is not used to change the model or change the protocol. The formal reranking time is 135.33 seconds, the total operation time with import, verification, loading and auditing is 141.98 seconds, and the peak process memory is about 1.11 GB; these are native measurements and do not include previous retrieval and network downloads.

Caches the identity record model, tokenizer, data, text rules, candidates, configuration and source code hashes. Download, preprocessing, model loading, reasoning, and evaluation are left respectively; the original ranking, new ranking, logit, related tags and truncation are saved one by one. The model weights are left in the explicit cache, and the source archive does not carry environments or weights.

5. Evaluation protocol: What is the upper limit of the ideal candidate sequence?

The main indicator nDCG@10 is "normalized loss cumulative gain": the higher the relevant documents are arranged, the greater the contribution, and then divide by the ideal score of the relevant documents marked. The denominator cannot only use the recalled document, otherwise the missing recall will be covered up. Auxiliary report top ten recall, top 50 recall, and the mean value of the first related result countdown MRR@10.

Make CqC_q for the frozen candidate set, RqR_q To be labeled related sets, the optimal nDCG in the candidate is:

Uq=∑i=1min⁡(10,∣Cq∩Rq∣)1/log⁡2(i+1)∑i=1min⁡(10,∣Rq∣)1/log⁡2(i+1).U_q=\frac{\sum_{i=1}^{\min(10,|C_q\cap R_q|)}1/\log_2(i+1)}{\sum_{i=1}^{\min(10,|R_q|)}1/\log_2(i+1)}.

Here, the binary related label determines which documents can contribute one point; all the documents that have been recalled can be reached to the top to reach this Oracle, that is, know the ideal reference of the tag. It is computed using answers and cannot be used as a deployable system, nor does it participate in model input or scenario selection.

If the model actually gets MqM_q, then 1−Mq=(1−Uq)+(Uq−Mq)1-M_q=(1-U_q)+(U_q-M_q). The right side represents the gap between the indicator gap caused by the candidate set and the upper limit of the current sorting relative candidate. It is an arithmetic decomposition under fixed annotation, fixed candidates and fixed indicators, and does not prove that the latter one is eliminated by the replacement model. Data bias, truncation and label omission may also affect it.

6. Actual results: more recalls, not necessarily better

The same hundred inquiries nDCG@10 Recall@10 Recall@50 MRR@10
Original RRF sequence 0.6709 0.7775 0.9050 0.6374
Cross encoder rearrangement 0.6691 0.8175 0.9050 0.6267
Ideal order in candidates 0.9061 0.9050 0.9050 0.9100

The difference between the main indicators is −0.001774, and the query is 20 wins, 62 squares and eighteen losses. The top ten Recalls are increased, but the average value of the main indicators and the first relevant results is slightly lowered, indicating that "entering the top ten" and "ranking to the top" are different goals. Confidence interval is not estimated, and the model cannot be claimed to be significantly worse according to this, let alone general SciFact or other Reranker .

The difference between the mean and query-by-queries generated by saving the result

Reranking kept the same fifty candidates for each query, so recall at fifty had to stay fixed. The independent audit verified this query by query. Nine queries had no relevant candidate. Eight had a relevant candidate but none in the top ten after reranking. Eighty-three had at least one top-ten hit. For one query, only some relevant documents were retrieved, illustrating why a hit does not establish full coverage. The mean candidate gap of 0.09387 and the mean ranking gap of 0.23700 together account for the remaining error.

7. Failure investigation: check the records first, and then put forward explanations

Automatically select the three minimum and maximum differences, and do not claim manual annotation. The first relevant document of Query 817 was reduced from the second place to the 15th place, which is a clear intra-sortment failure; the query 764 was upgraded from the 13th place to the second place. Inquiry 309 fell from the first place to the sixth place, and the top ten still hit, but lost a lot of losses. This explains why there are slightly more queries, and the total mean can still decline.

The investigation sequence should start with the number corresponding, the candidate set and the evaluation denominator, and then check the token and the content. 569 pairs in the 5,000 pairs were truncated, the original total was 1,834,958 tokens, and the truncation was reserved 1,761,679 This is the cumulative amount of text pairs, the same query is calculated repeatedly, and does not contain additional calculations that are dynamically supplemented. It cannot be directly compared to the amount of token coded by the dual encoder one-time document encoding.

All indicators are generated by the standard evaluator, and then the independent code is recalculated; at the same time, check the score and ranking of each pair, the ideal upper limit closed formula and error decomposition. Unlabeled documents are irrelevant according to the evaluation protocol, which does not mean that their scientific content is really irrelevant. Just looking at the sample of high-scoring errors can infer that the model "does not understand science", and the evidence is still insufficient.

It is also necessary to distinguish between zero recall and partial missing: when a query has multiple related documents, only one of the related documents may be found, but the ideal candidate score is still lower than the ideal value under the complete annotation. In this article, all relevant labels are used to calculate the denominator directly, and this difference is preserved. This upper limit depends on whether the existing annotations are complete; in the future, supplementary judgments may change the relevant sets and error decomposition, so it is the checkable boundary under the current protocol, not the final judgment of the real scientific correlation.

8. Testable problems: to retain the competition explanation for failures

[Author's Inference] The first explanation is to truncate the fragments that determine the correlation; the second explanation is that the ordering preferences obtained by MS MARCO training do not match the scientific statement retrieval. The current operation contains both possibilities, and neither can be written as a confirmed cause.

In the next article, this batch of inquiries, candidates, models and evaluations can be fixed, and the pre-registration title is spliced and the length is compared. If the recovery of the truncated document is significantly different from the untruncated group after adding the visible text, the support for the first explanation is added. Additional tokens and time-consuming are also reported; only the input is extended and improved, and the impact of calculating the budget cannot be ruled out. If recovery does not appear, this negative result should also be retained. All options are still limited to development resources, and the confirmation set continues to be isolated.

Code and instructions for reproducing the results

The source code has been public and completed anonymous pull verification: Fixed version directory、README and run instructions、SOURCE_MAP source map、Operation entry. All 51 remote files matched their local archive counterparts by hash. After unpacking the archive, we reran the entire experiment and reproduced the key results byte for byte.

After installing dependencies and downloading fixed resources by README, the minimum command is python run.py --cache ./cache --out results/smoke --smoke, it is expected to generate 50 pairs of scores, query indicators, shapes and staged states. This time, the test run and the complete 100 inquiries and 5,000 text pair experiments were actually carried out; omitted --smoke This range can be re-run. The source code provides data download, offline auditing and drawing portals, without the implied path on the author's machine.

Summary

This reranking did not improve the main indicators, but has already answered a more specific question: which errors are not recoverable under fixed candidates and which documents are reranked by the model. The next study should be revolved around variables that can be frozen and changed, and interpreted with a recalculated failure record constraint, instead of pre-reserving a successful conclusion for a method.

References

Retrieve and actual access date: 2026-09-28. When the browsing cache of the fixed source code is invalid, it has been downloaded and verified through the official original address; the dynamic list value is not referenced.

  1. Rodrigo Nogueira, Kyunghyun Cho, 2019 first edition, 2020 V5: Passage Re-ranking with BERT.
  2. David Wadden et al., EMNLP 2020: Fact or Fiction: Verifying Scientific Claims.
  3. Nandan Thakur et al., 2021: BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models.
  4. Sentence Transformers: MiniLM-L6-v2 Fixed Model Card and Configuration.
  5. BEIR: Fixed version Rerank implementation.
  6. Sentence Transformers: Pinned version of CrossEncoder.