Fixed configuration migration experiment cover

  • Date: 2026-09-30
  • For graduate students, early-career researchers and research engineers
  • Actual range: NFCorpus's BEIR export corpus 3633 documents, 323 test queries; source domain reuse SciFact 809 development query. Migrate BM25, miniM and RRF only, without reranking or training.

Summary: Move the agreement first, then look at the winner or loser

In the previous article, it was found that the title and truncated length will change the reranking result, but there is no condition exceeding the fixed mixed retrieval baseline. The newly added question in this article is: The relative relationship of the existing method, can it be maintained with a set of public retrieval data? To this end, we take the target data as a variable, freeze the model, tokenizer, field processing, candidates and fusion parameters, and register the protocol before downloading the target set.

This actual verification shows that the dense search is lower than the nDCG@10 of BM25 in both sets, and the RRF is higher than BM25. However, the dense Recall@100 of the dense search on the target set is higher, and the conclusion cannot be pressed into a sentence "Vector search is invalid". What is more worthy of reproduction is a project detail: the upstream program misses the 15 untouched queries; if the zero score is not returned, the denominator of the mean will change.

Reproducing value: what exactly did this time migrate

BEIR's research value lies in generalization with different retrieval task inspection methods, rather than letting a data set represent all retrieval scenarios. BEIR paper This evaluation framework is proposed; this article only takes two of the sets, far less than the complete cross-domain benchmark.

The input to SciFact is a scientific statement, and this series has processed it into a literature retrieval task. NFCorpus comes from popular health and nutrition content, and there may be a distance between the query language and the professional paper language. Original paper This vocabulary difference is discussed. The original full-text data is not the same scale as the BEIR export version; this time the latter operation cannot be directly compared with the original paper numbers.

There are also boundaries here: both sides contain scientific and medical content, so this is a transfer of collection and query tasks, which cannot be called sufficient remote field verification. The size of the corpus, query expression, the number of related documents and the labeling system change at the same time; the freezing calculating method only controls the variables on the algorithm side, and cannot attribute the changes of all fractions to the "field".

Core idea: Make configuration a verifiable constant

The freezing is not "I remember not changing the parameters", but to include the configuration, model files, data files and executing codes into the identity record. Any change should generate new cache keys. When the target set has not yet been evaluated, we have registered two main comparisons in the protocol: whether the direction of the dense minus BM25 and the RRF minus BM25 is maintained.

BM25 uses Elasticsearch 7.17.9's English analyzer, single-slicing, default word frequency saturation and length normalization parameters. The title and summary are reserved as separate fields, using best_fields, and the other field is merged with a coefficient of 0.5; its semantics comes from official documentation.

Is MiniLM still using "Embedding is really better than BM25? "The fixed weight and tokenizer, the title plus space and then the abstract, the longest is 256 tokens, and the output is 384-dimensional unit vectors, and the complete corpus is retrieved with the cosine similarity. token is the input unit after tokenization, which is not equivalent to Chinese characters or English words. Model card at the pinned revision To determine the version together with the model file hash, you can't just write a variable model name.

RRF is the fusion of countdown rankings. This time, "Why Mixed Search is effective: What rankings have changed in RRF?" ” rules:

s(d)=∑j∈{B,D}1[d∈Lj]60+rj(d).s(d)=\sum_{j\in\{B,D\}}\frac{\mathbf1[d\in L_j]}{60+r_j(d)}.

where, the document is dd, B,DB,D Denotes terms and dense search respectively, LjL_j It is the maximum 100 candidates for this road, rjr_j Count from the beginning; the document that does not enter the road contributes zero. After the complete union score, take 100, and the scores are the same in descending order of the docstring number. Formula Reference RRF original paper, the execution is officially implemented using fixed Pyserini.

Freeze migration and evaluation method diagram

The shared box means that the same configuration runs separately on each dataset. The corpora stay separate, and only the audit uses the labels. We reused a fixed version of the earlier source-domain results. RRF incurs the cost of two retrieval runs. Returning 100 results, as a single retriever does, does not equalize compute. The strong baseline that interleaved results in “Why Does Hybrid Retrieval Work? Which Ranks Does RRF Change?” was omitted from this transfer test. The experiment therefore cannot establish that RRF outperforms other fusion rules at the same budget.

Guide to the official implementation: find an interface that will change conclusions

Reading from the data loader, confirm which Qrels file is filtered by the query. Qrels are the queries and documents that are related to the document. Then look at BEIR's BM25Search.search: how it forms a return dictionary, filter number, and invocation field queries. The fifteenth record of the unintelligible hit is in this layer without the return key.

Dense paths look at SentenceTransformer.encode, mask mean pooling, unitization and BEIR full library cosine retrieval. The mask is used to exclude the complementary position; this time, two short texts are independently recalculated, the input shape is [2,8], the hidden state is [2,8,384], the pooling vector is [2,384], and the maximum difference from the official output is zero.

Finally, look at the recalculation and merge functions of Pyserini. The two official modules in the package keep the original byte, and the outer layer only has a unified ranking and the same division order. The source map gives the real commit, line number, license and the local corresponding location. The BEIR full library does not repeat the packaging, but the downloader obtains the pinned commit; the self-write adaptation and audit modules are all delivered with the source code.

Minimum experiment: check the resources first, and then run the full amount

The real test run uses five queries and thirty-two documents, only checking the entrance, shape and resources, and the score of the abbreviated library is not used as a benchmark conclusion. After the test run, the full amount of operation is determined according to the time and memory, and the parameters are not modified according to the score. The model is executed under the conditions of CPU, Float32, four threads, and batch sixteen, and the random seed 87 is used to record the reproducible execution state.

Officially run the [323,384] query matrix with the [3633,384] document matrix. Coding retrieval takes 37.13 seconds, including import, verification, indexing and evaluation total time 42.65 seconds; Python peak memory is about 1.04GB. Elasticsearch has another 512MIB heap, which only saves the heap snapshot at the end, and does not measure its peak resident memory. Downloads and service starts are not within this time-consuming, and there is no remeasurement of source domain inference costs.

The length limit truncated 2864 documents and no queries. The same limit thus affected a large part of the target corpus, but we lack the human annotations needed to tell whether it removed crucial evidence sentences. We retained the limit specified in the frozen protocol without lengthening the inputs after seeing the results.

Evaluation protocol: keep the level first, then talk about the migration difference

NFCorpus official statement that the relevance is constructed by direct reference, indirect link and subject relationship, not the medical correctness of one by one. Data Home Explain the construction rules and the scope of academic use. This BEIR Test actually contains 11,758 first-level and 576 second-level positive annotations, and the original three-level description cannot be mechanically set into the value range of the exported file.

The primary metric nDCG@10 is a normalized gain by location discount and dividing by the ideal sort score. Here follow the linear grade gain of TREC_EVAL:

nDCG@10(q)=∑i=110g(q,di)/log⁡2(i+1)IDCG@10(q).\mathrm{nDCG@10}(q)=\frac{\sum_{i=1}^{10}g(q,d_i)/\log_2(i+1)}{\mathrm{IDCG@10}(q)}.

qq is a query, did_i is the ii-th document, and gg Keep the labeling grade; the ideal score will sort all the known related documents by rank. The non-annotated document is counted as zero in this agreement, which does not mean that it has been manually confirmed that it is not relevant. Recall@100 calculates the ratio of the documented positive correlation document to be retrieved, and MRR@10 only cares about the location of the first positive correlation document.

The number of marked positive related documents in the target query ranges from one article to 475 articles, with an average of about 38.19 articles. For many queries of related documents, even if the first 100 digits are all related, Recall@100 may not reach one. This is a scale that is jointly specified by candidate truncation and label density. It is not advisable to directly subtract the two sets of Recalls and blame the model.

The source domain and the target domain are averaging the queries respectively, and do not hardly match the two sets of different queries into samples. The pairing difference is only calculated within the same query: the same target query uses a dense score to subtract the BM25 score, and then summarizes the victory, flat and negative. The original SciFact 300 confirmation queries have not been executed; the target test has been used for this migration and debugging, and the subsequent confirmation set cannot be called a confirmation set.

The actual result: the mean value, the coverage and the outcome are not equivalent to the outcome

The following tables are all from the saved real output, not the paper report or the results of the list.

data range BM25 nDCG@10 MiniLM nDCG@10 RRF nDCG@10
SciFact, 809 has been used for development query 0.69427 0.66020 0.71222
NFCorpus, 323 frozen migration queries 0.32689 0.31594 0.35311

The mean difference of the dense relative BM25 changes from −0.03407 to −0.01095, which is negative on both sides; the difference between RRF is +0.01795 and +0.02622, respectively, and both sides are positive. The direction of the prior attention is maintained, but the difference value cannot be explained as "how much ability has been improved" in new areas.

The dense retrieval of the target domain is 109 wins, 106 draws, and 108 losses: the winning query is slightly more, and the average difference is still the loss, indicating that the victory or loss range is equally important. Its Recall@100 is 0.31151, which is higher than the 0.24814 of BM25, and the top ten digits are not ranked better. The RRF's Recall@100 is 0.32248, and the pairing win and draw loss are 122, 136, and 65; there are still 43 queries in the first 100 without any labeled related documents.

The improvement of the RRF mean cannot be interpreted as benefiting every query. For example, the failure case of automatic selection, PLAIN-1320, has an nDCG difference of about −0.57097; the difference of gain case PLAIN-924 is about +0.55778. These numbers are used to track rankings and are not manual misclassifications. Confidence interval is also not provided this time, so it cannot be claimed that the overall is significant.

Failure investigation: the ability to run is not enough, the denominator must also be held

The first command to miss the test run parameter, start the full amount of BM25 in advance; query the integrity assertion that the missing items are found to exit, and the neural coding or aggregation evaluation has not been performed. Failed directory and logging. After repairing, use the same field to query the service directly to check the service, confirm that 15 items are indeed zero hits, and then make the empty candidates and let the evaluation be zero; the search score has not been changed.

This repair also makes the existence of the target set debugging access this time, and cannot be packaged as a complete blind test. If only the returned 308 items are averaged, the loss happens to be a failed query, and the report will be optimistic. Another problem that is easy to miss is the binary gain formula of SciFact; the rank difference is erased in the new collection. The 969 groups of query indicators and 46,852 fusion scores of the three methods of independent audit recalculation are recalculated, and the maximum error is about three times ten negative 16 to the power.

Such acceptance should precede the merits of the interpreted model. Beginners can first check a query: take out three sets of rankings, compare the labeling grades one by one, and calculate the discount contributions of the first few people. The engineering reader will check whether all the inquiries are complete, whether the candidates are from the same corpus, and whether the score is limited. The study reader further asked: Does the zero-annotated document calculate zero affect the relative evaluation of the two models? If the two retrievers find different types of documents, the label coverage difference may also enter the score. This article does not make a supplementary judgment, so this is still a limitation that requires new evidence.

The parameter invariance and input distribution should also be distinguished. The same tokenizer encounters a shorter query, more truncated summary, and the amount of information actually processed will still change. Protocols record these amounts in order to enable the next experiment to choose a clear variable; they themselves do not constitute a cause and effect conclusion, and they do not seek reasons for exemption for negative results afterwards.

Testable research questions: first register the counter-example, and then make improvements

The author infers that the dense path to increase the candidate coverage may not prioritize the high-level related documents to the top ten; at present, it can only be called one explanation. Competitive interpretation also includes annotation density, short query ambiguity and document truncation. None of them can be independently verified by an equal score.

Subsequently, the differentiation experiments can be registered on the data that is allowed to be explored: fixed candidate sets, only the sorting rules are exchanged, and whether the coverage advantage is still The front row gain cannot be converted; or the fixed model is compared with the candidate budget, and the predefined length interval is compared, instead of picking the threshold after winning or losing. To adjust the length or fusion constant, you must go back to the development process and leave additional data for the new confirmation evaluation. This time, if you don't promise in advance, you will win.

Code and instructions for reproducing the results

The source code of this article has been released and verified by anonymous downloads: Source tree at the pinned commit、README and run instructions、SOURCE_MAP source map. The 83 files of this version are the same as the local archives one by one, and the real model test run of the public copy has also passed.

After installing a fixed dependency by README, preparing the public data, and starting the local service, the minimum command is:

HF_HUB_OFFLINE=1 TOKENIZERS_PARALLELISM=false python run.py --cache ./cache --out smoke-new --smoke

The products include three-way ranking, query indicators, stage status and tensor checking; remove the test run parameters and replace them with a new output directory, that is, run the full target experiment of this article. The audit entry reads the original save result and caching of the verified data. Contains complete data, model weights, virtual environment or keys; the complete results described in the text are reserved separately from the library test run.

Summary

This migration retains the mean direction of the method, and also exposes the difference between "more coverage", "more front row" and "more query wins". The most valuable continuous research assets are freezing protocols, complete denominator and query records. They allow the next step to be refuted without having to re-pick the configuration to sustain the success story.

References

Retrieve with actual open date: 2026-09-30. The official implementation of permalinks, licensing and file hashing can be found in the source map.

  1. Nandan Thakur、Nils Reimers、Andreas Rücklé、Abhishek Srivastava、Iryna Gurevych,2021: BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models.
  2. Vera Boteva、Demian Gholipour、Artem Sokolov、Stefan Riezler,ECIR 2016: A Full-Text Learning to Rank Dataset for Medical Information Retrieval and Original data description.
  3. David Wadden et al., EMNLP 2020: Fact or Fiction: Verifying Scientific Claims.
  4. Gordon V. Cormack、Charles L. A. Clarke、Stefan Büttcher,SIGIR 2009: Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods.
  5. Sentence Transformers: all-MiniLM-L6-v2 Fixed Model Card;Elastic: 7.17 MULTI_MATCH Documentation.