
- Date: 2026-09-26
- For graduate students, early-career researchers and research engineers
- Scope: Complete corpus, all development queries, a freezing model; not an official test list.
- Delivery status: The supporting source code has been disclosed, and the pinned commit, file hash and reader portal have been verified.
Abstract: Average backward, new evidence may also be provided
The previous article locked the data, labeling and BM25 link of SciFact. This article only advances one question: after changing to a true pre-trained sentence vector model, what are the improvements and regressions of the same batch of inquiries? Embedding is a dense vector that encodes text into comparable dense vectors; BM25 is mainly scored according to terms matching, word frequency and document length. Name updates and more parameters can not replace the control experiment.
What we tested: On the same 5183 documents and 809 development queries, the nDCG@10 of all-MiniLM-L6-v2 is 0.6602 and BM25 is 0.6943. The query is broken down into 150 wins, 444 draws, and 215 losses. However, the dense search retrieved 25 queries that were completely missed in the first 100 BM25. The average backward and partial complementation can be established at the same time; this is more suitable as the starting point of subsequent research than the sentence "Vector retrieval is not enough".
1. Recover value: make the comparison object clear
The Sentence-BERT paper proposes to use the twin network to form a sentence vector that can compare the similarity of cosines; it provides a method basis, it is not the evidence that the minim checkpoint in this paper is the evidence that the minim checkpoint wins on SciFact. Paper and Model card Must be read separately. The BEIR paper also emphasizes cross-task search evaluation, and all areas cannot be deduced from a collection. BEIR
There is no training this time, and there is no search for ten models to pick winners. MiniLM is selected to complete real coding, exhaustive similarity calculations and case-by-case audits on the CPU. Freeze the model, maximum length, splicing method and indicators before running; the results are not ideal and retained. The new additions to 082 are dense representation and pairing comparison, reuse corpus, development query, relevance labeling and old BM25 ranking.
Also keep the most critical semantic boundary of the previous article: the Qrels here is the mapping to the relevant documents, and the corresponding CITED_DOC_IDS in this export cannot be equivalent to the support of the tag with scientific evidence. Finding relevant papers does not mean that the judgment is true or false. BEIR train corresponds to the original train; the BEIR Test left in this series corresponds to the original dev, and this article does not perform its retrieval. SciFact
2. Core ideas and formulas: from text to ranking, and then to the difference
The encoder first generates a token representation, and then the mean pooling and normalization according to the attention mask:
here is the -th token’s 384-dimensional representation, and It is a valid location mask; padding does not participate in the mean, and the special token is still involved. The batch hidden state shape is [B,L,384], after the pooling is [B,384]. The complete query and document matrix are respectively [809,384] and [5183,384], the similarity covers all 4193047 queries-document pairs, and there is no approximate index error.
The main indicator nDCG@10 is the relevance benefit of the rank discount and then the ideal sorting normalization. This data adopts binary correlation, and the hit contribution of the first place is greater than the tenth place. Auxiliary indicator Recall@100 indicates the ratio of the known documents to be retrieved by the first 100, and MRR@10 represents the countdown ranking of the first related hit, and missed zero.
is the same indicator of the same query, . First, the query ID is aligned and then subtracted, and the two independently sorted fractions cannot be directly subtracted. The difference is greater than If you remember the victory, if it is smaller than its negative value, record the negative, and the rest is flat; the flat is the indicator, which does not mean that the candidate list is the same.

3. Guide to the official implementation: check the default value, not just the interface
First look at the fixed revision modules.json: Transformer, mean Pooling, Normalize constitute the actual path. Read Sentence Transformers encode、Transformer.tokenize、Pooling.forward and Normalize.forward, check the evaluation mode, mask and truncation. Finally read BEIR's DenseRetrievalExactSearch.search and cos_sim, check the corpus sort, id mapping, and top-k return rules. The specific commit, line number and local implementation are related to the source map; the installation version of the source code is checked with the official file, and the line break difference is recorded separately.
The model card says that the input of more than 256 Word Pieces is truncated by default, and the upper limit of the length of the underlying tokenizer cannot be mistaken as a sentence vector model protocol. This time it is explicitly set to 256, and the document input is the title, a space, and a summary; the query is not added. The mask mean value of the two inputs independently is recalculated, and the maximum difference between the output and the official code is zero; the measured shape is measured [2,8,384]→[2,384], the model is in the evaluation mode and the number of trainable parameters is zero.
BEIR is sorted by length, and then the vector is mapped back to the document ID. If only the matrix is saved and the ID order is not saved, the result may be correct in shape and all semantics are wrong. This article additionally saves the order, truncation length, vector hash and ranking; these audit layers are supporting implementations, and the main code and search are still performed by the official library.
4. Minimum Experiment and Evaluation Protocol: What is fixed, what is not
Protocol write before reasoning PROTOCOL.md. Model and tokenizer use revision 1110a243fdf4706b3f48f1d95db1a4f5529b4d41, the weights and other eleven files are locked one by one hash. BEIR fixed ef83d293…, the data continues to use 082 validated file hashing; no data release date or immutable version number is fabricated.
The same complete corpus is searched on both sides, and each retains a maximum of 100 candidates; the dense side does not use the BM25 screening document first. In the previous article, BM25 only returns 27 valid candidates for a query, and this article does not forgery to make up. The two sides use the same QRELs, the same parallel sorting rule, and the same independent scoring function, and cross-check with the official evaluation.
Fairness also has boundaries. BM25 uses a matching combination of English analyzer and title and summary fields; minim splices text and truncates. They have different input budgets and pre-training calculations. This article answers "how these two sets of freezing configurations are different", which cannot isolate the causal advantage of a certain representation. It does not count the historical training cost of the model to zero.
The first import failed due to the lack of DataSets, and the log was reserved. After the completion of the completion of a command, a command is omitted to add the Smoke parameter, and the full development exploration is performed; the configuration is not adjusted accordingly. Then complete the resource check of five queries and sixteen documents, and then perform the official full-scale record. The official CPU runs with Float32, four threads, batch 16, seed 83, Python 3.12.14, PyTorch 2.6.0, Sentence Transformers 3.4.1, Transformers 4.49.0.
The total measurement time is 58.59 seconds, of which the query code is about 0.84 seconds, and the document encoding is about 52.67 seconds; the peak RSS of the process is 1160839168 bytes, about 1.08 GiB. The total time-consuming includes import and loading, excluding downloads; it is not a GPU video memory, and it cannot be directly end-to-end speed ratio with the time-consuming BM25 query that has started the service.
This pairing protocol also has an easily overlooked benefit: it separates the "model" and "problem". If a Retriever quietly discards the empty result query, the mean value may be inflated; if the two result files have different IDs, the direct comparison of the total score will lose the common denominator. Therefore, the audit procedure requires that the query sets on both sides are exactly the same as the label set, and the absence immediately fails, and the zero hit is still in the statistics. Insufficient candidates cannot be supplemented to 100 with duplicate documents.
The reproduction experiment should also treat the result cache as an identity product. This article includes the model and tokenizer revision, empty prompt, no decoding declaration, data hash, configuration, dependency and source code hash; different identities do not share vectors. Publishing packages only keep the required rankings and verification records, and the large weights, virtual environments and reconstructable vector caches are left locally. After the reader reconstructs the resources from the fixed download lock, they should first check the exit code and shape, and then see whether the score is close; the cross-machine consistency is still subject to manual verification.
5. Results: How the mean value is composed of winners and losers
| Development set metrics | BM25 | MiniLM | MiniLM minus BM25 |
|---|---|---|---|
| nDCG@10 | 0.69427 | 0.66020 | -0.03407 |
| Recall@100 | 0.93506 | 0.93243 | -0.00264 |
| MRR@10 | 0.66252 | 0.62613 | -0.03639 |
In the main indicators, the total contribution of win query , negative query contribution , all take all 809 as the denominator, and the sum of the two is the net difference. You can't just write the title with the mean of the benefit subset. No significance or Confidence interval is reported: these development claims may share original documents, query independence is not confirmed; a single run of a fixed model cannot represent the variance of the model family.

pictured by plot.py read results/development/paired.csv generate. Each point on the left figure is a pairing query, and the overlap point does not mean only one; the right picture is sorted by difference value, and the zero difference interval is not a missing sample.
Looking at candidate coverage, both retrievers found at least one relevant document for 732 queries. Only dense retrieval found one for 25, only BM25 did for 30, and neither did for 22. These counts classify queries by whether there was at least one hit. Macro-averaged Recall@100 instead averages the fraction of relevant documents retrieved for each query, a distinction that matters whenever a query has several relevant documents.
6. Failure analysis: keep the counter-example, don't rush to label the reason
Take the three maximum positive difference and the maximum negative difference according to the pre-registration rules, and break the juxtaposed by the numerical ID. The related documents of query 331 have been upgraded from BM25 bit 71 to dense 1st bit; the two related documents of query 69 are returned from bits 1 and 2 to bits 12 and 23. Query 199 is more direct: BM25 is the first to hit, and the first 100 dense top misses. The complete six cases and document IDs are saved in the pairing results, and there is no example of "best talk" by hand.
Author’s interpretation: Domain abbreviations, exact lexical matches, and information lost from long abstracts could each explain some regressions, but ranks alone cannot distinguish the causes. Truncation affected 3681/5183 documents, about 71.02%, and no queries. Of 1747679 document tokens before encoding, 1242251 remained after truncation. Relevant documents can be truncated in examples where performance improves as well as where it declines. Blaming truncation simply because it occurred would be a post hoc explanation.
Another verifiable failure comes from numerical precision: use float64 to recalculate the full library score for the saved vector, the internal order of the first 100 queries is different, and the maximum reserved score difference is about . The special audit confirms that their candidate collections and all indicators in this article have not changed. The text retains the official float32 result, and does not say low-precision playback is the same bit by bit.
During the investigation, first confirm the data hash, ID sequence, mask, normalization and truncation, and then discuss the semantics. It is also necessary to prevent automatic grouping without manual verification as "manual wrong type gold label". There is no manual annotation in this article, and there is no proof that the model pre-training corpus is zero overlapping with SciFact; the source of the scientific literature listed by the model card makes this boundary especially need to be retained.
7. From phenomenon to testable research questions
The first hypothesis is that there is a complementary complement to the two sets of rankings. The next article will only change the fusion rules, freeze the two rankings of this article, and compare the RRF and single-lane baseline under the fixed candidate budget. 25 exclusive hits provide motivation, but do not guarantee that they will be able to enter the top ten after fusion; if the increase is only from expanding the candidate budget, it cannot be attributed to the fusion formula.
The second competitive explanation is that the regressive part of the dense model comes from input cropping. The subsequent preprocessing chapters should control the splicing and length of the fields respectively, first register the main indicators and allowed calculation increments, and then observe the ranking changes of relevant documents. Extend the input If you increase the calculation amount at the same time, it is also reported. You can't keep trying to win on the failure sample of this article, and then call the confirmed set you left as unselected evidence.
Code and instructions for reproducing the results
This article has been published to the unified repository Source tree at the pinned revision, the reader can README and run instructions、SOURCE_MAP source map and Unified entrance run.py enter. Version is 8a3b931f2aa147a71db7812177f0ded4283ec9e4; All 54 distribution files have been pulled anonymously, which is consistent with the local archive-by-file verification. The directory keeps the source code, configuration, dependencies, licenses, failure logs and query records, and the model weight and complete corpus are obtained through a locked download script.
After installing dependencies and executing the download preparation, the minimum real command is python run.py --cache ./cache --out results/my_smoke --smoke. It runs a small-range check of the real model, and cannot be used as a full score. Get rid of --smoke It is the complete development experiment of this paper, the output ranking, query score, environment and stage state. audit.py You can recalculate the saved 809 pairing results offline; the vector precision check needs to run the complete inlet generation vector first. This is not trained, the confirmation set is not run, the fusion is not run, and the model is not used to replace the model.
Summary
This freezing of MiniLM has not exceeded BM25, but has left a more useful research asset: a set of traceable benefits, damage and common failure queries. The next value is not to ensure that the score is "higher", but to allow complementary, truncated and field processing these competitive interpretations to be tested separately. Readers can replay the evaluation from a fixed version that has been published and continue to test the fusion hypothesis of the next article.
References
Retrieval and actual visit date: 2026-09-26; 2026-09-27 Continue to review papers, model cards and fixed source code, and complete the public release. The conclusion of the paper, the description of the model card and this experiment are respectively marked. If the browsing tool cache is invalid, the fixed source code has been downloaded and read through the official RAW and hashed.
- Nils Reimers、Iryna Gurevych,2019: Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.
- Sentence Transformers: all-MiniLM-L6-v2 model card, this time the weight is fixed to the above revision.
- Nandan Thakur et al., 2021: BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models,v4.
- David Wadden et al., 2020: Fact or Fiction: Verifying Scientific Claims.
- Sentence Transformers 3.4.1 Fixed source code; BEIR fixed source code.