The cover of the experimental research literature search can be audited

Summary: Let each score have a source first

The previous teaching experiment let us know how to calculate the indicators, but did not answer a more practical question: if the assumptions in the code are still established, the assumptions in the code are still established? In this article, the research object is replaced by SciFact, and there is only one goal: starting from the fixed data file, searching the link through the official BEIR term, leaving the candidates, scores, versions and failure records that others can check. There is no neural model training here; the real target model is the BM25 sorting model in Elasticsearch.

The run covered the full corpus of over five thousand documents and more than eight hundred development queries. We neither compared dense models nor searched for better parameters at this stage. Compared with the previous round, the study moves to a real retrieval task and implementation. The corpus, fields, tokenization, candidate budget, and evaluation rules are fixed. The resulting retrieval baseline package is shared by the next five articles.

1. Reproducing value: retrieval of literature is not equal to verifying scientific conclusions

What the paper reports. Wadden et al. proposed SciFact at EMNLP 2020, which involves finding a summary, judgment support or rebuttal relationship with evidence, and positioning the basis. This article only reproduces this link of literature retrieval, and does not run a model for judging the position or extracting evidence. Recall cannot be called a fact verification accuracy. Original paper

What the paper reports. The BEIR work of Thakur et al. incorporates different retrieval tasks into a unified framework, and points out that the baseline of the term is still competitive. This gives us the reason to "establish a baseline first", not to predict that BM25 must exceed Embedding. The multi-data set conclusions used in the paper cannot directly replace a SciFact experiment with this machine. BEIR,2021,v4

What we tested. There are three types of entities in the file: Corpus archives, queries save queries, and the relationship between queries and related documents. They are connected by a string ID, and cannot use the file line number as the primary key. The program checks the duplicate ID, the related document is missing, the query and the document ID collide, and the evaluation query collection is not smaller after loading.

2. Data auditing: the most dangerous thing is the familiar name

BEIR file real amount Corresponding to the original SciFact
corpus 5,183 documents The original document ID is the same as the title
train qrels 809 Query, 919 pair relationship original train
test qrels 300 query, 339 pair relationship Original dev, not original untag test

This is the relationship after ID verification, and cannot be guessed according to the file name. The original Test Set is not publicly labeled, and the official description also clearly distinguishes between development evaluation and test prediction. This project treats BEIR Train as a development pool, and BEIR Test will be reserved for subsequent confirmation. At present, the confirmation set only does structural and mapping audits, and does not retrieve, pick wrong cases or adjust parameters; it is not an external sealing blind test, and it is not guaranteed to be independent at the document level. Official data description

More importantly, both Qrels correspond exactly to the original cited_doc_ids, not exactly evidence document keys. The development pool has more than 300 declarations that the Evidence is empty, and there are still retrieval related documents. Therefore, "find the documented literature" cannot be automatically interpreted as "found evidence that supports the statement". This difference determines that the subsequent misclassification should study the matching of literature, rather than the authenticity of the statement.

There is also a real failure: the initial audit requirements are the same verbatim, and the error is reported directly. The inspection found that there was a pure blank difference between the 1,055 documents and the original abstract sentence splicing results; after the normalization blank, all were consistent. We keep the failure log and split the audit into "the title is exactly the same" and "the same after the summary is normalized", and the search still uses the original BEIR file, without quietly rewriting the input.

The version cannot just write "the latest version". The original download address has latest, this time with the actual file SHA-256 locked content, it cannot prove that it will never change. Get the script and stop when encountering different bytes. The license is subject to the opened original repository: the statement and evidence are marked as CC by 4.0, the abstract is ODC-By 1.0, and the code is Apache 2.0. The data card still shows another old caliber and cannot be confused. Compressed packages no longer distribute complete corpus. Fixed version license

3. Core ideas and formulas: first look at how the fields participate in the sorting

BM25 is a search model that is scored by term matching. It rewards the occurrence of query words, suppresses the recurring benefits of the same word, and considers the length of the document. The English analyzer will affect the form and stop words, so "both are called BM25" do not represent the experimental equivalent. Use Elasticsearch 7.17.9 default parameters this time k1=1.2、b=0.75, do not search parameters; the former controls the saturation of word frequencies, and the latter controls the normalized length of length. Official Similarity Documentation

This link indexes the title and summary respectively, using best_fields, the synthesis relationship of the two fields can be written as:

s(q,d)=max⁡(st,sa)+0.5min⁡(st,sa).s(q,d)=\max(s_t,s_a)+0.5\min(s_t,s_a).

The query here is q and the document is d, and the two scalars are the matching scores of the title and the summary field, respectively. The coefficients come from the fixed source code tie_breaker, not the weight we called out. It is not the same protocol as "Title plus space plus a summary and a score" is not the same agreement, nor does it simply add the two scores. Official multi-field description

Retrieval and Evaluation Process

The relevance labels in the figure only enter the audit terminal and do not participate in the index or search score. The locked confirmation set does not return to the development process arrow. The method diagram only explains the relationship, and all values are from the saved JSON logs.

4. Guide to the official implementation: read four places along the data stream

Look first GenericDataLoader.load: It filters queries by Qrels, so the denominator audit must be placed after loading. Look again BM25Search.index: The title and body enter different fields, and cannot be spliced according to the function name. Look at lexical_multisearch: Determine the field, query type, cross-shard statistics and return quantity here. Look at last EvaluateRetrieval.evaluate: It calls the TREC_EVAL interface and keeps the aggregate value five decimals, and the original query value needs to be saved separately.

The actual operation also encountered two obstacles to achievement. The old Elasticsearch client referenced the deleted type alias for NumPy 2, fixed to NumPy 1.26.4 and then imported through. The current BEIR submission also requires the term retrieval class to implement two dense encoding abstract interfaces, resulting in failure of instantiation. accessary compat.py Only make these two non-applicable interfaces explicitly throwing errors; index, query and scoring are all inherited from the original implementation, and there is no other "approximate BM25" written. The source map records the precise commit, line number and adaptation boundary.

V. Minimum Experiment and Evaluation protocol

The freezing protocol has been saved before operation: first use the five development queries sorted by ID to make Smoke test, and then run all 809; the complete corpus is always maintained at 5,183 articles. There is no training stage, no optimal seeds or parameters are selected; seed 82 is used as a configuration record, and the baseline of the term is not dependent on random initialization. The index adopts a single shard, the query batch is 64, and each can reserve a maximum of 100 candidates.

The main indicator is the normalized loss cumulative gain nDCG@10:

nDCG@10(q)=∑i=110ri/log⁡2(i+1)IDCG@10(q).\mathrm{nDCG@10}(q)=\frac{\sum_{i=1}^{10}r_i/\log_2(i+1)}{\mathrm{IDCG@10}(q)}.

It is marked here as binary dependency, and when the document correlation in the digit of the third position is related to r is one, otherwise it is zero; the denominator is the ideal sorting score of the same query, and the missing position of the short list is calculated as zero. First query and calculate, and then do the macro average, you can't mix all hit documents into a total ratio. Auxiliary indicator Recall@100 measures the ratio of related documents to be retrieved, and MRR@10 measures the inverse of the ranking of the first related documents.

The official retriever requests one extra candidate and filters out any document with the same ID as the query. No such ID collisions occurred here. For 808 queries it returned 101 results, and for one it returned only 27. We saved the raw results, sorted by descending score with ties broken by descending document ID as a string, then kept up to 100. We checked the tie rule against trec_eval and added no unmatched documents. The resulting 80,827 candidate records contain the actual returns, without assuming a dense matrix of scores for every query–document pair.

"Complete candidates" here refer to all the return of this agreed budget, not all document scores are exhausted. The document that has not entered the return pool does not save scores, and it is impossible to use this document to calculate the recall at arbitrary depth. In the future reranking, only the same candidate pool can be used; if the search depth is increased, it should be registered as a new experiment. For parallel scores, the current collation only constrains the candidates that have returned candidates, and there is no guarantee that the cross version of the boundary candidate set is exactly the same.

6. This actual verification: failure to retain outside the mean

Fixed Development Agreement Test result
nDCG@10 0.69427
Recall@100 0.93506
MRR@10 0.66252
The top ten do not have any relevant documents 133 / 809
The first 100 people do not have any relevant documents 47 / 809

These values come from the machine run and are not a paper report or a list of transcripts. All queries nDCG and Recall are recalculated with an independent scalar formula, and the maximum error with the official interface is about one multiplication of the negative 16th power of ten. There are also independent checks for empty results by zero, missing query refusal to pass, sequential order and second place hit. The check supports "procedures are calculated according to the established caliber", and does not prove that the labeling is complete.

The CPU environment is macOS 26.6 ARM64, Python 3.12.14, and Lucene 8.11.1. The preprocessing is about 0.149 seconds, the index is about 1.281 seconds, and the search is about 0.651 seconds, and the evaluation is about 0.066 seconds; the total entrance is about 2.867 seconds, excluding download and service startup. Python peak resident memory is about 112.4 MiB; the upper limit of the service heap is 512 MiB, the heap occupancy snapshot is about 269.8 MiB at the end, and the service peak memory is not measured. The service has been warmed up by Smoke, so it cannot be called the time-consuming cold start performance.

7. Failure investigation and verifiable research problems

First check the experimental failure, and then explain the model. Network agents are unreachable, ports are rejected by sandboxes, dependencies, and abstract classes are all recorded; they are not retrieval failures. It is also more reliable to explicitly refresh and check the number of documents after indexing, which is also more reliable than waiting for a fixed number of seconds. When the index is reused in the future, the program will check the stored fields by document to avoid the old data mixed in the index with the same name.

Query 173 has only 27 candidates, and the related documents do not appear. This is just a locatable phenomenon and cannot be immediately named "Semantic Divide". Possible explanations include lexical processing, insufficient matching of proper nouns, and the gap between the annotation corresponding literature and expression. The unfinished manual misclassification is clearly reserved for manual verification, and there is no forgery of the conclusion of manual review.

Author’s interpretation. The next article can test a limited hypothesis: after freezing the same corpus, query and candidate budget, does the dense model mainly improve this missing recall query, and at the same time lose a tokenization item hit query? You need to look at the matching difference between query and pairing and add and lose documents, not just the total mean. If the lift is only from longer inputs or more candidates, it cannot be attributed to presentation ability. The previous failed samples can be used for development analysis; the confirmation set must wait for the plan to be frozen before re-evaluating, and the literature may be shared between the queries, and the subsequent interval is estimated to be audited for the grouping correlation.

8. Source code and reproduction entrance

The full code of this article has been published on GitHub, and the following entrances are fixed to submit cae9252, subsequent maintenance will not change the version referenced in this article:

First press README to install fixed dependencies, verify data and start local services, and run in this source directory python run.py --cache /path/to/research_cycle6 --out results/my_run. Please check the above link for the specific implementation and configuration, no blog ZIP attachment is required.

This is the actual verification of the complete development set entrance; plus --limit 5 You can try it out. Expected products include original candidates, TREC runFile, query metrics, aggregated results, data auditing, environmental and stage states. This execution to the real retrieval and official evaluation, no dense coding, reranking, training or confirmation set is run. Cache identity includes model build, analyzer, data, configuration, code and dependency versions; this article has no prompt and decoding, and the explicit record is empty. The source code package excludes environment, index, complete corpus and service distribution package, and external resources are recovered by a fixed hash script.

The code is kept as a complete snapshot of this experiment together with the article, and the running data and environment are placed in a separate cache directory. If the shared module is extracted later, the required version should be fixed, and the required code should be included in each experimental package; historical articles cannot directly rely on changing public directories. This can not only continue a research project, but also allow readers to download a certain article and review the results at that time.

9. Summary

The harvest of the first real reproduction is not just a score, but knows what the score is measured. Divide names, relevance semantics, field combinations, and number of candidates will change the interpretation. After saving the failure and version, the next article is eligible to ask "Which queries have been improved in another way". This article has established a reproducible development baseline, and the next article will study the query-by-query changes brought about by the dense search of the same protocol.

References

Retrieve with actual access date: 2026-09-25. The year, author and version of the paper have been checked; the source code is permanently linked, the license and local mapping are shown in SOURCE_MAP.

  1. Wadden, Lin, Lo, Wang, van Zuylen, Cohan, Hajishirzi. Fact or Fiction: Verifying Scientific Claims. EMNLP, 2020.
  2. Thakur, Reimers, Rücklé, Srivastava, Gurevych. BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS Datasets and Benchmarks, 2021.
  3. BEIR Authors. Official code, pinned commit.
  4. Allen Institute for AI. SciFact Pinned commit and Data Description.
  5. Elastic. 7.17 Multi-field search、BM25 Default Parameters、English analyzer.