
Date: 2026-08-28
Suitable for readers: graduate students, new scientific research, engineering researchers who want to establish an auditable RAG experiment
Search date: 2026-08-28
Abstract
a lot of RAG The demonstration treats "can answer after uploading the document" as a completion, but the reproduction of scientific research really needs to answer four different questions: whether the candidate paragraph is It is retrieved, whether the sorting pushes the evidence to the front, whether the generated content is referenced one by one, and whether the reference really supports adjacent statements. Just looking at the final answer, it is difficult to know which link is the impact of the change.
This paper establishes a minimal and detachable baseline: run BM25 sparse search and Embedding dense search on the same corpus and query, and use the fusion results to form a candidate set, and then the Reranker The second-stage reranking, and finally the evidence with the stable document ID is generated and the reference is generated separately, and the search and reference indicators are separately reported. The matching script is fixed SEED, which provides a pure standard library protocol Smoke Test, and also retains the real dual encoder and Cross-Encoder optional entry of Sentence Transformers.
The boundary of the evidence needs to be clear first: this paper does not reproduce the paper values of RAG, DPR, BEIR, or ALCE, and does not download large-scale corpus or neural models. This actual verification only covers the deterministic operation and reference evaluation unit test of first-hand data verification, code syntax, and six-segment TOY Corpus; the feature hash vector and transparent interaction score in the TOY backend are not well trained Embedding/Reranker, can not be extrapolated to the real retrieval quality.
Reproduce value and goal: first split the system into falsified chains
The original RAG paper treats external documents as non-parametric memory and combines Retriever with the generator for knowledge-intensive tasks. This article does not pursue end-to-end training, but reproduces more basic experimental ledgers: freeze corpus snapshots, segmentation rules, query sets and gold support; save the doc ID for each stage with scores; and finally, each answer statement points to the evidence-recoverable evidence.
The minimum goal is not to prove that a certain Retriever is "best", but a positioning error. If the GOLD paragraph has never entered the TOP-100, it is a recall failure; when entering the candidate but not TOP-5, it is a failure of sorting; the evidence is correct but the answer is wrong, it is a failure to generate or prompt; the answer is [d3] If it is not supported by D3, it is a failure of attribution. The four must use different indicators, and cannot be smoothed by a total score.
Data Contracts and Sample-by-Sample Logs
A set of reproducible RAG experiments requires data contracts instead of model names. Save at least the corpus doc_id, source URI, snapshot or file hash, chunk_id, title, body, original character interval and segmenter version. doc_id It must be run stably across; if it is re-divided, a new version should be generated and the parent document relationship should be retained, and the same ID cannot be quietly pointed to another paragraph of text. The dynamic web page also needs to save the crawling time, otherwise even if the URL is the same after a few weeks, the content of the evidence may have changed.
The query table is saved at least query_id, original question, Gold Support ID, whether it can be answered, data segmentation and annotation description. The meaning of gold support is also written clearly: is it "including the answer string" or "enough to support the statement independently"? The former is suitable for cheap answer matching, and the latter is close to reference evaluation. If you ask for two pieces of evidence in the multi-hop problem, one of the hits cannot be counted as a complete recall; if you cannot answer the question, you need to include the correct abstention into the protocol, rather than forcing the system to always find a reference.
Each query should write a JSONL log, which records the ranking and original scores of BM25 and Embedding, fusion candidates, RERAANK Score, actual text to the generator, model output, resolved reference ID, delay and error. In this way, the final answer can be played back to the specific stage, and it can also check whether the top-k is deduplicated, the permission filtering or the context is truncated and quietly deleted. Only saving the average Recall and the final answer will lose the trajectory of the most valuable research value.
Corpus, queries, and models should be divided into versions. If the chunk length, fusion weight, or candidate k is selected with dev set, the final test must be kept; if the document is continuously increased, the time segmentation and index time should be written to the result. Otherwise, the "new method boost" may only be the index to see the updated evidence, or the test query has been repeatedly leaked into the training set.
Core Module: Four Steps instead of a Black Box
1. BM25: Strong baseline for word surface matching
BM25's common single-item scores for query words t and document d can be written as:
Among f(t,d) is the word frequency, |d| is the number token number, avgdl is the average length of the corpus; k1 control word saturation, b Control length normalization. The tensors in the implementation are not complicated, and the essence is the score table of "query words × candidate documents". The difficulty is tokenization, capitalization, Chinese cut words, stop words and field weights: the preprocessing changes, the so-called BM25 control group is no longer the same. Lucene Official BM25Similarity The default value is k1=1.2, b=0.75, this article also uses this set of values, but does not claim that it is completely equivalent to Lucene's tokenization and IDF details.
2. Embedding: Independent encoding of queries and documents
Double tower dense search to calculate query vector separately with document vector , and then sort with inner product or cosine similarity:
Document vectors can be indexed offline, and each query needs to be encoded only once, so the approach scales to large corpora. DPR uses separate question and passage encoders trained with in-batch negatives. Dense retrieval can match texts that use different words, but its performance depends on the model’s domain, query and document prefixes, pooling, truncation, normalization, and approximation error in the index. The script’s default 256-dimensional feature hashing simply checks that encoding, similarity scoring, and ranking work together; --backend neural is the only setting that loads a Sentence Transformer by its fixed model name.
3. Rerank: Only do joint reading of small candidates
Cross-Encoder will [query, passage] Send a Transformer together to output a correlation scalar . It allows Query token to interact with the Passage Token, which is usually more refined than a standalone vector, but has to be re-formed for each candidate. Therefore, the reasonable agreement is to first let BM25/Embedding take top-k, and then rerank dozens or hundreds of items, instead of scoring the entire library one by one.
Candidate collections must also be written. This article uses Reciprocal Rank Fusion: accumulate the ranking r in each order . This fusion method works even when BM25 and cosine similarity scores are on different scales, making it a transparent baseline. It is neither the only approach in the paper nor a default. A reranker can reorder only the candidates that retrieval returns. A missed gold document stays missing.
4. Quote: Existence, correctness and completeness are three different things
Reference tags exist and can only prove that the format is correct. At least distinguish: Validity checks whether the DOC ID exists; Citation Precision checks whether a single reference is relevant or supports statements; Citation Recall/Completeness Check if each verifiable statement is fully supported by referenced collections. The ALCE paper uses the NLI model to evaluate the implications of statements and reference paragraphs, and clearly distinguishes the Citation Recall and Precision.
This article Toy Evaluator does not run NLI, but uses the artificially given claim→support doc set to cooperate as Oracle. It deliberately puts a misreference and a non-existent ID to ensure that the test can be obtained separately 2/3 validity、1/3 precision、1/2 Recall. The real experiment must report the Oracle label or NLI judgment error together, and the automatic entailment cannot be used as a fact judgment.

Official paper and code reading route
Start with Section 2 of the RAG paper. The authors treat the retrieved document as a latent variable. In RAG-Sequence, one document accounts for the whole output sequence. In RAG-Token, each token can depend on a different document. The pipeline here borrows the idea of retrieving from external memory before generating an answer. It does not reproduce either probabilistic model.
The second stop is to read section 3 of the DPR paper and the author repository. Look first generate_dense_embeddings.py How to generate a Passage vector, and then look at dense_retriever.py Query vector, index and search_knn How to connect, finally check the answer matching of QA Validation. The repository has been archived as read-only, and the commit should be recorded when reproducing, instead of assuming that dependencies are still seamlessly installed.
Read Sentence Transformers official Retrieve & Re-Rank in the third stop: Bi-Encoder is responsible for quick candidate retrieval, and CrossEncoder only rearranges candidates. When replacing the example with your own data, first fix the model card, maximum length, whether to normalize, batch size, device and candidate K.
Read ALCE's fourth stop retrieval.py、eval.py with configuration. It not only scores the final answer, but also saves the top-100 search results, and distinguishes between the integrity of the branch and the independent reference in the Citation Quality. The cost of large indexes is very high, and the author repository also clearly gives the disk and GPU burdens; this means that "download success" is not the same as "protocol has been reproduced".
Minimum Experiment: How to Run and Record
Pure standard library checking does not require installation dependencies:
python3 code/minimal_rag_baseline.py --check-only
python3 code/minimal_rag_baseline.py --backend toy --top-k 4
The real neural backend needs to be downloaded from the Internet to the Internet, and this article does not run:
python3 -m pip install -r code/requirements.txt
python3 code/minimal_rag_baseline.py --backend neural --top-k 4
The script prints BM25, Dense and Reranked Doc IDs, and then calculates Recall@3 and MRR. Recall@k here is defined as: for each query, as long as any Gold Support appears in the top-k, it is denoted as 1, and then the query average is average; the MRR uses the inverse of the first relevant document ranking. If a query has multiple necessary evidences, it should be reported to SET Recall or nDCG, and it is not allowed to "hit any one" to cover up the multi-hop missing evidence.
Formal minimum reproducing recommendations to compare at least four groups: BM25; Embedding; BM25+Embedding Fusion; Fusion + Reranker. Each group uses the same corpus snapshot, chunk, query, gold, filter rules and k to record the query ranking, not only the average value. In the generation stage, three controls are added to the closed-book, gold-context and retrieved-context to distinguish the upper limit of model capacity, retrieval loss and generation loss.
Evaluation protocol and Report Form
Retrieve layer prioritize reports Recall@k, MRR or nDCG@k, and give candidate scale, delay and index size at the same time. The Rerank layer should not only focus on the top-k after the reranking, but also report first-stage oracle Recall: if gold is not in the candidate set, there is no room for improvement in the second stage. The end-to-end layer reports task metrics and sample-by-sample answers, but must be parallel to Citation Validity, Precision, and Recall, and it is best to manually sample and check NLI for misjudgment.
Segmentation is the most easily overlooked variable. If the chunk is too short, the context will be dropped, and if it is too long, the keyword will be diluted and the Cross-Encoder/generator truncation will be added; overlap will create a near repetition, so that the Recall seems to be improved but redundant references are added. The chunk code and corpus hashing should be frozen, and the length distribution, repetition rate, empty block and truncated ratio should be counted.
At least the link to run multiple SEEDs is model training and generating sampling; deterministic BM25 does not need to forge SEED variance. ANN indexes, GPU kernels, or samplings may be non-deterministic and should record library versions, hardware, and parameters. If only one neural retrieval is run, the results should be exploratory rather than stable conclusions.
Failure analysis: check by stage
The first category is the mismatch of the vocabulary. BM25 cannot find synonymous expressions, and Embedding presses the abbreviation or the wrong place for the abbreviation or proper name. First check the query top-20, tokenization and query/document input templates, and then decide whether to mix retrieval or fine-tuning; do not directly call the generation prompt to mask the recall failure.
The second category is incomparable scores. Directly adding BM25, cosine and Cross-Encoder logits will make a certain path dominant. RRF can be used first, or only the normalization and weights can be fitted on the dev set; once the weight is adjusted with the test set, the evaluation leak occurs.
The third category is reranking degradation. Generic reranker may not understand the relevance in the art, or the candidate is too long to be cut off evidence. The rankings should be checked before and after rerank, and the hard negatives should be retained. Average points increase but key long documents fall, and the final reference may still be compromised.
The fourth category is "reference decoration". The model refers to the real doc ID, but puts the conclusion that the paragraph has not said; it may also stack five sources per sentence to improve the recall. Fixes require claim-level alignment, irrelevant reference penalties, and manual spot checks, rather than just checking the parentheses format.
The fifth category is that the answer indicators cover up evidence errors. Exact Match may be correct due to the memory of the model parameters, even if the error document is retrieved; in turn, the evidence is correct and may be wrong because the answer is different from the GOLD. The closed-book and gold-context controls can help identify these two situations.
Questions for further research
- Under a fixed delayed budget, how should more first-stage candidates and stronger rerankers be allocated?
- Whether the failure query of BM25 and Embedding is complementary, how will the complementarity change with the field and chunk length?
- After using Hard Negatives fine-Tuning Reranker, can the improvement come from real evidence recognition, or is the shortcut to the word data set?
- How robust is the automatic NLI judgment of Citation Recall and Precision to long statements, multi-source and conflicting evidence?
- After "abstention" is exported as a legitimate output, retrieve Confidence, evidence conflict and answer how Accuracy is calibrated together?
Summary
Instead of connecting vector libraries, LLMs, and interfaces, a trusted RAG Baseline establishes a replayable chain of evidence: BM25 provides the Baseline of the word plane, Embedding provides semantic candidates, Reranker To jointly score small sets, the generator can only refer to the stable DOC ID, and the evaluation separates the recall, sorting, answer and reference support.
This article completes the minimum reproduction of the protocol level: formulas, modules, official code routes, deterministic scripts, and intentional failure of reference unit tests have been aligned; large-scale indexing, real neural models and papers have not been run. In the next step, you should run four groups of ablation on the frozen small data and save the query log, and then consider expanding the corpus or replacing the generative model.
References
Search date: 2026-08-28. Official documents, repository and model defaults will change; real reproduction requires a locked version, commit, model revision and corpus snapshots.
- Patrick Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020.
- Vladimir Karpukhin et al., Dense Passage Retrieval for Open-Domain Question Answering, EMNLP 2020.
- Stephen Robertson et al., Okapi at TREC-3, NIST TREC-3 Proceedings, 1994; this official catalog contains the original paper and author information.
- Apache Lucene, BM25Similarity API, parameter semantics and default values.
- Nandan Thakur et al., BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models, 2021; Official repository.
- Sentence Transformers, Retrieve & Re-Rank official example.
- Tianyu Gao et al., Enabling Large Language Models to Generate Text with Citations, EMNLP 2023; ALCE Author Repository.
- Meta Research, DPR official reference repository and
dense_retriever.py. - Sentence Transformers,
cross-encoder/ms-marco-MiniLM-L6-v2Model card, the matching script can choose the model source and usage of the neural Reranker.