Dense Retrieval Twin Tower Search Cover

Abstract

Dense Retrieval compresses the query and document into a fixed-length vector, and then uses dot product or cosine similarity to find the nearest neighbor. What it really deserves to be reproduced is not "calling an embedding API", but three verifiable links: how the double tower produces a vector, how a negative sample shapes the space, and how the off-line index and online query form an evaluation protocol. This article takes DPR and Contriever as two main lines, read the official implementation, and provides a minimum experiment with no third-party dependencies. The script only validates the mechanism and logs, and does not claim to reproduce the BERT, Wikipedia, Natural Questions, or BEIR results of the paper size.

Reproduce value and goal: restore the vector interface to a training problem

The word face search will reward the common word item; the dense search hopes to bring the "red planet" and "Mars" closer. The cost is: the correlation is no longer directly determined by the readable word frequency formula, but is jointly determined by the training pair, negative samples, pooling, normalization and indexing. Therefore, this goal is not to chase a ranking number, but to answer four questions:

  1. What is the tensor shape of the query tower and document tower output?
  2. Why does a batch provide both positive and negative samples?
  3. Why can documents be encoded offline and the query must be encoded online?
  4. When Recall@K rises, is it just a training set memory, a fake negative sample leak or a change in caliber?

Core Idea: Double Tower, Fraction Matrix and Contrast Loss

1. The two encoders are read separately first, and then they meet in the vector space

Given the query (q_i) and the document (p_j), the two towers are calculated separately:

hiq=fq(qi)∈Rd,hjp=fp(pj)∈Rd\mathbf{h}^{q}_i=f_q(q_i)\in\mathbb{R}^{d},\qquad \mathbf{h}^{p}_j=f_p(p_j)\in\mathbb{R}^{d}

where (f_q) is the query encoder, (f_p) is the document encoder, and (d) is the vector dimension. If a BATCH has (B) queries and (B) body documents, the tensor shapes on both sides are ([B, D]). There is no cross-attention token by token; this is exactly why the document vector can be precalculated.

The similarity is often dot product:

sij=(hiq)⊤hjpτs_{ij}=\frac{(\mathbf{h}^{q}_i)^\top\mathbf{h}^{p}_j}{\tau}

Hence, a matrix multiplication of ([b,b]) the fractional matrix. Temperature (τ>0) controls the sharpness of softmax; if the vector is first normalized with L2, the dot product is equivalent to the similarity of the cosine. Whether normalization cannot rely on habit guessing, it must be consistent with the training and index configuration.

2. The diagonal is a positive sample, and the rest is a negative sample within the batch

If the body document of the (i) query is also the line (i) document, the loss is:

L=−1B∑i=1Blog⁡exp⁡(sii)∑j=1Bexp⁡(sij)\mathcal{L}=-\frac{1}{B}\sum_{i=1}^{B} \log\frac{\exp(s_{ii})}{\sum_{j=1}^{B}\exp(s_{ij})}

The remaining (B-1) documents in the denominator are in-batch negatives. Batch getting bigger usually means more comparisons, but it's also easier to miss unlabeled related documents as negative samples. A hard negative sample is a document that looks a lot like but not annotated answers; it can increase the training signal, and it may also amplify the marking gap.

3. DPR and Contriever change the supervising source, not the retrieval skeleton

DPR (Karpukhin et al., EMNLP 2020) uses Q&A supervision to train the query encoder and Passage encoder, the paper implements BERT representation and dot product as the core, and combines random or BM25 Difficulty negative sample. Abstract report: In its open domain Q&A setting, DPR has an absolute increase of 9–19 percentage points compared to the TOP-20 Passage Retrieval Accuracy of the strong Lucene-BM25. This is What the paper reports, which is not the conclusion of this TOY data in this article.

Contriever (Izacard et al., TMLR 2022) focuses on unsupervised migration: construct two disturbing views from the same text, and form a pseudo-positive alignment with the enhancement of cropping and deletion, and then do comparison pre-training. The official model defaults to the last layer of token after doing the average pooling; the official page reports that its unsupervised model exceeds 11 of the 15 datasets of BM25 in BEIR's Recall@100. It does not prove that dense searches outperform BM25 in any area; indicators, datasets and migration conditions are indispensable.

Data and Negative Sample Contracts: Evidence Freezes Before the Model Starts Training

The most easily overlooked input is not text, but correlation. Each sample should be retained at least query_id, original query, positive_passage_ids, Negative Sample ID and Source; the Passage table must have a stable document ID, title, body and slicing version. A query may have multiple correct documents, so the single positive sample selected during training cannot be mistaken as a unique true value. If Qrels allows three related documents, the other two should also be correct in the predicted hit; otherwise the so-called "difficult negative sample" may only be a positive sample of the missed standard.

Negative sample sources determine what boundaries the model learns. Random negative samples will quickly become too easy; negative samples in batch are cheap, but rely on the subject distribution of BATCH; BM25 Negative samples are similar in terms of words, suitable for forced models to distinguish between entities and relationships; the nearest neighbors dug by the old dense models are closer to the current decision boundary, but will inherit the deviation of the old model. The experimental log should record the number of four types of negative samples, the proportion and loss contribution of the false negative, and not just a general num_negatives.

Split the training, validation, and test sets by source as well. If adjacent passages from the same article appear in both training and testing, the model may find matches through shared wording rather than meaning that generalizes to other texts. For Contriever-style self-supervised training, save the original text, both augmented views, and the random seed. Changing which words are deleted, how much text is cropped, or how windows are sampled changes what the model learns to treat as invariant. For DPR-style question-answer supervision, record how answer strings are mapped to passages and how a positive is chosen when several passages could supply the evidence.

DPR / Contriever two tower training and retrieval process

Guide to the official implementation: follow the data flow first, do not follow the configuration items first

Read the DPR repository for four steps. first, look dpr/models/biencoder.py How to return the representation of the query tower and the context tower; second, see BiEncoderNllLoss.calc How to form a fraction matrix, do log_softmax, and then hand over the positive sample position to NLL; third, see generate_dense_embeddings.py How to shard the document vector; fourth, look dense_retriever.py How to load the FAISS index and output top-k. The official repository has been archived in 2023, so recurring commits and dependencies should be fixed, rather than assuming that the latest Transformers must be compatible.

When reading the Contriever repository, first look at src/contriever.py: forward Clear the padding position, and then do the average pooling according to the number of valid tokens. See later src/inbatch.py comparative learning entrance, generate_passage_embeddings.py fragment preservation, and passage_retrieval.py query encoding, index loading, neighbor search, and answer match. The official README also specifically states that the answer preprocessing of Natural Questions is slightly different from the DPR version; if the data version is not recorded, the two "Recall@k" may not be comparable.

Minimum experiment: handwriting is more explanatory than encapsulation calls

code/minimal_dense_retrieval.py Fixed random seed 57 with only the Python standard library. It maintains two independent 24-dimensional word vector tables for the query tower and the document tower, and does the average pooling of the sentence, manually calculates the 6×6 dot integrals, the softmax cross entropy and gradient, and then passes 10 passages. into an off-line index. run:

python3 code/minimal_dense_retrieval.py --check-only

What we tested The shape of the query vector is ([6,24]), the password index is ([10,24]), and the training score matrix is ([6,6]); the loss after 240 steps is reduced from 1.7899 to 0.0044, the toy set Recall@1, Recall@3, and MRR are all 1.0, the assertion passes. This result only proves that gradients, indexes, qrels and metrics are self-consistent. The training query is the same as the test query, and there are only ten corpses, so the full score is mainly a memory ability check, and cannot be extrapolated as the generalization results of DPR or Contriever.

To upgrade to a paper level reproduction, the vocabulary should be replaced with the official CheckPoint, and the Passage segmentation, query/answer preprocessing and negative sample files should be strictly reused; generate the document embedding The model hash, dimension, normalization and dtype is recorded; finally, the difference between the mass-delay of the exact search and the approximation of the FAISS approximation index is recorded. Mockup download, full Wikipedia index and BEIR review Not running, wait for manual verification.

From TOY to official model: reproduce in three paragraphs, do not zoom all variables at once

The first stage is reproducing the protocol: Keep the small corpus used here. With the standard library or a small neural network, check the indices of the positive examples, whether higher or lower scores indicate better matches, the Recall and MRR calculations, and the mapping from outputs to IDs. Deliberately swap the relevance judgments, reverse the scores, or shuffle the passage IDs. Each change should make the test fail. If the metrics stay the same, investigate the evaluation script before blaming the model.

The second stage is reproducing the model: Keep the corpus and relevance judgments fixed and switch only to an official DPR or Contriever checkpoint. Start with exact matrix multiplication and leave approximate indexing off. Spot-check whether the same passage gets the same vector when encoded alone and in a batch, whether padding contributes to mean pooling, and whether the model is in eval mode. DPR’s query and document encoders are not interchangeable. For Contriever, pooling and text normalization must match the checkpoint configuration. This verifies the model interface. Reproducing the paper’s benchmark results requires a separate test.

The third stage is reproducing the full system: Use the paper’s version of the data, generate embeddings for all passages in shards, build a FAISS index, and run the official evaluation. For each shard, record the passage ID range, vector count, dimensionality, checksum, and time taken. After merging the shards, check the total number of IDs and verify that each is unique. Compare results against exact top-k search on a few queries before enabling approximate search. Once the data, model, and exact search agree, you can interpret the speed and recall loss from approximate nearest-neighbor search.

Changing one layer per stage makes failures easier to locate. In the protocol stage, look for an implementation error or a problem with the relevance judgments. In the model stage, check preprocessing and the checkpoint. Only at the system stage do sharding, indexing, and resource allocation become the primary suspects. Jump from the toy corpus straight to a million documents and every error can look like poor model performance.

Evaluation protocol: first freeze Qrels, then talk about the advantages and disadvantages of the model

Problem Metric The caliber that must be fixed
Whether the first k results are overridden with relevant documents Recall@k Relevant document collection, deduplication rules for each query
The first relevant document is more advanced MRR@k Cut the depth, miss the zero
Is multi-level correlation sorting reasonable? nDCG@k Relevance Grade and Discount Formulas
Is the index worth deploying QPS, P50/P95 delay, memory Hardware, Batch, Precision, Index Parameters

Open domain question and answer commonly used whether answer String appears in the passage to be hit, but the string rules are affected by case, alias, tokenization and normalization. Pure retrieval data uses Qrels (Query Relevance Labels, query - document relevance labeling). The two cannot be mixed into the same "accuracy". At least BM25, EXACT DENSE SEARCH and approximate Dense Search are reported at the same time. The value of BEIR is to check this migration across tasks and fields, but the average score should still be read in conjunction with data-by-data set results.

Failure Analysis: From Symptoms to False Reasons

Losses go down but Recall doesn't rise. First check whether the subscript of the positive sample corresponds to the diagonal of the score matrix, and then check whether the training and search use the same tokenizer, pooling and normalization. If the TOY set is trained with full marks and the set crashes, it is overfitting evidence, not an index bug.

Documents with near semantics and false facts come in the front. This is usually not difficult enough to expose negative samples. Hard negatives dug up with BM25 or old models, while auditing whether they are actually related documents; otherwise the model will be forced to separate both correct answers.

Exact search is normal, and the FAISS result is reduced. Fixed the same batch of vectors by query to compare top-k IDs, check the distance type, normalization, vector sequence and document ID mapping, and then adjust nprobe, graph search depth or quantitative configuration. The error of the approximate index cannot be counted on the encoder head.

After changing the domain, BM25 overtook. First, it is divided into buckets according to the query type, document length and proper noun ratio. Dense models may learn semantic shortcuts to training domains, while the word plane signal is more stable to numbering, entity names, and terms. Reasonable baseline is to report sparse, dense and hybrid, rather than deleting the "not advanced enough" BM25.

The result cannot be reproduced. Save Query, positive sample ID, negative sample source, model and data hash, the Recall curve of each CheckPoint, and retrieve the output instead of only aggregate fractions. The index must be traced back to the model where it was generated; the model is updated but the index is not rebuilt, and the most hidden version mismatches will be generated.

Questions for further research

  1. Under the fixed training budget, which one is more effective to increase the negative sample in the batch or improve the quality of the hard-negative? The false negative rate should be measured at the same time.
  2. Contriever-style self-supervision enhancement On medical abbreviations, code identifiers, or multi-language text, will the tokens that really determine the relevance are deleted?
  3. How can vector normalization, temperature and embedding dimensions together change the hard-cased local neighborhood, not just average Recall?
  4. How is the drift of the old Passage Embedding detected when the corpus is continuously updated? Can I use the small-scale Exact-Search Sentinel set to trigger a rebuild index?
  5. Are the errors of dense retrieval from representation, annotation, or an ANN index? The Encoder Error needs to be reported separately from the Search Error.

These problems can be extended from the script of this article: increase the false negative sample of the difference between the query, construct the word difference, record the similarity matrix at each step, and then replace it with the real checkpoint. The study value comes from controlling variables and evidence of failure, rather than changing the model names to be bigger.

Summary

The key constraints of the double tower retrieval are very simple: the query and document are independently encoded first, the training is used to shape the space with the comparison loss, the AT Inference time document is built offline, and the query is found online. DPR indicates that there are supervised Q&A and difficult negative samples can train strong Retriever; Contriever indicates that unsupervised comparison pre-training can improve cross-domain starting points. But any conclusion that "dense is better than sparse" depends on data, indicators and versions. The minimum reproduction should first make the score matrix, QRELs, index and log check, and then gradually enlarge the model and corpus.

References

Search date: 2026-08-29.

  1. Karpukhin et al., 2020, Dense Passage Retrieval for Open-Domain Question Answering(EMNLP / ACL Anthology)
  2. Meta AI Research, DPR official code repository (archived)
  3. Izacard et al., 2022, Unsupervised Dense Information Retrieval with Contrastive Learning(TMLR / OpenReview PDF)
  4. Meta AI Research, Contriever official code repository (archived)
  5. Thakur et al., 2021, BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models(NeurIPS Datasets and Benchmarks)
  6. Meta AI Research, FAISS Getting Started