Cross-Encoder Rearrangement Cover

Abstract

Retrieval systems often narrow a collection of a million documents to a few dozen to over a thousand candidates before applying a finer-grained reranker. A cross-encoder joins the query and passage in one sequence so their tokens interact directly at every layer. This usually gives more nuanced scores than a dual encoder, at the cost of a fresh forward pass for every candidate. Following monoBERT and the official Sentence Transformers implementation, we reproduce a minimal protocol from candidate retrieval through joint scoring and reranking to MRR/NDCG. We examine hard negatives separately: candidates that closely resemble the query but are in fact irrelevant. The accompanying script checks the data, loss, rankings, and recall ceiling, without reproducing MS MARCO scores.

Reproducing Value and Goal: Reranker is not the second Retriever

Twin Towers precoding documents as vectors, suitable for large-scale recalls; Cross-Encoder must see a pair of text at the same time, suitable for fine-scheduling in smaller candidate sets. The most misunderstood thing is that Reranker can only change the order of candidates.

This recurrence revolves around four checkable questions: whether the input is a pair of text or two independent vectors; how to sort Loss in a point type two classification; How NDCG should be reported separately. We do not download the large model, but use the white box linear Pair Scorer to simulate the interface of the Cross-Encoder so that we can see every step on the CPU.

Core Idea: Joint Coding, Scalar Scores and Training Objectives

1. Query and Passage merge before entering the model

On query qq and candidate paragraphs did_i, BERT-style Reranker construct:

xi=[CLS]  q  [SEP]  di  [SEP]x_i=[\mathrm{CLS}]\;q\;[\mathrm{SEP}]\;d_i\;[\mathrm{SEP}]

If the batch has BB For text, truncated length is LL, the input id and the attention mask are both [B,L][B,L]. Transformer outputs the last layer of hidden state Hi∈RL×hH_i\in\mathbb{R}^{L\times h}, and then from [CLS] Location hi∈Rh\mathbf{h}_i\in\mathbb{R}^{h}, the via linear header gets a scalar logit:

si=w⊤hi+bs_i=\mathbf{w}^{\top}\mathbf{h}_i+b

Among hh is a hidden dimension, sis_i Not probability; available when needed σ(si)\sigma(s_i) turn into [0,1][0,1]. The Sentence Transformers document specifically reminds that the MS MARCO series model returns Logits by default, but sigmoid is a monotone function, so the sorting under the same query will not be changed.

The essential difference from the two towers is not "one more layer of MLP", but the query token and the password token can affect each other layer by layer through self-attention. The cost is also very direct: if each query is reranked KK Candidate, you need to calculate KK For each union sequence, the results on the document side cannot be offline as double-tower embedding.

2. Point BCE is sufficient to form a strong baseline

give related tags yi∈{0,1}y_i\in\{0,1\}, the binary cross entropy can be written as:

LBCE=−1B∑i[yilog⁡σ(si)+(1−yi)log⁡(1−σ(si))]\mathcal{L}_{\mathrm{BCE}}=-\frac{1}{B}\sum_i [y_i\log\sigma(s_i)+(1-y_i)\log(1-\sigma(s_i))]

monoBERT's original paper regards the relevant and irrelevant passages as a two-class sample, and scores one by one on the BM25 recall candidates. The current official example of Sentence Transformers still uses BCE as a strong baseline, and also provides RankNet, ListNet, ListMLE and LambdaLoss for NDCG. The paper reports the results of specific data and versions; what this paper actually verifies is whether the BCE can separate the positive and negative pairs, and does not infer that some kind of loss is generally better.

3. Hard Negative teaches the model to identify the difference between "relevant topics" and "answer questions"

Random negative samples usually have nothing to do with query, and the model quickly learns topic classification, but it may not be able to distinguish close neighbors in the same topic. For example, the question is "Where was Mrs. Curie born", the negative sample "Pierre Curie was born in Paris" contains the characters' surnames, birth relationships and locations, the surface is highly matched but the answer is physically wrong. Such candidates are difficult negative samples.

Difficult negative samples are usually dug up by BM25 or double tower search, and then excluded and filtered. If you only believe in sparse Qrels, the unlabeled real relevant paragraphs will be false as negative examples, forming false negative (false negative samples). RocketQA clarifies this problem: use cross-encoder Confidence to de-noise, and then train Dense Retriever. Current Sentence Transformers mine_hard_negatives It also exposes the ranking interval, absolute/relative margin, maximum score and sampling strategy. The key here is not "the harder the better", but the difficulty and label credibility are controlled at the same time.

Cross-Encoder and difficult negative sample method diagram

Guide to the official implementation: first follow the data contract, then follow the model class

The first stop is to read Nogueira and Cho's paper Section 2: It is limited to the second stage Reranking, and query is used as Sentence A and Passage as Sentence B, as well as [CLS] Classification probability sorting. Then see the official dl4marco-bert Repository: convert_msmarco_to_tfrecord.py Curing Query, Passage, Label and maximum length, run_msmarco.py Responsible for training and output RUN files, metrics.py Implement evaluation. Repository README uses the BM25 TOP-1000 to develop candidates; this candidate source must enter the experimental record.

Look at the current code of Sentence Transformers at the second stop. examples/cross_encoder/training/ms_marco/training_ms_marco_bce.py Put MS MARCO's query、passage_text、is_selected Map as a pair of text with 0/1 tags, and then give them to BinaryCrossEntropyLoss. If you study the sorting target, go to the Lambda, ListNet, and RankNet scripts in the same directory, check whether the input is pair or the document list of each query, and you cannot just replace the loss name.

third stop reading sentence_transformers/util/hard_negatives.py. The point is not to call functions, but to audit range_min/range_max, margin, positive case exclusion, output format and Miner version. One mining should save Query ID, Candidate ID, Miner score, Qrels determination and filtering reasons, otherwise it is difficult to locate the false negative sample.

Minimum Experiment: White Box Pair Scorer Validation Protocol

run:

python3 code/minimal_reranker.py --check-only

The script fixes random seeds 58 and constructs 4 queries; each query contains a positive case, a irrelevant Easy Negative and a Hard Negative with a similar topic but a wrong answer. The feature tensor shape is printed as [batch, vocab+2]: This is just an interpretable joint pair feature that simulates the interface of "a pair of text → a scalar", which does not contain Transformer/self-attention.

We trained the same BCE scorer twice. One run used positives and random negatives only. The other also used hard negatives. We deliberately placed a hard negative ahead of the positive in the initial candidate ordering. We then removed the positive for the last query and reranked again. Assertions verified that once candidate Recall@3 fell from 1.0 to 0.75, reranking could not restore it. These outputs come from the CPU smoke test. Its tiny vocabulary, query set, and candidate pool establish how the protocol works and where it fails, without supporting claims about monoBERT, MS MARCO, or real-world generalization.

When replacing TOY with a real model, the minimum upgrade route is: fixed a public checkpoint; freeze BM25 or double tower candidate RUN; save tokenization and truncation statistics; call CrossEncoder in batches; output qid, pid, score, rank; Use the same Qrels to report candidates Recall@k, reranking MRR@10 and NDCG@10 at the same time. For training upgrades, human audits should be sampled manually first, and then random-only, hard-only and mixed sampling should be compared.

Reappear in three layers, avoid mistakenly write "can run" as "reproduction papers"

The first layer is the reproduction of the protocol, that is, the work completed by the script in this article: confirm that each sample contains query, passage, label, confirm that a pair only generates one score, and confirm that the sorting indicator is querying by query. Aggregate, and test the recall upper limit with missing positive examples. It does not need to pre-train the model, and is suitable for basic errors such as ID dislocation, sorting reversed, and missed query being skipped.

The second layer is CheckPoint inference reproduction. Select a clear version of open source Cross-Encoder, and only do reasoning on a fixed candidate run; at the same time, run the control of "maintain the original order of the first stage". The research question here is whether the coding really improves the same batch of candidates, not Retriever, the number of candidates or Qrels has changed. The model commit, tokenizer file hash, maximum length, inference accuracy and number of pairs should be recorded. If the official model card reports the MS MARCO score, but uses different candidates or evaluation scripts locally, it can only be called a migration verification, and cannot be called to reproduce the number.

The third layer is the training reproduction. After the Query set is fixed, the random-only data will be formed, and then the same Miner will be used to generate two sets of data of hard-only and mixed; Each experiment runs at least multiple random seeds, reports mean and discrete degrees, and saves the optimal checkpoint selection rule. If the best one is posted, the sampling fluctuation and selection deviation will be mistakenly written as a method benefit.

Hard-negative files must be auditable

It is recommended to save the mining result as a pair of JSONLs in a row, at least contain qid、positive_pid、negative_pid、miner、miner_rank、miner_score、qrel_label、filter_reason and corpus versions. The training loader then generates a pair or list from this immutable list, instead of re-dig each epoch online. In this way, once the result decreases, it can be judged whether the Miner has changed, the filter threshold has changed, or the model training itself is unstable.

Manual audits do not have to cover all data. You can sample layered from RANK 1–10, 11–50 and 51–200, and mark negative examples as "clearly irrelevant", "relevant but not answered", "another correct answer" and "unjudgmentable". The first two categories are negative examples of trusted training; the latter category should be excluded or retained with soft labels. If different annotators are frequently different, it means that the task definition or Qrels are insufficient, rather than simply raising margins can be solved.

Evaluation protocol: disassemble the recall cap, sorting quality and cost

MRR (Mean Reciprocal Rank) takes the inverse of the first related document ranking and re-appoints:

MRR=1∣Q∣∑q∈Q1rankq\mathrm{MRR}=\frac{1}{|Q|}\sum_{q\in Q}\frac{1}{\mathrm{rank}_q}

If there is no relevant document in a query candidate, the item must be 0, not deleted from the denominator. NDCG@K allows grading related tags and loses profits by position; only binary and when each Query is single positive, it is highly relevant to the first positive case ranking. Candidate Recall@k answers "Whether the positive case has entered the reranking pool". Therefore, it should be reported simultaneously retriever → candidate Recall and reranker → MRR/NDCG, and fix the candidate set comparison model.

Also record throughput, P50/P95 delay, average token number, truncation rate and candidates per query. It is possible to increase the recall from 50 to 100 to increase the recall, but approximately double the number of joint forwards; if different models use different k, the final score cannot be directly attributed to reranker . The dynamic list, managed checkpoint and framework interface may change, and the versions that are not run in this article are all manually verified.

In order to locate changes, the query log is more important than the single average. Each query should keep TOP-K before and after reranking, related labels, score difference and truncation length, and then divide the bucket according to the length of the problem, the density of the entity, the overlap of the word surface, whether it contains numbers, and whether there are multiple answers. If the average MRR rise only comes from queries that are highly overlapping words, and the proper nouns or long document buckets fall, the conclusion should be written as "improvement of a specific subset", rather than "reranker understands more semantics as a whole".

The boundary of the evidence in this paper should also be clear: the paper and the official code support the structure of the Cross-Encoder, the training form and public implementation path of the monoBERT; this actual operation only supports the toy Pair Scorer's Loss, sorting and candidate upper limit assertions; regarding the proportion of fake negative samples in real corpus, the pros and cons of different loss and online delay, are all problems to be tested. The author infers that the hard-negative inventory and candidate runs can be audited to be audited, which is usually more reliable than blindly enlarging the model.

How to diagnose failures

The training loss declines, and the MRR does not rise. First check whether the batch is grouped by query, whether the label and password id are misplaced, and then see if the negative sample is too easy. Accuracy high on random negative samples does not mean that Retriever Top-K can be reranked.

After joining Hard Negatives, it declined in full. The highest negative case of the sample re-bid, and the negative rate is counted; the bucket is divided according to the Miner and Rank intervals and the query type. If you only use extremely difficult samples, the model may lower the reasonable semantic similarity; mix in random or medium difficulty negative samples, and use margin to avoid suspicious terms that are too close to the positive case.

After rearrangement, the TOP-1 becomes better, and the NDCG@10 becomes worse. Check whether the point-style BCE only learns the local rule of "the most positive case"; if the task has multiple levels of correlation, you can compare the listwise/pairwise target on the same candidate set. However, you must keep Miner, Batch, Qrels and evaluation scripts unchanged, and avoid mistakenly conceived of data changes as loss improvements.

Long PASSGE is abnormal. Save the length of the token before and after the truncation, and confirm that the length of the reserved query and the truncation direction of the truncation are confirmed. When the key information is at the end, the upper limit of the 512 token may cut off the evidence; this is not a problem that sort loss can be fixed.

Offline is good, slow or drifting online. Check online candidates K, maximum length, padding, mixed precision, batching and sigmoid settings. SIGMOID does not change the sorting, but the threshold, calibration and cross-query score comparison will be affected; the model score cannot be verified as probability.

Questions for further research

  1. With the labeling budget fixed, which improves cross-domain NDCG more: having humans relabel top-ranked hard negatives or expanding the pool of random negatives?
  2. Can the false negative probability of hard-negative be estimated with multi-datorgic evidence, instead of binary deletion?
  3. The benefits of PointWise, Pairwise, and ListWise loss come from the target itself, or from different data organization and sampling?
  4. What is the cost of the candidate K, the sequence length and the size of the model - recall - the frontier?
  5. Does the single positive MRR cover up the coverage and diversity failures when query has multiple intentions or multiple correct paragraphs?

These questions require the candidate RUN and query logs to be reserved. There is no replayable candidate, and the researchers cannot judge the income from the first stage, the negative sample of difficulties, the joint coding or the evaluation caliber.

Summary

A cross-encoder scores queries and passages through joint token interactions, at the cost of recomputing each candidate. Binary cross-entropy (BCE) is a dependable starting point. Hard negatives shape whether the model learns fine-grained relevance, but aggressive mining or incomplete labels can turn relevant passages into false negatives. For a credible replication, freeze the candidate set, report candidate recall separately from reranked MRR/NDCG, and save the sources of negatives and the truncation statistics. Reranking cannot recover documents that retrieval missed. Establishing the protocol before scaling up the model makes failures more informative than a single leaderboard score.

References

Search date: 2026-08-30. The following are the original paper, official project page, official code or official framework document.

  1. Nogueira & Cho, 2019, Passage Re-ranking with BERT
  2. NYU DL, DL4MARCO-BERT Official Restoration Repository
  3. Nguyen et al., 2016, MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
  4. Microsoft, MS MARCO official project page
  5. Sentence Transformers, Cross-Encoder Usage official documentation
  6. Sentence Transformers, MS MARCO Cross-Encoder training example
  7. Sentence Transformers, mine_hard_negatives Official source code
  8. Qu et al., 2021, RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering(NAACL / ACL Anthology)
  9. PaddlePaddle, RocketQA official code repository