
- Date: 2026-09-05
- For graduate students, early-career researchers and research engineers
Abstract
Deleting many lines with one corpus means that the experiment is more credible? Not necessarily. The repetition can be the exact same document, a copied paragraph, or it can be the same question that has been changed; the high similarity may also come from a sentence with the opposite conclusion. This article focuses on an executable goal: build an audit script that retains word-face evidence, vector scores and data sources, observe how to miss and rewrite, and how to accidentally injure negative sentences. The minimum experiment does not train the language model, and the actual verification is the candidate discovery and isolation protocol; the memory, training efficiency and evaluation in the paper are improved, and it is not used as the result of this reproduction.
Reproduction value: first distinguish "delete duplication" and "check pollution"
Deduplication studies are the redundancy of the Training Set combined: repeating the sample will change the sampling weight, so that some expressions are repeatedly learned. Contamination detection studies the overlap between training materials and evaluation questions, which is related to whether the test can still measure generalization. The training set has been de-weighted, and a complete test question may also be retained; in turn, the text on the same topic does not automatically cause the leakage of the question.
The paper report: Lee et al. also examined the overlap of near-repetitive documents, long-repeat substrings and training verification in the study of ACL 2022, and observed a reduction in memory output after deduplication. This conclusion depends on its data and training settings, and cannot be deduced as "the more data is deleted, the better". This article defines the output as a back-reviewable text pair and processing list, rather than a denominator cleaning rate. Papers and Conference Versions
Contamination evidence also needs to be layered: find the same fragment, prove that there is overlap in the visible corpus; find a rewritten question, support further verification of the source; prove that a model has actually used the material, and need a training list, snapshot or controlled experiment. Just relying on the correct answer, low confusion, or search similarity, you can't cross the last step.
Core ideas and formulas: three scales, two directions
The first layer is the exact match after normalization. The script uses Unicode NFKC normalization, case folding and tokens to extract, and then calculates SHA256; the hash is still relatively normalized text after hitting. This can eliminate the case and punctuation differences in this example, but the code, chemical formula, and digital precision may depend on these differences, and the formal data cannot copy the same rule. The original text and the transform version must be preserved at the same time.
The second layer is n-gram, that is, the fragments of continuous n tokens. Let the clip collection of documents be , define:
It is a symmetrical Jaccard similarity, suitable for comparing full-text overlap ratios; It is to query the directional coverage of the document, which is suitable for finding short questions to be embedded in long texts. Here is the benchmark text, is a training document. When the denominator is empty, the script returns zero and tests the boundary of the short text individually. The two empty sets cannot be judged as duplicates. The three-word fragment of the experiment is taken only to make the short case observable; the Chinese should be qualitative or tokenization protocol.
The third layer is Embedding, that is, the encoder maps the text into a vector. If unitized, it is expressed as , then shape of , element is the cosine similarity. represent the number of samples, Indicates dimension. Candidate discovery only checks the upper triangle to avoid self-matching and repeated counting; cross-collection matrix is . A vector proximity means that the encoder is considered similar, and it does not necessarily mean that the two paragraphs can be replaced by each other.

The corpus and benchmark in the figure are the same normalization, but always retain their own Split identification; merging does not mean adding benchmarks to training. The word surface and the vector branch produce candidates, and after review, it is decided to keep or isolate. The benchmark dungeon remains read-only, and the isolated list is for the material to be trained.
Guide to the official implementation: From Input Contract to Delete Conditions
Read Section 4 of Lee et al. first, and then look at the ExactSubstr route of the official repository: scripts/make_suffix_array.py Scheduling index build, src/table.rs handle suffix arrays, src/main.rs Provide repeated inquiries. The suffix array sorts the sequences that start at different locations to facilitate the discovery of long common fragments; the focus of reading is how the byte locations are mapped back to the document boundary. The repository has released the ExactSubstr implementation and the NearDup document cluster, and the Jaccard of this article cannot be regarded as the original NearDup. official implementation
Then read the representation of SemDeDup for extraction, clustering, and intra-cluster filtering. compute_pretrained_embeddings.py The normalized representation is written to the memory map file according to the global index; semdedup.py Calculate the similarity matrix in the cluster, and then find the maximum value of each column along the upper triangle. The current source code is M > 1 - eps The mark is deleted, so when the score is fixed, increasing the EPS will expand the delete collection. README's description of the parameter direction is inconsistent with this condition, and the reproduction should be confirmed with the specific source code and small example, and cannot be guessed by the parameter name. Sorting of the representative samples also affects who is retained. Indicates the extraction source code、Filter source code
Paper report: SemDeDup uses pre-training to discover semantic redundancy, and studies data efficiency in its graphic and language experiments; it is not a standard that automatically determines that all synonymous texts should be deleted. Large corpus narrows the comparison by clustering, and may also miss the repeat pairs on both sides of the cluster boundary. SemDeDup v3
Finally, read Yang et al. rewrite the pollution paper and LLM Decontaminator. The description of the paper description and translation can bypass the string detection; the official entrance requires the training set and the test set to be converted to contain text JSONL, then retrieve the candidate and review. Tracking while reading top_k: If there is no candidate to enter the candidate pair, no matter how strong the follow-up judger is, it cannot be retrieved. Paper V2、Official repository
Minimum Experiment: Let misses and false positives appear at the same time
with script code/minimal_dedup.py We constructed six training texts and three benchmark texts. The training set contains an original sentence, a variant with different case and punctuation, a paraphrase, a negated version with “never” inserted, an unrelated sentence, and a longer passage containing the original. Only three pairs are labeled positive for whole-text equivalence. A long passage containing the short sentence does not qualify, giving whole-text deduplication and cross-set coverage distinct criteria.
The default vector comes from the five groups of handwritten synonyms mapped by the bag of words, which is by no means a pre-trained model. It deliberately retains an interpretable flaw: Never only adds one dimension, and the sentence conclusion is still close, although the sentence conclusion is reversed. The fixed random seed is 64, the vocabulary is determined, and both the input and script are hashes. Execute from the article directory:
PYTHONPYCACHEPREFIX=/private/tmp/codex-ai-064-pycache python3 -m py_compile code/minimal_dedup.py
python3 code/minimal_dedup.py
What we tested: the CPU of Python 3.9.6 runs through, and the vector shape is [9,26], comparing 15 pairs within the training set and 18 across sets. Exact matching and three-word-shingle Jaccard similarity at a threshold of 0.5 each yielded 1 true positive, 0 false positives and 2 false negatives. Bag-of-words cosine similarity at a threshold of 0.9 yielded 3 true positives and 3 false positives, for precision 0.5 and recall 1.0. The three false positives paired a negated sentence with each of three equivalent formulations.
A cross-set coverage threshold of 0.8 produces four pairs of candidates, where the original sentence has a coverage ratio of 1 in the long text, and the full text Jaccard is below 0.5. The script only automatically isolates the normalized and repeat T1, and retains the negative sentence T3; the rest of the approximate relationship is left for review. All numbers, threshold scans and inputs are saved in code/results.json, see the real output code/smoke_test.txt. This set of data is constructed by the author, there is no independent test set, and the numbers only describe the behavior of the program.
Optional branch can load a local SentenceTransformer snapshot, see the dependency instructions code/README.md. This environment lacks relevant dependencies, and the real model reasoning has not been run, and it is manually verified. Candidate model all-MiniLM-L6-v2 is English and 384 dimensions. The model card indicates that the default truncation of the input exceeds 256 Word Pieces; it cannot directly undertake the Chinese long-language experiment. The default bag score cannot be written as the result of the model. Model card
Evaluation protocol: denominator, threshold and budget are fixed
First define the labeling unit. The whole article, partial copying, topic rewriting, and general template sharing should be counted separately, and it should be added to the difficult example of entity, value, negation and conditional change. Training deduplication reports paired accuracy and recall, number of reserved documents and tokens, and reserve ratio of each source; the number of benchmark questions hit at least once hit by the polluted audit report is divided by the number of benchmarks, and the number of texts cannot be used as the ratio of the questions.
Next, open the search and judgment. Fixed candidate budget, test candidate Recall, and then measure Accuracy ; after selecting the threshold for the development set, and then evaluate it after freezing. MinHash uses random signatures to approximate the similarity of the set, and locally sensitive hashes are used for recall candidates, and neither replaces the original text verification. Fixed the signature seeds, the bucket method and the upper limit of the candidates, and then compared with the small-scale exhaustion, can we know whether the missed detection comes from the index or the judgment.
At the execution level, an independent pollution registration form is also required: each record keeps the reference number, the training document number, the original matching segment, the character offset, the transformation rule, the candidate score, the source time and the review conclusion. If only the deleted data is saved, the next time the tokenizer is updated, it cannot explain why the sample disappears. The reversible isolation list is first generated, the selection and source ratio of the representatives are checked, and then a new training snapshot is derived; the benchmark questions cannot be written back to reduce the overlap rate.
Data segmentation should also consider repeating families first. If the rewrite version is randomly assigned to training and verification, and then only deduplicated within each, the leak still exists. It can be divided into groups of verified families or original sources, and additional external benchmarks can be audited. The family here comes from a reliable source relationship or manual confirmation, and cannot be directly replaced with a similar map connected component. The manual random inspection should record the random seeds and sampling probability, and the high-risk candidate ratio drawn by the stratification cannot be directly regarded as the overall pollution rate.
Compare model performance under two protocols: a fixed training-token budget and a fixed number of passes through the data. Training may finish sooner after examples are removed simply because there are fewer updates. Elapsed time alone cannot distinguish better efficiency from less training. For the subset with no detected match, report scores, sample count, and difficulty distribution separately. Removing easy questions can change the score without establishing an effect of contamination. When the model version and training snapshot are unavailable, limit the claim to “no match detected in the audited corpus.”
Failure investigation: similar graphs are not equivalent relationships
If the short questions in the long text are omitted, check the denominator, cut and cut; if the Chinese recall crashes, check the tokenization unit and multi-language representation; if the template causes false positives, check whether the public disclaimer dominates the set of fragments. If the negative example is deleted, it is preferred to look back at whether the negative words, numbers and conditions are normalized and erased. The original offset and the transform log are more diagnostically valuable than a single score.
Another type of hidden error is the merge of connected components. The script structure directions are three unit vectors of zero, twenty, and forty degrees respectively; at the threshold value of 0.9, the two adjacent pairs have edges, but the head and tail are not similar enough. If only one connected component is left, the endpoint that is not similar to the representative may be deleted. The assertion has run through. The formal process can check the similarity of the member to the representative, and randomly check the length of the chain, and cannot mistake the transmission closure as a semantic equivalent certificate.
Questions for further research
The authors infer that it is worth continuing to study not to find a general deletion threshold, but to preserve information diversity under a limited review budget. The different encoders can compare the negation and the mistakes of rare entities by different encoders during the fixed retention rate; compare the influence of random retention, central samples and difficult samples on downstream generalization; study candidate recalls for cross-language rewriting pollution. Each problem should first define independent annotation set, source layering and failure criterion, and then start training.
Summary
Reliable deduplication depends on keeping track of the evidence. Lexical methods locate overlapping text, vector methods broaden the pool of candidate matches, and human or controlled review determines the semantic relationship. Provenance helps establish whether that relationship bears on training contamination. We reproduced four edge cases: missed paraphrases, misclassified negations, overlap between long and short texts, and chains of similar examples. The logs can be replayed. The next step is to run the protocol on an authorized real corpus and verify candidate quality before evaluating training gains.
References
Search date: 2026-09-05. The above links are actually opened. The branch source code is recorded with the visit date, and the official is not claimed to be fixed;
- Katherine Lee, Daphne Iippolito et al., 2022, Deduplicating Training Data Makes Language Models Better, ACL Finalized Full Text.
- Amro Abbas、Kushal Tirumala、Dániel Simig、Surya Ganguli、Ari S. Morcos,2023, SemDeDup: Data-efficient learning at web-scale through semantic deduplication, Paper、official code.
- Shuo Yang、Wei-Lin Chiang、Lianmin Zheng、Joseph E. Gonzalez、Ion Stoica,2023, Rethinking Benchmark and Contamination for Language Models with Rephrased Samples, Paper.
- Sentence Transformers, all-MiniLM-L6-v2 model card.