Cover illustration

Date: 2026-08-15
Suitable for readers: graduate students, introductory researcher at LLM pre-training, AI readers with engineering background
Search date: 2026-08-15

Abstract

The core problem of Data-Centric AI is not "the more data the better", but: what training stage, what model scale, and what evaluation objectives are really useful? For LLM research, this problem has changed from empirical engineering to research topics that can read papers, do ablation, and write reproducing reports. DCLM, FineWeb, RedPajama, Dolma/OLMo, Nemotron-CC Data filtering, deduplication, quality score, data mixing, synthetic rewriting and evaluation feedback are gradually disclosed; tools such as WIMBD, Data-Juicer, and NeMo Curator make the corpus audit and processing process more reproducible.

Start with classic papers on deduplication and data selection to understand repetition, contamination, and distribution matching. Next, read open data engineering reports such as DCLM, FineWeb, RedPajama, and Dolma/OLMo to see how corpus processing decisions can be compared experimentally. Then turn to work from 2025 onward, including FineWeb 2, Nemotron-CC, REWIRE, JQL, and DataPrep-Bench. It traces a shift from heuristic filters toward model-based assessment, validation of synthetic data, and prediction of training utility. The goal is to learn to read, dissect, and reproduce the work, then turn a data problem into a research question of your own.

Contents

  1. Why is this topic important
  2. Core reading method
  3. A reading guide to the key papers
  4. Methods and experiments compared
  5. How to reproduce the results
  6. Common misconceptions
  7. Research questions for graduate students
  8. Summary and References

Why is this topic important

The most easily underestimated part of LLM papers is often data. Many model reports will write in detail the number of parameters, optimizers, context length and benchmark scores, but only summarize the training corpus in a few sentences: "Filtered web data, code, math, books, synthetic data". But if you really want to reproduce or improve the model, the data part is usually harder than the architecture part: how to extract the original web page? Which rules have deleted low-quality pages? Repeat text to remove by EXACT HASH, MinHash or semantic dedup? What is the positive and negative samples trained with the mass classifier? Is the filtered data really worthless? Is synthetic data supplement, rewrite or replace natural data? These issues determine the credibility of training costs, generalization ability, pollution risk, privacy risk and the conclusion of the paper.

The changes are especially evident in 2025-2026. DCLM makes "fixed model and training code, only competing for data" into benchmark; FineWeb and FineWeb 2 expose Common Crawl cleaning, deduplication, filtering and multilingual extension processes; Nemotron-CC Organize the quality classification, deduplication and synthetic data generation into a runable Recipe; OLMo 2 continues to emphasize the training data, training code and evaluation of Fully Open; datapre-bench starts to "whether LLM can prepare training data" itself as benchmark. These work instructions, Data-Centric AI is no longer an engineering miscellaneous in the appendix, but a main problem in the LLM research.

For graduate students, the training value of this topic lies in three points. First, it forces you to dismantle the "model" into a verifiable causal factor, rather than credit all the promotions to the new architecture. Second, it is naturally suitable for small-scale reproduction: you do not need to train the 70B model, and you can reproduce the directional conclusion of filtering, deduplication, mixing, and synthesis data on a 100M-1b scale. Third, it produces a lot of reliable follow-up ideas, because open data engineering still has a large number of unresolved problems: cross-language quality score, synthetic data pollution, domain data fidelity, benchmark Leakage, data selection and learning rate Schedule coupling, etc.

Core reading method

When reading the Data-Centric AI paper, it is recommended to draw a "data life cycle line" first, and do not directly look at the final score.

First, what is the original corpus. Record source, time range, language, license, format and scale. Common Crawl, GitHub, arXiv, StackExchange, Wikipedia, books, exam questions, user conversations, and synthetic Q&A represent completely different data distributions. Don't think of "web data" as a homogeneous concept.

Second, what is the processing flow? Split Pipeline into Extraction, Language ID, PII processing, rule filtering, quality classification, Exact Dedup, Fuzzy Dedup, Semantic Dedup, Domain Mixing, tokenization, shuffling, evaluation. Each step has to ask: what is the input, how many outputs are deleted, determine where the signal comes from, and whether the metadata of the deleted sample is retained.

Third, how to define quality. Many papers say high-quality data, but quality may refer to readability, educational value, low repetition, low toxicity, low PII, similar to the target domain, can improve MMLU, can improve code capabilities, or improve security. Quality is not a single scalar; Longpre's Pretrainer's Guide has prompted that there is trade-off between mass filtering, toxicity filtering, and domain coverage, and all tasks cannot be explained with one score.

Fourth, where the evidence comes from. The strongest evidence is not that "filters look reasonable", but Controlled Ablation: the same model, the same tokenizer, the same training budget, the same evaluation protocol, only change the data processing strategy. The value of DCLM is to put data research into the testbed of fixed training code and more than 50 downstream evaluations.

Fifth, whether the reproduction boundary is clear. Data papers must explain code, configuration, random seeds, Common Crawl snaplshot, model size, training tokens, evaluation version and failure examples. If only the final dataset is released and no processing scripts are released, the subsequent research can only be used and cannot really be explained.

Data-Centric AI reading roadmap

A reading guide to the key papers

1. Starting from de-repeating: repetition is not a small problem

The first group of mandatory papers are deduplication Training Data Makes Language Models Better and deduplication Training Data Mitigates priors in Language Models. The former shows that there are near and long duplicate fragments in the common corpus, and deduplication can reduce the probability of the model's silent repeat of the Training text and reduce the train-test overlap The impact on the evaluation; the latter further points out that duplication will increase the risk of Privacy attacks, and the number of Training samples is highly correlated with the probability of being regenerated. When reading these two articles, don't just remember "removing weights", but distinguish three things: the Training set internal repetition, the Training set overlaps with the validation / test set, and the long substring memorization.

The second layer is fuzzy/semantic dedup. FineWeb uses MinHash to handle the near duplication of web pages; DCLM baseline includes cleaning, Bloom filter deduplication and model filtering; SemDeDup extends duplicate similarity from the text surface to semantic similarity, through Embedding, clustering, and similarity thresholds remove redundant samples. The intensive reading question here is: After deleting the repetition, the model has learned less noise, or is some high-frequency facts rare? An important inspiration of FineWeb is that the global de-weighting across all Common Crawl dumps is not necessarily better than de-weight by dump, because excessive de-weighting may change the time and mass distribution. It is not the more aggressive the better.

Reproduction training can start with a small corpus: take 5-10GB web page text, do three versions of Exact Hash, MinHash and Embedding-Based Semantic Dedup; train the same small model or do it Continued pretraining; compare Validation loss, MMLU subset, Memorization Probe, repetition rate and domain coverage changes. This is better understandable than reading "So-and-so dataset is stronger" to understand the cause and effect chain.

2. Data selection and mixing: good data depends on the target distribution

DSIR and DoReMi are two classic routes to understand data selection. DSIR formalizes the data selection to select a subset from a large-scale unlabeled RAW Corpus, making it close to the target distribution in the low-dimensional feature space; DoReMi uses the Excess loss of the Proxy / Reference model. Automatically optimize multi-domain data mixing weights. Together, they explain that data quality is not just deleting low-quality text, but also "make training distribution useful for target tasks".

When reading such papers, it is recommended to establish three levels: document-level selection, domain-level mixing, and schedule-level curriculum. document-level selection cares about whether a single text is reserved; domain-level mixing cares about web, code, math, books, academic, instruction data ratio; curriculum cares the sequence of training phases. 2026 ICLR Work on Curriculum-Based LLM Pretraining also reminds, data quality sequence and learning rate Decay Conflicts may occur: High-quality data may be "wasted" if it always occurs at the low learning rate stage. This kind of conclusion is very suitable for graduate students to do small-scale verification.

3. DCLM, FineWeb, RedPajama: Turn data engineering into experimental science

DCLM is the CORE node of the Data-Centric AI route. It fixed the training code and model scale, allowing researchers to design filtering and mixing strategies from the Common Crawl candidate pool, and then use a unified evaluation to determine whether the data is better. The official page emphasizes "Models are fixed and the challenge is data", which is very important for research training: you can minimize the noise of model innovation and focus on data processing itself. The official DCLM repository also prompted that CORE / EXTENDED calculation had a Baseline bug fix in 2025, so the historical leaderboard number cannot be compared unconditionally; this is also read data benchmark Examples of versions must be recorded.

The value of FineWeb is transparent Web-Scale Pipeline. It starts from Common Crawl and passes through URL Filter, Trafilatura, language filtering, Gopher/C4/FineWeb rule filtering and MinHash deduplication, and exposes a lot of ablation. FineWeb-Edu uses Llama-3-70B-Instruct to rate the sample education quality, and then train the classifier to screen educational texts. When reading FineWeb, you should pay attention to two problems: one is that it defines "quality of education" as a useful filter signal for knowledge and reasoning, rather than a general text aesthetics; the other is its dedup ablation Explain that the data distribution will be profoundly changed by the processing policy.

RedPajama-V2 represents another design philosophy: try to provide large-scale data pools and 40+ quality signals, and let researchers decide the filtering strategy for themselves. Instead of publishing only one "best filter version", it is provided with Metadata, such as natural language indicators, repetition, Perplexity buckets, and quality classifiers. For scientific research, this high-recall data pool is more suitable for data selection experiments, because you can look back at the filtered samples, rather than only accepting the final product.

Dolma/OLMo emphasizes Fully Open Model Flow. Dolma provides mixed corpus and tools for 3T tokens; the OLMo 2 / OLMo 2 32b pages are open for pre-training, mid-training, post-training data stages, such as OLMo-Mix-1124, Dolmino-Mix-1124 and training token information. When reading OLMo data, the key is not to memorize the model score, but to learn how "Model Card + Data Card + Configuration + Checkpoint + Evaluation Suite" to form an auditable research object.

4. 2025-2026 Trend: Quality classification, synthesis rewriting and multilingual

FineWeb 2 extends the idea of FineWeb to 1000+ languages, and adapts to language recognition, deduplication per language, filter threshold for each language, and anonymization of PII. It reminds us that English Web Pipeline cannot be migrated directly to multilingual. Quality scores, repetition rates, script recognition, and filter thresholds may vary in low-resource languages.

Nemotron-CC is a Data engineering case worth reading since 2025. The official information of NVIDIA writes pipeline as Common Crawl -> Extract & Clean -> Deduplicate -> Quality Classify -> Synthetic Data Generation, and provide Recipe in NeMo Curator. Its key proposition is that pure filtering will lose a large amount of medium-quality but convertible Data, so it can be synthesized with quiz, knowledge extraction, rewriting or translation on high-quality Data, and reopen the trade-off between the amount of Data and the Quality. The official blog gives 6.3t tokens, which is about 1.9t Synthetic tokens; these numbers and subsequent Nemotron-CC-v2 / v2.1 The Data set will change, and it must be re-checked before the official reference.

REWIRE (Recycling the Web) asks whether discarded web data can be put to use. Its guided rewriting method turns low-quality pages into text better suited to pretraining. The paper compares results at the 1B, 3B, and 7B scales on the DCLM benchmark. Here, synthetic data comes from a controlled transformation of existing pages. Treating it as a validated supplement also avoids the extreme scenario highlighted in work on model collapse, where synthetic data recursively replaces naturally occurring data.

JQL focuses on multilingual quality filtering: use LLM annotation capabilities to distil up lightweight Multilingual Annotator to reduce large-scale filtering costs. DataPrep-Bench further dismantles LLM-driven data preparation into data construction and data quality Evaluation, emphasizing that "quality" should refer to the downstream training utility, not the surface text attributes. These two types of work instructions, the keyword of the Data-Centric AI in the next stage will be Utility Prediction: whether it is worth training a batch of data before actually training the model.

5. Data Auditing Tool: Don't Just Trust Data Cards

WIMBD is asking the simplest but most important question: what is in my big data? It supports multiple Corpus Analysis with Count and Search, checking repeating, compositing content, low-quality content, PII, toxic language and Benchmark Contamination. Learn an attitude when reading WIMBD: the source description in the data card is not equal to the data reality; there must be a runnable audit tool and sampling checks.

Data-Juicer, NeMo Curator, DataTrove, and Dolma Toolkit are all worth reading as infrastructure for reproduction experiments. They modularize filters, deduplication, quality classification, visualization, distributed processing, and log management. When graduate students do data papers, the tool selection itself is not a contribution, but it is the premise of contribution to the reproduction of pipeline.

Methods and experiments compared

branch representative data core problem Intensive check
Reduction and Privacy Lee et al. 2022, Kandpal et al. 2022, SemDeDup How repetition affects memory, privacy and training efficiency Exact/Fuzzy/Semantic standard, threshold, cut ratio, retention policy
Data selection DSIR, DoReMi, Pretrainer's Guide How to Match Target Distribution and Optimize Mixed Ratio Target distribution, proxy model, domain weights, downstream tasks
Open Data Engineering DCLM, FineWeb, RedPajama, Dolma How to make data processing auditable and reproducible Pipeline, configuration, code, snapshot, metadata, version
Multilingual and Low Resources FineWeb 2, JQL Can English filters be migrated? Language recognition, scripting, thresholds per language, low-resource evaluation
Synthesis and Recovery Nemotron-CC, REWIRE, model collapse When and when is it harmful to synthesize data? Original source, generative model, validator, natural/composition ratio
Quality Assessment WIMBD, DataPrep-Bench How to predict training utility and risk Quality Definition, PII, Pollution, Utility Proxy, Manual Spot Check
Tool system Data-Juicer, NeMo Curator, datatrove, Dolma Toolkit How to run pipeline operator, log, distributed, failed recovery, output audit

How to reproduce the results

In the first stage, a minimum data cleaning pipeline is reproduced. Select a small batch of Common Crawl WET, news pages or open English corpus to record the number of original documents, tokens, language distribution, average length, repetition rate and PII hits. Run language filtering, length filtering, and repeat filtering in sequence, leaving each step removed. The key is not to pursue the optimal filter, but to learn to explain "what has been deleted".

In the second stage, do the deduplication ablation. Construct four versions of RAW, EXACT DEDUP, MinHash DEDUP, and SEMANTIC DEDUP, train a small model with the same tokenizer and the same training script, or do Continued pretraining on a small model. Report Validation loss, a small number of downstream tasks, Memorization Probe, domain coverage and sample diversity. Negative results are also written: If semantic dedup hurts a certain area, check whether the high-frequency but necessary template or term is deleted.

The third stage is to do mass filtering. Use an open source quality classifier, a heuristic rule set, and a LLM Judge small sample labeler to filter data separately, and train the same Token Budget model. Compare the difference between "Small Repeat Training of High Score Data" and "Middle Quality Data for More Unique Tokens". Don't take the MMLU single item as the only conclusion, at least add Perplexity, QA, code or mathematical subsets, and manual sampling.

In the fourth stage, the synthetic data reproduction experiment is done. Choose between 1-20,000 medium and low-quality documents, and use an open source model to rewrite the list of clearer texts, questions and answers, or knowledge points. Faithfulness check must be added: rewrite whether to introduce new facts, whether to lose details, whether to template. Then mix natural high-quality data, natural medium-quality data, and synthesized rewriting data in different proportions to test whether there is a upper limit of income.

In the fifth stage, write reproduction report. The report must include: data source and retrieval date, processing script commit, filtering configuration, random seed, ratio of each step reserve, sample audit screenshot or fragment, training budget, evaluation protocol, failure case and "to manually verify" item. The Data-Centric AI reproduction report is most afraid of only giving final scores; what is really valuable is to let others understand how the data changes.

Common misconceptions

Misunderstanding 1, quality is equivalent to readability. Readable text is not necessarily useful for target tasks; low-readable code, formulas, forum quiz, logs, and tables may be important for specific abilities.

Misunderstanding 2, regard de-weighting as a fixed pre-processing. Deduplication granularity, threshold and time range change the corpus distribution. Excessive deduplication may delete reasonable duplicate teaching materials, and too weak to de-weight will amplify memory and pollution.

Misunderstanding 3, treat synthetic data as free data. Synthetic data requires source, generation strategy, deduplication, fact verification and proportional control. Nature 2024's Model Collapse paper warns of recursive training and indiscriminate use to generate content; it does not deny verified synthetic data, but requires us to record the boundaries of real data and synthetic data.

Misunderstanding 4, only look at the final benchmark. Data processing is often opposite to different tasks: more educational may improve knowledge reasoning, but damage natural dialogue or long tail styles; toxicity filtering may affect the task of toxicity identification; code filtering may affect natural language ability. Must see slices.

Misunderstanding 5, ignore version drift. DCLM LeaderBoard, Hugging Face Dataset Cards, Nemotron Recipe, Common Crawl Snapshots, quality classifiers and synthesis models are all likely to be updated. When writing the paper, the search date should be specified, and the changeable numbers should be re-checked.

Research questions for graduate students

  1. Dedup Scaling Law on Small Models: Compare 100M, 300M, 1B EXACT / Fuzzy / Semantic Dedup The gain curve of the model to study whether the model scale changes the optimum deduplication intensity.
  2. Multilingual quality score migration: Migrate English quality classifiers to low-resource languages, compare direct migration, LLM labeling distillation, and a small amount of manual labeling calibration.
  3. Synthetic rewrite of the fact-fidelity evaluation: Design a document-level Faithfulness Benchmark to measure whether Rewrite retains entities, values, references, and causal relationships.
  4. Data filtering and learning rate Schedule: Validate the difference between high-quality data in the early, mid-term, and Annealing stages of training, and reproduce the coupling problems of curriculum and LR decay.
  5. Data Audit Report Template: Write WIMBD-style Audit for an open corpus, including duplicate, PII, pollution, domain distribution, synthetic traces, and sample error types.
  6. Utility Predictor: Train a lightweight model to predict "whether a batch of data is used to improve certain tasks after Continued pretraining" and compare it with heuristic quality scores.

Summary

The research training of Data-Centric AI, most importantly, is to turn "data feel better" into reproducible evidence. You need to read the pipeline, version, filter, deduplication algorithm, quality definition, synthesis strategy and evaluation feedback; you also need to learn to ask about the deleted data, the retained data, the repeated training data and the generated data.

If the main line of LLM research in the past few years is scale, architecture and instruction alignment, then the more clear point in 2025-2026 is that the data itself has become the experimental object of model capabilities. Graduate students do not have to have huge computing power from the beginning. As long as you can make clear data ablation, honest audit report and reproducible training log, you can ask valuable questions.

References

Search date: 2026-08-15. The following links give priority to first-hand data such as paper pages, official project pages, official documents, GitHub, Hugging Face data cards, ACL Anthology, pmr, OpenReview, Nature and other first-hand information. Models, datasets, benchmark and repository status may be updated, please re-check before submitting or publishing.

  1. DataComp-LM official site: https://www.datacomp.ai/dclm/
  2. DataComp-LM paper page, arXiv 2406.11794: https://huggingface.co/papers/2406.11794
  3. DCLM GitHub repository: https://github.com/mlfoundations/dclm
  4. DCLM-Baseline Hugging Face dataset card: https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0
  5. FineWeb dataset card: https://huggingface.co/datasets/HuggingFaceFW/fineweb
  6. FineWeb paper page, arXiv 2406.17557: https://huggingface.co/papers/2406.17557
  7. FineWeb blog / technical report: https://huggingfacefw-blogpost-fineweb-v1.static.hf.space/index.html
  8. FineWeb 2 GitHub README: https://github.com/huggingface/fineweb-2/blob/main/README.md
  9. RedPajama-V2 announcement / documentation: https://secretive-tarsal-fe8.notion.site/RedPajama-Data-v2-An-open-dataset-with-30-trillion-tokens-for-training-large-language-models-27689a07b1d24f49b3d16743d63f2db0
  10. Together AI RedPajama-V2 FAQ: https://www.together.ai/blog/redpajama-v2-faq
  11. Dolma official documentation: https://allenai.github.io/dolma/
  12. OLMo 2 official page: https://allenai.org/olmo2
  13. OLMo 2 blog: https://allenai.org/blog/olmo2
  14. OLMo release notes: https://docs.allenai.org/release_notes/olmo-release-notes
  15. Deduplicating Training Data Makes Language Models Better, ACL Anthology: https://aclanthology.org/2022.acl-long.577/
  16. Deduplicating Training Data Mitigates Privacy Risks in Language Models, PMLR: https://proceedings.mlr.press/v162/kandpal22a
  17. SemDeDup GitHub repository: https://github.com/facebookresearch/SemDeDup/
  18. Data Selection for Language Models via Importance Resampling, arXiv 2302.03169: https://huggingface.co/papers/2302.03169
  19. DSIR GitHub repository: https://github.com/p-lambda/dsir
  20. DoReMi blog and implementation: https://crfm.stanford.edu/2023/09/14/doremi.html
  21. DoReMi GitHub repository: https://github.com/sangmichaelxie/doremi
  22. A Pretrainer's Guide to Training Data, arXiv 2305.13169: https://arxiv.org/abs/2305.13169
  23. WIMBD / What's In My Big Data?, OpenReview: https://openreview.net/forum?id=aydSC2ewu0
  24. Data-Juicer GitHub repository: https://github.com/datajuicer/data-juicer
  25. Data-Juicer paper page: https://www.alphaxiv.org/abs/2309.02033
  26. Nemotron-CC data curation docs: https://docs.nvidia.com/nemotron/nightly/nemotron/data/curation/nemotron-cc.html
  27. NVIDIA Nemotron-CC announcement: https://developer.nvidia.com/blog/?p=94818
  28. NVIDIA Nemotron-CC pipeline blog: https://developer.nvidia.com/blog/?p=99540
  29. NVIDIA Nemotron GitHub / dataset hub: https://github.com/NVIDIA-NeMo/Nemotron
  30. NeMo Curator GitHub repository: https://github.com/NVIDIA-NeMo/Curator
  31. Recycling the Web / REWIRE paper page, arXiv 2506.04689: https://huggingface.co/papers/2506.04689
  32. AI models collapse when trained on recursively generated data, Nature: https://www.nature.com/articles/s41586-024-07566-y
  33. Collapse or Thrive, ICML 2025 PMLR: https://proceedings.mlr.press/v267/kazdan25a.html
  34. Judging Quality Across Languages, arXiv 2505.22232: https://www.alphaxiv.org/abs/2505.22232
  35. DataPrep-Bench paper page, arXiv 2607.20465: https://huggingface.co/papers/2607.20465
  36. Data-Centric Lessons To Improve Speech-Language Pretraining, OpenReview ICLR 2026: https://openreview.net/forum?id=4amNkYCDqX
  37. SmolLM2 paper page, arXiv 2502.02737: https://arxiv.org/abs/2502.02737
  38. How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining, ML Anthology / ICLR 2026: https://mlanthology.org/iclr/2026/luo2026iclr-learning/