Cover illustration

Date:2026-07-05
Who this is for: Graduate students, engineers who do model training/evaluation, technical readers who are concerned about the open source LLM Recipe
search date:2026-07-05

Abstract

From 2025 to 2026, synthetic data grew from a stopgap for missing labels into a central resource for post-training. In classical knowledge distillation, outputs from a large model or an ensemble train a smaller, cheaper student. LLMs extend that idea into a more complex cycle. Production logs, human preferences, public problem sets, tool execution results, and model-generated trajectories become data for SFT, DPO, RLHF/RLAIF, RLVR, rejection sampling, verifier filtering, and distillation. The resulting data feeds successive generations of models.

The core judgment of this article is that the real value of synthetic data is not "infinite generation", but "verifiable, traceable, iterable". DeepSeek-R1, Tulu 3, Llama 4, Qwen3, Magpie, UltraFeedback, OpenThoughts The work has shown that cutting-edge capabilities are increasingly dependent on the data pipeline itself: who can define task distribution, structure difficult cases, filter low-quality samples, control pollution, and feed back the evaluation to training, who can turn a model release into a continuous improvement engineering system.

Contents

  1. Research background: Why post-training begins reorganization around the data flywheel
  2. Roadmap for the past year: from Instruction Data to Reasoning Data
  3. Key papers by research question
  4. Comparing the methods
  5. Technical trends
  6. Implications for deployment
  7. Limitations and open questions
  8. Questions to watch
  9. Summary
  10. References

1. Research background: Why post-training begins to reorganize around the data flywheel

The pre-training stage is to "press the world into the parameters", and post-training solves "let the model work according to people and tasks". After GPT-3, InstructGPT, and ChatGPT, the familiar process is roughly: first use manual demonstration to do supervised fine-tuning, then collect preference data to train reward model, and finally use PPO and other reinforcement learning optimization strategies. This paradigm proves an important fact: the scale of the model is not automatically equal to the available products, and a large part of the behavioral alignment and task abilities come from trained data and feedback.

This is the bottleneck of synthetic data. Manually write prompt, write high-quality answers, standard Preference, standard safety boundaries, high cost, slow speed, and limited coverage. The contribution of Self-Instruct is to make "generate instructions, generate input and output, filter, and then fine-tuning" into an automatic process; UltraFeedback replaces human feedback with large-model feedback, emphasizing scale and Diversity; Magpie further uses the chat template of the aligned model to extract user questions and answers from the "empty context". By 2025, DeepSeek-R1 and a batch of Reasoning Datasets will push this path to inference tracks, verifiable answers, code execution and multiple rounds of difficult case screening.

But "model generation data training model" is not a free lunch. Synthetic data brings distribution shrinkage, pattern collapse, style homogenization, false self-reinforcement, benchmark pollution and copyright/licensing risks. 2024 Nature About Model Collapse The work reminds us that if the new data is just recursively replacing the real data, the tail part cloth and rare modes may be gradually lost; there are also subsequent work points out that if the real data continues to accumulate and the synthetic data is verified, the risk is not uncontrollable. A more reasonable statement in engineering is not "replacement of human data with AI data", but "amplifying expensive human signals with models, and amplifying the process with validation and audit constraints".

2. Roadmap for the past year: from Instruction Data to Reasoning Data

2022-2023: Instruction generation and preference learning become infrastructure. InstructGPT establishes the productization path of SFT + Preference + RLHF; Self-Instruct proves that a small number of seed tasks can guide the model to synthesize a large number of instruction data; DPO Rewrite preference optimization into a more trained classification goal; UltraFeedback constructs large-scale AI Preference data with GPT-4 level feedback. The key word at this stage is "aligned chat behavior".

2024: Data synthesis begins to focus on diversity, roles, and evaluation of pollution. Work such as Magpie, Persona Hub, Instruction BackTranslation, etc., have advanced the problem of synthetic instruction from "generating more" to "covering more task perspectives". The open Recipe of Tulu 3 is particularly important. It discusses data sets, code, training recipes, evaluation and decontamination together, indicating that post-training is not a single algorithm, but a data governance project.

2025: Reasoning Data and Verifiable Reward become the main line. DeepSeek-R1 connects large-scale RL, cold-start data, rejection sampling, regular rewards and small distillation small models into a route. Tulu 3 uses SFT, DPO and RLVR to emphasize verifiable rewards; LIMO in turn challenged the intuition of "the more data the better", thinking that in the strong base model On the other hand, a small number of high-quality cognitive templates may also inspire reasoning ability; the OpenThoughts series has made the open Reasoning Data Recipe as a reproducible experimental object.

2026: The data flywheel starts to validate convergence with product logs, Agent environments, and tools. This part is still changing rapidly, and many Proprietary systems do not disclose details, and the following judgments are marked as "waiting for manual verification": the frontier system tends to put online failure cases, tool call logs, and automatic RED TEAM, code execution results, search/browsing tracks and user feedback back to training and evaluation. The focus of synthetic data is extended from "text samples" to "experienced samples with environment, with tools, and with validators".

Post-training Data Flywheel Framework

3. Interpretation of the papers on behalf of the paper

3.1 Knowledge Distillation: From Compression Model to Migration Reasoning Behavior

Hinton, Vinyals, and Dean systematically distilled knowledge distillation in 2015: compressing the predicted distribution of multiple models or large models into a single model to reduce deployment costs. The distillation of the LLM era is no longer just a soft label compression, but also includes the migration of instruction style, inference trajectory, tool call format, rejection policy and preference sorting.

DeepSeek-R1 is one of the most representative cases of 2025. The arXiv page shows that the paper 2025-01-22 was first submitted, revised 2026-01-04, and published in Nature 2025. The abstract emphasizes that pure RL can stimulate self-reflection, verify and dynamic strategy adaptation, and these inference patterns can be used to enhance small models. The important thing here is not a single point of BenchMark, but a replicable industry path of "Strong Reasoning Teacher -> Synthetic Reasoning Traces -> Small Model SFT/Distillment".

Meta also puts the Teacher Model at the core position in the Llama 4 official blog: Llama 4 Behemoth is described as the teacher model of Scout/Maverick, Maverick passes Co-Distillation gains quality improvements; the blog also mentions dynamic weighted distillation loss, lightweight SFT, Online RL, lightweight DPO, and easy samples to remove with model Judge. These details show that modern distillation has been embedded in multiple stages of pre-training, mid-training and post-training, rather than the compression step after training.

3.2 Instruction synthesis: from Self-Instruct to Magpie and Persona Hub

The process of Self-Instruct is very simple: use the language model to generate instruction, input, output, filter invalid or similar samples, and then use these samples to Fine-tuning the original model. Its value is to automate "task design" so that researchers do not have to label a large amount of data from zero for each new abilities.

Magpie's idea is more aggressive: use the user message location in the aligned model chat template to let the model automatically generate user queries and answers. The paper says it generated 4M Instruction-Response and chose 300K high-quality samples for training and evaluation. It reminds us that many closed-source/semi-open source Instruct Models imply a set of "user distributions" that can be extracted by prompts. However, this kind of method also brings about licensing, model usage terms and data source auditing problems, and it is not only possible to see the effect when it is actually implemented.

Persona Hub approaches diversity by using a large collection of personas to supply different perspectives for generating data. The engineering insight matters more than the figure of 1 billion personas. A handful of templates can yield synthetic data with much the same style and mix of tasks. Explicitly controlling persona, scenario, difficulty, language, domain, and safety level makes coverage something a team can adjust.

3.3 AI feedback with preference data: from RLHF to RLAIF/DPO

The classic process of InstructGPT relies on manual demonstration and manual sorting; Constitutional AI replaces some supervisors with a list of rules and AI feedbacks; UltraFeedback puts GPT-4 Feedback extends to large-scale preference data; DPO lowers the engineering threshold for Preference Optimization. Together, they drive a change: Preference data itself can be produced, filtered, de-biased, and versioned.

Two things need to be careful here. First, AI feedback is not objective truth, it inherits the bias, blind spot and security policy of Judge Model. Second, the preference data will strongly shape the style of the model, and over-optimization may make the model verbose, conservative, catered or lost exploratory. The official blog of Llama 4 mentions that SFT and DPO may be over-constrain model, so it is worthy of the engineering team to pay attention to this experience.

3.4 Reasoning Data: Verify the answer to change the way of data production

The key to reasoning data is not to "let the model write a long thinking chain", but "to make the training signal verifiable". Mathematics can compare the final answer, the code can run the test, formal proof that the Proof Checker can be called, and some scientific questions can be verified by using a simulator or retrieved evidence. DeepSeek-R1, Tulu 3's RLVR, and OpenHoughts data Recipe put verifiability in the center.

The goal of the OpenHoughts 2025 paper is to expose the Reasoning Model data Recipe. The OpenThinker2-32B trained by OpenHEGHTS2-1M can match the standard Reasoning Benchmark of DeepSeek-R1-Distill-32B, and pass 1000+ controlled experiments Improve the data generation pipeline. Regardless of whether these specific numbers are refreshed by the new model in the future, the direction it represents is very clear: the public community has begun to use "data formulas" as reproducible subjects of model capabilities.

LIMO provides another angle. The paper proposes that a small number of well-designed Reasoning Demonstrations may be sufficient to act as a cognitive template on the premise that knowledge has been encoded by Base Model. It reminds us that synthetic data flywheels cannot only pursue sample size; there is a strong interaction between data quality, task selection, trajectory structure and the existing ability of the Base Model.

3.5 Open Recipe: Tulu 3, Qwen3 and reproducible training

The importance of Tulu 3 is to "expose the full Recipe". The arXiv page shows that it publishes models, data, code and training recipes. The training algorithms include SFT, DPO and RLVR, and introduce development sets/unreviewed, standard Benchmark implementation and decontamination. This is more valuable for researchers than simply publishing a high-score model, as it makes post-training an auditable, comparable, and reusable system.

The Qwen3 technical report reflects another trend: unify Thinking Mode and Non-Thinking Mode in a model series, and reduce the cost of small model construction through flagship model knowledge. It does not fully unfold all the data details, but shows that the large model family is designing the inference mode, distillation, small model efficiency and multi-language coverage as the same product line.

4. Method comparison table

Method Main data source training target Advantage Risks and Attention Points
SFT on human demonstrations Artificial demonstrations, expert answers, production cases Imitation of standard answer formats and behaviors Stable, easy to debug, suitable for cold start High cost, limited coverage, easy overfitting templates
Self-Instruct / instruction synthesis Model self-generated tasks and answers Expanded instruction coverage Extend task space at low cost Homogenization, wrong bootstrap, high filtering cost
AI feedback / RLAIF Judge Model Rating, Rules of Rules Align Preferences and Security Policies Scale Preferences Judge bias, reward hackers, over-hatch
DPO / preference optimization chosen/rejected pair Direct optimization of preference probability Simpler than RLHF, low engineering threshold Data quality determines the upper limit, it is easy to learn style shortcuts
RLVR / rule-based reward Math answers, code tests, format rules Optimize verifiable task performance Clear feedback and can be automated Only cover verifiable subsets, may sacrifice open Q&A quality
Rejection sampling distillation Teacher generates multiple candidates, validator filtering Learn High Quality Tracks Suitable for reasonsing/code/math High computing power consumption, the deviation of the validator will be amplified
Data flywheel Online logs, failure cases, evaluation, synthetic data Continuous closed loop improvement Can follow the evolution of real tasks Privacy, licensing, pollution, and governance are complicated

5. Key technology trends

Trend 1: Data has changed from a static dataset to a versioned pipeline. In the future paper, "used a certain dataset" is not enough. The real amount of information is to generate prompt, teacher version, temperature, sampling times, filter, deduplication strategy, validator, pollution detection, mixing ratio and failure sample reflow rules.

Trend 2: The validator determines the upper limit of synthetic data. In math, code, tool calls, and scientific simulations, the validator can upgrade synthetic samples from "look like" to "executable". This is also one of the reasons why Reasoning Post-Training is progressing faster than open writing. Open Q&A, medical advice, and legal analysis lack simple validators, and it is necessary to combine retrieval evidence, expert review and conservative security strategies.

Trend 3: The teacher model is not only bigger, but also more suitable as a data producer. A good teacher should be able to generate a variety of, verifiable, clearly interpreted, and consistent with the target style; sometimes a top-level chat model may not be the best data generator. Both Llama 4 and DeepSeek-R1 remind us that the ability structure, output format and training phase of the Teacher will affect the final behavior of the student.

Trend 4: Small model capabilities are increasingly dependent on high-quality distillation. Smaller Model is required for edge deployment, enterprise privatization, and low-latency services. Distillation allows the small model to obtain the behavior pattern of reasonsing/coding/tool-use, but also inherits the errors, security boundaries and language preferences of teachers. The small model is not a "cheap replica", but a product that needs to be individually evaluated and constrained.

Trend 5: Data governance becomes part of model capabilities. Deduplication, decontamination, copyright license, PII filtering, security red team, data lineage and rollback mechanism is no longer just a compliance sub-item. An untraceable data flywheel will soon cure online bugs, benchmark leaks and model bias into the next round of training.

6. Inspiration of the project landing

First, the synthetic data system is dismantled into observable components. Requires at least Source Registry, Prompt Template Registry, Teacher Registry, Generation Jobs, Filter Jobs, Eval Jobs, DataSet version, training run, and model card. Each sample is best traced back to the source, generation parameters, filtering reasons, and used model versions.

Second, don't take "more synthetic data" as the default optimization direction. The more feasible approach is to produce data according to the ability gap: first sample from eval failure, user complaints, low Confidence tasks and business high-value scenarios, and then decide whether to manually label, teach distillation, Retrieval-Augmented Generation, tool validation, or directly modify the product tips and Guardrail.

Third, establish three layers of filtering: format filtering, semantic filtering and result verification. Format filtering ensures that JSON, tool calls, and reference structures are correct; semantic filtering checks duplication, low information, obvious hallucinations and security questions; results validation use unit test, math answer, retrieval evidence, expert rule or Judge ENSEMBLE bottom. A single LLM Judge should not be the last referee.

Fourth, the Training Set and evaluation sets are managed by decentralization. The easiest mistake to make in the data flywheel is to convert the evaluation failure sample into training data, and still use the original evaluation set to report progress. The correct way is to keep Frozen Holdout, label the reflow samples, use time to split or source, and do contamination scans if necessary.

Fifth, institutionalize the "real data anchor point" of synthetic data. Even if most of the samples are generated by the model, it is still necessary to continuously inject artificial expert samples, real user authorized samples, production environment error samples, and the domain Ground Truth. Otherwise, the data distribution will shrink toward the comfort zone of the Teacher Model.

Sixth, when the enterprise lands, it starts with the verifiable task. code repair, SQL Generation, customer service classification, document extraction, form verification, knowledge base question and answer, and policy compliance checks are more suitable for building data flywheels than open strategic suggestions, because they have logs, acceptance standards, and error labels, and are easier to do offline playback.

7. Limitations and Disputes

Model collapse is not a slogan, but it is not an inevitable outcome of synthetic data. Recursive training is the greatest risk when the real data is completely replaced; if the real data is continuously accumulated and the synthetic data is strictly screened, the risk can be mitigated. The problem with the engineering is that many teams do not have enough lineage and testing to know if they are recursively contaminated.

Synthetic data amplifies the blind spot of the teacher. If the teacher does not have a certain type of problem, it is difficult for student to learn from the teacher output; if the teacher has a bias in security policy, cultural background or language style, the student will inherit or even strengthen these biases.

The interpretability of the inference trajectory is still controversial. The training model growth chain-of-thought is not equivalent to obtaining the real internal reasoning process. Many systems will distill the Reasoning Style, but the final product may hide or compress the thought chain. When comparing models, researchers need to distinguish "tracks as training signals" and "tracks as explanatory evidence".

Benchmark numbers are increasingly contaminated with data flywheels. When data generators, filters, and training tasks are all around open benchmark iterations, no reviews and real task reviews are more important than leaderboards. Tulu 3 emphasizes that Decontamination is a good signal, but the industry as a whole still lacks a unified standard.

Risk of legal and platform terms rises. Use a business model to generate training data in batches, and whether to allow training competition models depends on the terms of service and the authorization of the data source. This article does not provide legal advice; when involving commercial training, it must be verified by the legal and data governance team.

8. What's next worth noting

  1. Can verifiable rewards be extended to open world quests? Mathematics and code are relatively clear, and scientific research, medicine, legal, corporate decision-making requires more complex Evidence Graph and expert review.
  2. How does the data flywheel handle long-term memory and privacy? The Agent system will generate a large number of user trajectories, which can be trained, which can only be personalized memory, and which must be deleted, and clear boundaries need to be clear.
  3. Will small model distillation become the main battlefield of open source ecology? If the Reasoning Traces of strong teachers are available, the 7B-32B model may be sufficiently available in a large number of vertical tasks.
  4. Will the review move from benchmark-first to replay-first? Real systems require offline playback, A/B Shadow Eval, and risk stratification evaluation based on production logs.
  5. Can the data Recipe reproduce like code? Future valuable open source projects may not just be weights, but complete Data Generator, Filter, Eval Harness and training scripts.

9. Summary

Synthetic Data, Distillation, and Post-Training Data flywheels together point to a change: the model's capabilities are increasingly like the product of a "Data operating system". Algorithm names are still important, but the upper limit is often where the Data comes from, how to generate, how to verify, how to mix, how to reflow, how to audit.

For researchers, the next step is worth paying attention to the public RECIPE, validator design and data pollution control, rather than simply comparing a model's score on a single list. For the engineering team, the most pragmatic starting point is: around a verifiable business task, the production failure sample, teacher generation, filter, offline evaluation and small model distillation string are closed-loop. Only when this closed loop is traceable, rollable, and measurable, the synthetic data has changed from a "cheap sample" to a real model asset.

References

The search date is 2026-07-05. Change information such as model version, official product availability, code repository status and benchmark ranking should be manually verified before release.

  1. Geoffrey Hinton, Oriol Vinyals, Jeff Dean. “Distilling the Knowledge in a Neural Network.” arXiv, 2015. https://arxiv.org/abs/1503.02531
  2. Long Ouyang et al. “Training language models to follow instructions with human feedback.” arXiv, 2022. https://arxiv.org/abs/2203.02155
  3. Yizhong Wang et al. “Self-Instruct: Aligning Language Models with Self-Generated Instructions.” arXiv, ACL 2023 version. https://arxiv.org/abs/2212.10560
  4. Yuntao Bai et al. “Constitutional AI: Harmlessness from AI Feedback.” arXiv, 2022. https://arxiv.org/abs/2212.08073
  5. Rafael Rafailov et al. “Direct Preference Optimization: Your Language Model is Secretly a Reward Model.” arXiv, 2023. https://arxiv.org/abs/2305.18290
  6. Ganqu Cui et al. “UltraFeedback: Boosting Language Models with Scaled AI Feedback.” arXiv, ICML 2024 version. https://arxiv.org/abs/2310.01377
  7. Zhangchen Xu et al. “Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing.” arXiv, 2024. https://arxiv.org/abs/2406.08464
  8. Tao Ge et al. “Scaling Synthetic Data Creation with 1,000,000,000 Personas.” arXiv, 2024. https://arxiv.org/abs/2406.20094
  9. Nathan Lambert et al. “Tulu 3: Pushing Frontiers in Open Language Model Post-Training.” arXiv, v5 revised 2025-04-14. https://arxiv.org/abs/2411.15124
  10. DeepSeek-AI et al. “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.” arXiv, first submitted 2025-01-22, revised 2026-01-04. https://arxiv.org/abs/2501.12948
  11. Yixin Ye et al. “LIMO: Less is More for Reasoning.” arXiv, 2025. https://arxiv.org/abs/2502.03387
  12. Ke Wang et al. “A Survey on Data Synthesis and Augmentation for Large Language Models.” arXiv, 2024. https://arxiv.org/abs/2410.12896
  13. Mihai Nadas, Laura Diosan, Andreea Tomescu. “Synthetic Data Generation Using Large Language Models: Advances in Text and Code.” arXiv, 2025. https://arxiv.org/abs/2503.14023
  14. Ilia Shumailov et al. “The Curse of Recursion: Training on Generated Data Makes Models Forget.” arXiv, 2023. https://arxiv.org/abs/2305.17493
  15. Matthias Gerstgrasser et al. “Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data.” arXiv, 2024. https://arxiv.org/abs/2404.01413
  16. Etash Guha et al. “OpenThoughts: Data Recipes for Reasoning Models.” arXiv, 2025. https://arxiv.org/abs/2506.04178
  17. An Yang et al. “Qwen3 Technical Report.” arXiv, 2025. https://arxiv.org/abs/2505.09388
  18. Meta AI. “The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation.” Official blog, 2025-04-05. https://ai.meta.com/blog/llama-4-multimodal-intelligence/
  19. OpenAI Developers. “Supervised fine-tuning / Distilling from a larger model.” Official API docs, accessed 2026-07-05. https://platform.openai.com/docs/guides/distillation