Laboratory cover for synthetic data screening

  • Date: 2026-09-06
  • For graduate students, early-career researchers and research engineers

Abstract

Generating ten thousand instructions is easy. Explaining which thousand to keep is harder. A fluent answer can hide a mistake, high-scoring examples may all share a template, and the hardest question may simply be mislabeled. We select synthetic examples under a fixed retention budget, using an inspectable quality classifier, diversity constraints, and a quota for hard examples. We preserve the evidence for each decision. We examined the official Self-Instruct and DEITA implementations and ran a pure Python experiment to check the filtering behavior. We did not reproduce the language-model training gains reported in the papers.

Reproduce value: split "good data" into three falsifying problems

Quality asks whether a single sample is worth learning, Diversity asks whether it provides new information relative to the selected set, and the difficulty asks whether the current learner has not mastered it. The three cannot share an undefined score. Repeated correct answers may be of good quality but redundant; rare tasks may be valuable or may be generators; high losses may also come from knowledge gaps and wrong labels.

Paper report: Self-Instruct is boot generated through seed instruction, and then filter invalid or similar samples for instruction fine-tuning, which proves the feasibility of the synthetic instruction route. Its conclusion depends on the model, generation settings and evaluation at that time, and cannot be understood as the natural and reliable generation of the generated text. ACL 2023 Papers

DEITA measures complexity, quality and diversity, and then chooses data together. Its research object contains an instruction data pool from different sources, and not all records are generated by the same generator. This article uses its selection ideas to process controlled synthetic data, and adds additional executable checks and difficult case quotas; the latter two are teaching experimental designs, do not pretend to be the original algorithm. DEITA Finale V2

Core Ideas and Formulas: Fractions are agents, and sets are output

Quality Classifier refers to the quality classifier: The degree to which the predicted sample belongs to the "available" category is based on the label. The label must be written first, and this case only learns the difference between readable arithmetic interpretation and empty answer and repeated filling. Let the feature matrix be X∈RN×4X\in\mathbb R^{N\times4}, using logistic regression:

pi=σ(w⊤xi),L=−1N∑i[yilog⁡pi+(1−yi)log⁡(1−pi)].p_i=\sigma(w^\top x_i),\quad \mathcal L=-\frac1N\sum_i[y_i\log p_i+(1-y_i)\log(1-p_i)].

NN is the number of training samples, xix_i Including intercept, whether non-empty, truncated words and tokens de-weight ratio; ww is a four-dimensional parameter, yiy_i is a binary label, output pp For the length NN vector. Adding two-norm regulars with non-intercept parameters during code optimization. This model cannot calculate the authenticity of the answer, and the output is not calibrated with an independent probability and cannot be called "accuracy".

DEITA uses another scorer: first sorts the scoring of the homologous evolution version, and then trains the model to predict complexity and quality. The official reasoning scores on the tokens corresponding to the six scores and seeks expectations to obtain scalars; it is not the two-class training of this article. Scoring source code

to vectors that have been normalized ei∈RDe_i\in\mathbb R^D, the diversity rule of this example is:

max⁡j∈Sei⊤ej≤τ.\max_{j\in S}e_i^\top e_j\leq\tau.

SS is a reserved representative set, DD is a dimension, τ\tau It is the upper limit of similarity; the empty set directly receives the first item. A candidate comparison yields a length ∣S∣|S| score. The similarity is clearly used here: the bigger the closer. This vector is the one-hot encoding of the artificial task category, that is, one position is one and the rest is zero, and the true semantic coverage cannot be proved accordingly.

Quality, Diversity and Difficult Cases Screening Process

The figures first exclude verifiable invalid samples, then score and construct various representatives, and finally arrange difficult cases and audit in the budget. The bars in the figure only show the score, and the card color only shows the task difference. Difficult cases must have passed validity and quality checks, and the quota cannot reintroduce the wrong answer into training.

Guide to the official implementation: from field to inequalized sign

Read Self-Instruct first bootstrap_instructions.py. It checks the similarity of the instruction with the longest common subsequence ROUGE-L, which is skipped when it exceeds 0.7; at the same time, it filters the short and long instructions, non-ASCII starts, etc. The latter serves the original English settings, and the process will directly delete the normal command by mistake when copying to the Chinese process. This threshold is also not a semantic equivalent standard. Instruction generation source code

Read again prepare_for_finetuning.py The parsing and filtering functions: the empty output, the same input and output, and the suspected incomplete records at the end of the colon will be filtered; the group instance will be discarded when the same non-empty input has different outputs. This helps clear up the conflict label, but open writing may have multiple reasonable answers, and the rules must obey the task definition. Instance cleaning source code

DEITA from score_pipeline.py How to track multiple rounds of dialogues how to form an instruction answer according to the position; then look at filter_pipeline.py, the score list is written to the vector table by row, and the selected idx Also index the original JSON. If the vector is shuffled separately after the vector is exported, it will be silently mismatched even if the length is consistent. When reappearing, the stable sample number should be kept, and the original abstract should be checked one by one. rating pipeline、Filter Pipeline

Final reading combined_filter.py: The quality and complexity of the wheel by round are multiplied, and then summed and sorted. Don't be misled by the plus signs spliced in the log. Enter filter/base.py after, it is called compute_distance The cosine branch actually returns to the normalized dot product, and is excluded from being strictly greater than the threshold. The paper calls this amount of distance, and the source code reveals the specific direction; this case also tests the behavior that is exactly equal to the threshold. joint sorting、similarity screening

Minimum Experiment: Keep High Scores, Coverages and Correctness Separately Change

code/minimal_filter.py Construct data with deterministic templates without calling the language model. There are 24 training sets and six development sets, which are separated according to the arithmetic seed family, but the template is shared, so it is only a control experiment. The development set scans the five thresholds, and the blending average value of the recall is selected at the precision rate and the recall average value, and the value closest to 0.5 is taken in a draw; the candidate data is never involved in the fit or selection threshold.

There are fourteen candidate pools, including addition, sorting, string reversal, square and four categories, as well as a smooth error answer and an empty answer. The square sum is defined as a difficult case by hand and does not represent the difficulty of the actual student model. Correctness checks only compare structured answer With the built-in operation results, each sentence in the natural language interpretation is not reviewed. Execute from the article directory:

PYTHONPYCACHEPREFIX=/private/tmp/codex-ai-065-pycache python3 -m py_compile code/minimal_filter.py
python3 code/minimal_filter.py > code/smoke_test.txt

What we tested: Python 3.9.6, CPU, seed 65; training, development and candidate feature shapes are respectively [24,4]、[6,4]、[14,4]. The training binary cross-entropy was reduced from 0.693147 to 0.027918, the selected threshold was 0.5, and the twelve candidates passed both mass and structure checks. Developing templates is too easy, and these numbers cannot be generalized performance as real quality classifiers.

Strategy, the reserved budget is three Four types of tasks coverage Difficult number of cases The total number of answers
Random sampling of qualified pools 0.75 1 46
Quality sort only 0.50 2 17
quality plus diversity 0.75 1 22
Re-join the difficult case quota 0.75 1 22

Diversity adopts a threshold value of 0.9, first traverse all qualified candidates to get a collection of representatives, and then intercept the budget. Difficult case policies have priority reserved for at most one legal and difficult case representative, and then populated by score. Here, the quality sorting is already preferred for squared and answering with short vocabulary, and the quota for difficult cases has not changed the result; random sampling has also reached the same coverage, and it is impossible to claim that the joint strategy is better than random selection.

The quality score of the wrong answer is 0.894078, exactly the same as its correct version, both crossing the mass threshold. The reason is that the replacement number does not change the four characteristics; the structural check excludes the wrong version. Although it did not enter the top three this time, it still proves that the threshold release is not a guarantee of correctness. See all samples, weights and selection trajectories code/results.json; Syntax, empty pool, zero vector rejection and threshold boundary checks are all passed.

Evaluation protocol: Fixed Budget, but also retains independent referees

The real experiment first formulates the labeling scale: whether the task can be answered, whether the answer is correct, whether the format meets the requirements, and explains whether the conclusion is supported. The status of the ambiguity is listed separately, and the recorder is divided. Divide into the original seeds, rewrite the family and generate batches, to prevent the same question from appearing in the training and development set at the same time. After freezing the threshold, the available sample accuracy, RECALL, and stratified errors are measured on the unseen source.

The selection results report quality, diversity, and cost at least simultaneously. In this case, the denominator of the coverage rate is a preset four types of tasks, and the number of words is the result of the English regular tokenization, not the model tokens. The official fine-tuning should supplement the training tokens budget under the fixed tokenizer, because the three long answers may consume more calculations than the three short answers. Compare the number of fixed samples first, and then compare the number of fixed training tokens; record the number of optimization steps, learning rate, truncation length and random seeds.

Filter migrations should also be evaluated separately. Leave all the output of a generator for external inspection, and compare the error types under different languages, templates and answer lengths; the good results obtained randomly by randomly segmented from the same source may only indicate that the model recognizes the style of writing. The rejected data is as important as the retained data, otherwise it is impossible to estimate the ratio of valuable and difficult cases to be deleted by mistake. If the layered sampling review is used, the sampling probability needs to be saved, and the difficult ratio of deliberate sampling cannot be regarded as the overall ratio.

Downstream evaluations must be independent of generators and filters. Mathematics can be verified by answers, the code can be tested in isolation, open tasks combined with blind evaluation and manual random inspection; Random, only quality, quality plus diversity, and even more difficult cases to make at least the same budget control, repeat multiple seeds and report fluctuation. This time, only one screening seed is run, there is no training model, and there is no error bar, and the follow-up results are subject to manual verification.

The audit record should retain the generation source, prompt version, raw response, parent example ID, scorer version, reason for exclusion, and final assignment. For labels adopted after human review, record separately when the review took place and what justified the decision. This guards against changing labels after the fact to improve a metric. Run deduplication and contamination checks independently. Even a highly diverse training set can contain test questions.

Failure investigation: check who was screened first

The average quality is increased and the effect decreases, first check the retention rate according to the length, task and source. This example has a higher score for short answers, indicating that a seemingly reasonable combination of features will also learn shortcuts. Only create slices for smooth wrong answers, short correct answers and template rereadings, so that you can see the hidden problems of the overall equality.

When the coverage is abnormal, check whether the vector is wrong, whether it is normalized, how the zero vector is handled, and the similarity threshold direction. The task one-hot coding makes the category boundary of this example too ideal; the real embedding may aggregate different abilities, or it may be synonymous and rewritten. Increasing the proportion of difficult cases makes the training unstable, then randomly check the labels, truncations and unsolvable conditions of high-loss samples, and do not attribute them to the model capacity first.

Questions for further research

The authors infer that it is worth studying the trade-offs under the fixed budget, rather than looking for a general high score threshold. The length distribution can be fixed to check whether the quality classifier still has screening advantages; the independent correctness label can be fixed, and the category quota and neural expression screening can be compared; then the error rate of freezing students is defined, and the profit curve of legal and difficult cases to vary from zero to different quotas is studied. The failure criteria are prescribed, such as a drop in the long tail task or an increase in the retention rate of the wrong label, so that the negative results can also be reported.

Summary

For the reproducible synthetic data filtering, the collection of samples that can be checked back and the reason for selection should be output. This time, the actual verification of high-scoring error answers, insufficient coverage of quality sorting, and the situation that the quota of difficult cases does not generate gains. In the next step, independent labeling and real representation should be expanded, and then controlled fine-tuning should be carried out; official weighting reasoning, original paper training and cross-domain benefits have not been verified.

References

Search date: 2026-09-06. All the links cited in the text have been actually opened. OpenReview encounters the verification page, and uses the author arXiv to finalize the verification; the official branch is recorded according to the visit date, and the commit is not fixed, and the compatibility is re-run for manual verification.

  • Yizhong Wang、Yeganeh Kordi、Swaroop Mishra、Alisa Liu、Noah A. Smith、Daniel Khashabi、Hannaneh Hajishirzi,2023, Self-Instruct: Aligning Language Models with Self-Generated Instructions, ACL Finalized Full Text、Official repository. The preprint was first published in 2022.
  • Wei Liu、Weihao Zeng、Keqing He、Yong Jiang、Junxian He,2024, What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning,ICLR 2024, arXiv V2 Finalization Instructions、Official repository. V2 revision date is 2024-04-16.