
- Date: 2026-09-08
- For graduate students, early-career researchers and research engineers
Abstract
Does a falling training loss mean a small model is getting better at following instructions? We examine three questions. What does changing the task mix teach? How does supervising only the response compare with supervising the entire text? Can the model still generate the right answer under a new template? Drawing on instruction-tuning papers and official code, we trained a tiny recurrent language model to check the protocol and provide a script for transferring it to a local base model. We report the low accuracy, large variation across seeds, and template failures as results of the toy experiment.
Reproducing value: why a small amount of data still needs strict comparison
Supervising fine-tuning, that is, SFT, is to continue to optimize the model with input and demonstration answers. The ability provided by pre-training, the tasks of demonstration coverage, the output format and evaluation method jointly determine the benefits, and the results cannot be explained only by the number of data.
The paper report: LIMA of Zhou et al. has studied alignment with a thousand selected examples on the already 65 billion parameters LLaMA. It supports small sample effects under specific base models and data conditions, and cannot prove that a small network from zero training only needs this data. LIMA original paper
Wang et al.'s Tülu work compares a variety of open instruction data, and finds that the different data helps differently in the ability dimension, and there is no mix of the best in all the evaluations. For newcomers to study, this means to determine the ability to slice first, and then decide the ratio, not just the average preference score. Tülu Papers
What this paper actually verifies is the training mechanism and measurement link. The miniature model facilitates the preservation of each checkpoint and checking gradient, but lacks large-scale pre-training knowledge. The real small model interface also uses the SmolLM2-135M basic model as a preparation object; the model card distinguishes the basic and instruction versions, and the two cannot be mixed and still claim to perform the first instruction Fine-Tuning. official model card
Core Ideas and Formulas: Mixed samples, supervising are tokens
Serialize a conversation into a tokens sequence , only calculate the loss in the answer position:
The model output is [B,T,V], respectively represent batch size, predicted position number and vocabulary size; mask For [B,T], take one at the answer and end tokens, and take zero at the prompt and fill target. Must be greater than zero. The model still reads the Prompt, and the gradient can also be returned in the hidden state; the predicted loss of the shielded Prompt is not the same as cutting the input information.
Chat templates are responsible for turning roles, content, and round boundaries into sequences that the model sees. In the same question, the character marker or answer to the starting position of the same question may change the condition distribution. Print the final tokens and valid labels first, and then check if the inference prefix is consistent with the training. Framework official documentation reminder: When the rendered template is tokenization again, you should avoid adding special tokens repeatedly. Template documentation
hypothetical source The sampling probability is , the average effective answer length is , when the global average is averaged, its expected supervision ratio is approximately . This estimate assumes stable length and no extra truncation; long answer sources may contribute more positions. If the weights are averaged for each sample, the weight relationship will change again, so the ratio must be reported together with the loss normalization.

In the figure, first divide the training data according to the problem group, and then only mix the training data; the development set selects checkpoints, and the test set only checks the freezing scheme. The development and test data also require consistent coding and scoring processing. These repeated steps are omitted in the figure, and the arrow does not indicate direct input to unprocessed text.
Guide to the official implementation: Go from Data Entry to Loss
Read Open Instruct README first to confirm the study version and entry. The current repository already contains post-generation training programs; reading today's main is not the same as the original environment for obtaining the 2023 paper. This article records the search date, and the submission is not locked, and the complete reproduction is still subject to manual verification. Official repository
Look again dataset_transformation.py of load_dataset_configs: Strings in the mixed list "1" is interpreted as a sample, "1.0" the proportion of the entire source. follow-up _tokenize_tulu_sft_with_assistant_labels Look at the character offset and answer tags of the template rendering, the quick tokenizer; the initial label is -100, and then restore the supervised area. You can't use simple string search instead of multi-round boundary derivation. data source code
Finally finetune.py Look for model(**batch)、outputs.loss and back-propagation, and track effective predictive tokens statistics. Check whether the causal shift has been done, and avoid cutting the label again. There is additional processing for the distributed path of the repository, which is not executed in this article. training source code
If TRL is used, differentiate between answering role supervision and complete oversight of prompt-completion. assistant_only_loss=True Relying on templates can mark the assistant generation area, you can't just turn on the switch to assume the mask is correct. The actual template, version and a valid tag after decoding should all enter the record. SFTTrainer documentation
Minimum experiment: first build a causal language model that can be played back
code/minimal_sft.py Using NumPy to implement a recurrent neural network, that is, tokens update the RNN of the hidden state, hide the dimension 48, and the verb table, a total of 3,516 parameters. It is not a Transformer, nor is it a natural language pre-trained model.
Tasks are copy or flip binary string bit by bit. First warm up for 600 steps on the sixteen task sequences of the three-digit string, and then fine-tuning the four-bit string. This is a synthetic warm-up and cannot be equivalent to large-scale text pre-training. The four inputs are grouped according to the original string, training eight groups, four groups of development, and four groups of tests.
The three branches of each seed start with the same warm-up weight and reset the optimizer. The eight replicated samples are used for a single source branch twice; the balanced branch is copied and flipped with eight samples. The batch size is 16, the Adam learning rate is 0.003, the upper limit of the gradient norm is one, and all training is 400 steps.
python3 code/minimal_sft.py > code/smoke_test.txt
python3 -m py_compile code/minimal_sft.py code/local_hf_sft.py
Need to press first code/README.md prepare the environment. This time, Python 3.12.14, NumPy 2.3.5, and CPU are actually used; the system Python lacks NumPy. Enter [16,12], output [16,12,12]. We scheduled checks of development-set response negative log-likelihood (NLL) at 20, 100, 200, and 400 steps. Lower values are better. This was the sole criterion for choosing a checkpoint. Test accuracy played no part in the choice.
The following table shows the strict entire string matching rate of initialization seeds 67, 68, and 69, which requires the content and end tokens to be correct; greedy generates up to eight tokens. "±" is the sample standard deviation of the three seeds, not Confidence interval .
| fine-tuning settings | Match rate with template test | Template test matching rate | Supervise tokens at each step |
|---|---|---|---|
| Copy only, answer supervisors | 0.125 ± 0.125 | 0.000 | 80 |
| Balanced mixing, answering supervisors | 0.333 ± 0.260 | 0.000 | 80 |
| Balanced mix, full text supervision | 0.167 ± 0.072 | 0.000 | 192 |
The warm-up model has zero match rates on the four-bit Test Set. The mixed branch has a high average but a lot of fluctuations. Each seed has only eight test records, and it cannot be promoted according to this. The full text branch changes the supervised position and denominator, and although the input is the same as the number of steps, it is not a fixed supervision tokens budget control.
Template will be changed USER/ASSISTANT To TASK/INPUT/ANSWER packaging. Although these tokens are in a fixed vocabulary, they do not appear as a warm-up or fine-tuning input; zero points are external stress tests for extreme distribution, which cannot prove that the real model will crash for ordinary words and rewrites.
All data, generation, development curves and weights are saved. The maximum error of the finite difference gradient of the five types of parameters is about , the first answer position, the end tokens count and the batch mean check are passed. additionally attach local_hf_sft.py, it can be loaded from the local basic model, trained according to the answer mask, and the selection model is developed; there is no torch in this machine, only the syntax check is passed, and the real weight loading, training and interface compatibility are to be manually verified.
Evaluation protocol: test fitting, generalization and formatting separately
Fixed the basic model and tokenizer version, and save the original template, data source and deduplication rules. Segmentation according to the problem family is more important than random dismantling: the rewriting of the same problem and the two problem-solving versions should be in the same group. This article tests new input combinations, and the task type has still been seen, and cannot be called cross-task generalization.
The evaluation prefix itself should also be preserved as an experimental material. For example, in this example, the training sequence of the input is copied to connect the four answers tokens and the end tag with the assistant mark; during the test, only the previous part of the assistant is provided, and the real answer cannot be quietly left in the input. Save generation can check this leak and make it easy to recalculate more stringent metrics.
Task Accuracy, format compliance rate, answer NLL should be separated. The former checks free generation, and NLL uses the real answer prefix, so low NLL does not guarantee that all long sequences are correct. Natural language tasks should also record the wrong type and original output, avoiding the format default hidden in the "correct answer" after cleaning the extra explanation.
The match study reports at least the number of samples, valid answers tokens, total input tokens, and optimized steps. Long answers, truncations, and repeated sampling will all change the budget. In this case, the single source duplicate data is used to match the scale of each step, but there are fewer independent samples. This is the structural difference of the ratio comparison. It cannot be said that all factors are the same.
Three initializations and sharing of all points, and its standard deviation only reflects the sensitivity of training. Formal reporting requires more problem groups and pre-frozen formulas, which can be paired and resampled according to the problem groups; only expanding the number of training seeds will not make up for the insufficient test coverage.
Failure investigation: look at the development set first when the training drops
For the balanced branch with seed 67, increasing training from 100 to 400 steps lowered training-answer NLL from 0.0588 to 0.0021, while development NLL rose from 0.6055 to 1.0407. We therefore selected the 100-step checkpoint. In this run, fitting the training data more closely brought no development-set gain. Other runs may behave differently.
When there is an error in the generation length, first check whether the end tokens is supervised and the maximum generation length. A flip test answer of Seed Sixty-seven should be 0101, model generation 01100 It ends later, indicating that the error involves both content and stop, and cannot only be classified as a format problem.
The exception of small losses may also be supervised easy-to-predict templates, truncated answers, or all labels are invalid. Decode a valid tag first, and then check the predicted position of the first answer tokens. When the sample packaging is enabled, that is, when multiple records are spelled into a sequence, it is also necessary to confirm the cross-sample attention and boundary processing; this time there is no packaging, and the throughput result cannot be pushed to this function.
Questions for further research
The authors infer that the benefits of a balanced mix may come from the coverage of the second task, and may also be more relevant to independent samples. In the next step, the independent input number is consistent, the single source, multi-source and repetition count are compared, and then repeated under the fixed effective tokens budget, these factors can be separated.
Template problems can be shifted from extreme new tags to the same vocabulary, the same length packaging perturbation, and multi-template training controls are added. If the improvement only occurs on the packages you have seen, it may still be a format matching. After migrating to the real small model, you should also add the original ability test to observe whether the instruction adaptation is accompanied by forgetting.
Summary
We completed a reproducible experiment covering data grouping, response supervision, selection on the development set, and free generation. It exposed signs of overfitting as well as template failures. The paper supplies the research hypothesis, and the official source code shows the protocol actually used. Reliable replication means checking both against the local evidence, item by item. An attractive training curve cannot substitute for an independent test.
References
Search date: 2026-09-08. The links of the text have been actually opened, and the following are the core version and reading portal; dynamic code and model operation compatibility are subject to manual verification.
- Chunting Zhou et al., 2023, LIMA: Less is More for Alignment, arXiv v1 full text.
- Yizhong Wang et al., 2023, How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources,arXiv v2, the page indicates NeurIPS 2023 Datasets and Benchmarks camera-ready.
- AllenAI, Open Instruct official repository, this article reads the data and training module of main, and submits a snapshot of non-original papers.
- Hugging Face, Chat Template Documentation、TRL SFTTrainer and SmolLM2-135M Basic Model Card.