Template changes and fixed research objects

  • Date: 2026-10-05
  • For graduate students, early-career researchers and research engineers
  • Scope: Real small model, six types of templates, two model states; development experiments, non-complete benchmarks.
  • Acceptance status: All calculations, automatic auditing and maintenance of real-life labels have been approved; the artificial conclusion is directly confirmed by the user in this session on October 5th.

Abstract

The previous article changed the mixing ratio of training data, but did not get the credible news ability to maintain gain. In this article, the selected checkpoint is fixed, and the question is reduced to: the same film review is another question, does the fine-tuning income still exist? We reuse 64 emotional development samples and 32 news development samples, and run 640 real generations. The original JSON template and instruction rewriting after fine-tuning are both correct and correct, but there is one gain and loss; there are only 55 pieces left after the order of exchange of instructions, and the format is still all qualified. The original word output convention is reduced from 57 articles to thirty-three articles. The results show that format, content and task migration must be measured separately. The user has confirmed that the real person label remains reviewed; this is still a development exploration of the complete log, and cannot be renamed the blind test accordingly.

Contents

  1. Reproducing value and core ideas
  2. Formulas, Variables and Evaluation Protocol
  3. A guide to the official implementation
  4. Minimum Experiment and Source Code Entry
  5. Results, Failure Troubleshooting and Research Questions
  6. Summary and References

1. Reproducing value: don't take the familiar output as a universal ability

Supervise the fine-tuning, that is, update the model parameters with input and target answers, which may change the task judgment, answer wording and output format at the same time. The success rate is only measured on the prompts during training, and these changes cannot be separated. JSON is a structured text format; generating a legitimate object does not guarantee that the emotional judgment in the object is correct. Conversely, writing the correct emotional words but adding more explanations may not satisfy the interface convention.

What the paper reports: The ICLR 2024 papers of SCAR, Choi, Tsvetkov and Suhr study the change of the prompt format of semantics. It is recommended to examine the performance range in various reasonable formats. Original Paper V2 Supporting "single tips is not enough to portray robustness" and does not provide the expected score of this experiment. Yan et al. Contrastive instruction Fine-Tuning paper Further study the consistency of the expression of equivalent instructions; this paper does not run its training algorithm, and cannot borrow the paper to improve as its own evidence.

Compared with 091, the new variables in this article are the way of presentation of user instructions, freezing model status, original text, label mapping, chat wrapping and decoding rules. The model comes from SmolLM2-135M-Instruct, and the original checkpoint itself has passed the instruction Fine-Tuning; "Fine-Tuning" only means that before our small incremental training, it is by no means an untrained language model.

2. Core thought: gradually extrapolate along three distances

Fixed sample and paired evaluation process

The nearest layer is the same task, the same output convention, and only change the wording of the task sentence. The main comparison has changed the "emotions of this film review of classification" to "judging the emotions expressed by this film review", and retained the positive/negative tag and complete JSON requirements. The second variant only exchanges the positions before and after the task sentence and the output requirements, and the two are secondary comparisons.

A further layer of the original Plain template is used, and only one lowercase emotional word is required to be output. This also changes the output convention and cannot be attributed to synonymous rewriting. The furthest layer is the classification of news topics, using A/B/C/D to correspond to World, Sports, Business, Science and Technology, and compare the original instructions and rewrite instructions. The two-tier diagnostic helps limit the gain, but is not mixed with the primary control as an average score.

News did not enter the incremental training of the selected Target_Only checkpoint, but participated in the selection of the development set in 091; the pre-training data cannot be fully traced. Therefore, it can only be called "task family for training exclusions", and cannot be called untouched tasks or blind test migrations. Binary emotion is different from the difficulty and number of four types of news, and it is not advisable to compare the original accuracy of Accuracy in horizontal direction.

3. Formulas and Evaluation protocol: Format and content, write a stroke

Check whether the strict format under the template T of the i-th sample is qualified as F_it, and whether the only explicit category in the output matches the original label as C_it. The joint is correctly defined as:

Jit=FitCit,Δt=1n∑i=1n(Jit−Ji0).J_{it}=F_{it}C_{it},\qquad \Delta_t=\frac{1}{n}\sum_{i=1}^{n}(J_{it}-J_{i0}).

Here, F, C, and J are both zero or one; n is the number of samples for the same task, and the zero number template is the original template. The difference between each row belongs to negative one, zero, and positive one, corresponding to losing, leveling, and winning. The calculation is paired according to the original sample, and multiple templates of a sample are not considered as independent observations. The body only reports count and descriptive differences, and there is no Confidence interval or multi-substance declaration.

Format scoring rejects additional JSON keys, duplicate keys, error case, and out-of-object interpretation. The content scoring continues with the existing unique tag extractor; a paragraph with both positive and negative, which is considered ambiguity, cannot be guessed by the program. C is a recalculated content proxy and cannot be called artificial semantic Accuracy. Condition Accuracy also cannot just keep the formatting sample and omit the denominator, otherwise the most difficult answer may disappear quietly.

Semantic retention is another evidence. The program has checked the original text verbatim, labels and category mappings, and generates a manual review package without model answers before reasoning, a total of 320 templates and samples are combined. The user then directly confirms that "real person review has passed" in this session, and we have recorded the retrospective review and saved the original text and the review package hash. The user did not submit another line-by-line annotation and the completion time of the precise review, so the original blank form was reserved, and 320 judgments were not filled on behalf of him, and the confirmation record time was not posed as the time for the review.

4. Guide to the official implementation: follow a sample from input to output

Read the fixed model card and tokenizer configuration first, confirm the chat template and special symbols, and then enter the Transformers apply_chat_template. This article revises the task sentences inside the user message, and the character package has not changed. Disable duplicate add special symbols when tokenization again after stitching, and record the hint hash and length. Source map List the verified permanent line numbers.

Read after LlamaForCausalLM.forward and generate greedy branch. The actual first entry is [1,83], hidden as [1,83,576], the vocabulary logits is [1,83,49152]. The three dimensions are the batch, sequence length and hidden width or the size of the vocabulary. The first generation token is the same as the maximum value of the last position logits. Checking can find common errors such as intercepting input prefixes, but does not prove that the answer is correct.

In order to allow the reader to restore the fine-tuning checkpoint independently, the entrance also contains 64-step training reconstruction. The official causal cross entropy will shift the label, we pass in the unshifted label, and only supervise the assistant to answer to the real end character, and the prefix and fill are ignored. Each step has 16 effective supervision tokens, totaling 1,024; When EOS and padding use the same number, the valid EOS must be retained according to the true length.

The four Transformers files that are actually installed and a PyTorch SGD file are consistent with the fixed official submission byte for byte. The teaching code is independently responsible for the experimental arrangement, masking and evaluation, and does not claim to reproduce the official complete training formula; the license, submission and the corresponding relationship with the local module are written into SOURCE_MAP.

5. Minimum Experiment and Source Code Reproduction Entrance

The public source code has been published and anonymously checks 56 files: Directory at the pinned commit、README and Reproduction Instructions、SOURCE_MAP corresponds to official code. The public confirmation record adds the real-person review conclusion, and the pending log at runtime is still preserved as historical evidence.

The model and tokenizer fixed revision as 12fd25f77366fa6b3b4b768ec3050bf629380bac; SST-2 and AG News are accurate revisions, and twenty resource file hashes are all in the resource lock. Emotional training 32 articles and 64 development articles will be divided into old divisions; 32 News development samples will be eight in each category. Article 128 Emotional and 64 News confirmation samples have never been reasoned. The development sample has been repeatedly analyzed and cannot be re-named as a test set .

After installing the lock dependency and executing the download verification, the minimum entry is as follows, and the directory is specified by the reader:

python run.py --cache ./cache --out smoke-new --private ./private-new --smoke
python audit.py --cache ./cache --out smoke-new

The test run still rebuilds all 64-step training, and only narrows the reasoning to two emotions and four news, a total of 32 generations. Expected products include step-by-step losses, checkpoint parameter hashes, case-by-case prediction, matching results and environmental records. The test run took twenty-seven points and four seconds, and the peak value was about 2.65 GiB; the loss of reconstruction parameters and all training steps was consistent with 091.

The 091 checkpoint that has been verified is directly reused in the formal experiment, and there is no new update; 640 generations take 92.7 seconds, and the peak value is about 1.60 GiB, which is the time after the actual measurement and import dependencies of the local CPU. The device is Apple platform CPU, Python 3.12.14, PyTorch 2.6.0, Transformers 4.49.0, four threads, single precision, greedy generation upper limit 32 Token, GPU memory is not measured.

The cached identity records the model, tokenizer, data, template, hints, decoding, code and checkpoint hashes at the same time. The parameter content is checked during recovery, and the file hash difference caused by the change of the serialized file name should not be misjudged as inconsistent training. The data card license is not clear, and the distribution package does not include the original movie review, news, weight or input token; it can be rebuilt by a fixed download portal.

6. Actual results: the same points may also have losses to each other

Each table in the following table is the number of qualified format, the correct content agent, and the correct number of joints.

Template number of samples fine-tuning before f/c/j F/C/J after fine-tuning
original JSON 64 0/50/0 64/58/58
rewrite task sentences 64 0/54/0 64/58/58
Swap order 64 0/53/0 64/55/55
original plain 64 64/57/57 64/33/33
Original News 32 30/8/8 29/8/8
rewrite news instructions 32 31/8/8 29/7/7

What we tested: after fine-tuning, the main comparison had one win, 62 ties and one loss, for a net difference of zero. Example 650 changed from a correct negative to an incorrect positive, while 736 improved in the opposite direction. Reporting only the improvement would hide the change in 650. Swapping the order produced one win, 59 ties and four losses, a net loss of three. Examples 237, 178, 436 and 571 all changed from correct negatives to incorrect positives, while 222 improved.

These outputs all satisfy the JSON convention, so this partial loss cannot be attributed to the JSON parsing failure. The format of the original plain has always been passed, but the content has dropped by twenty-four articles. The news is still close to the eight correct floors of the constant classifier on each type of equilibrium sample, and it is impossible to prove that it has obtained stable transferable news capabilities.

Input length and costs are listed separately in the log. The rewriting of the task sentence makes each movie review more than two input tokens; the total input token in the exchange order is the same as the original template, but the position is different. Seventeen of the rewriting templates before fine-tuning have reached the upper limit of the generation length, and there is only one original template. Therefore, the difference may be mixed with the output length and extraction behavior, and cannot be collectively referred to as "pure semantic sensitivity".

7. Failure investigation and verifiable research problems

We first checked for implementation errors in the weight hashes, example set, label mapping, raw prompts, truncation, and scorer. No input was truncated. We reparsed all 640 original outputs individually, and eight parser edge cases passed. For eight long outputs that might reproduce source material, the public package contains hashes and derived scores. The other 632 can be reparsed directly. The complete audit of raw outputs remains local, and the difference in public coverage is documented.

Re-arranged to explain cross-border. Author’s interpretation is that the update may have adapted fairly consistently to JSON responses without improving sentiment classification across all output conventions. That interpretation fits the table but has yet to be tested by a causal intervention. The results cover a single small model, one seed, a small English development sample, and a limited set of templates.

Two competition explanations can be checked in the future: first, the loss comes from the instruction position; the other, the loss comes mainly from the category bias caused by the output convention. After the manual transformation review is passed, the next step should be to register the position of the token length, and re-test on multiple training seeds according to the freezing scheme. If the loss direction under different seeds changes repeatedly, the judgment of the stable template effect should be withdrawn; You can't read the confirmation set and then change the template to find the winner.

8. Summary

The most valuable evidence this time is not a beautiful total score, but the content loss that still occurs when the format is completely correct, and the upper interpretation limit brought by the low baseline of the news. The body, source code, images and logs have been saved, and the user has also confirmed that the real person has passed the review. Computational verification and manual confirmation have records, and the original failure and exploration boundaries continue to be retained; after completing the acceptance of this article, the next article will enter the multi-seed research.

References

Retrieve and actually open the date: 2026-10-05; the following are all first-hand materials.

  1. Melanie Sclar、Yejin Choi、Yulia Tsvetkov、Alane Suhr,ICLR 2024: Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design,v2.
  2. Tianyi Yan et al., Findings ACL 2024: Contrastive Instruction Tuning.
  3. HuggingFaceTB: SmolLM2-135M-Instruct Fixed Model Card.
  4. Richard Socher et al., EMNLP 2013: Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank; SST-2 Fixed Data Card.
  5. Xiang Zhang、Junbo Zhao、Yann LeCun,2015: Character-level Convolutional Networks for Text Classification; AG News Fixed Mirror Data Card.
  6. Hugging Face and PyTorch: Fixed Generation Implementation、Fixed SGD implementation.