FINE-TUNING biaxial baseline check

  • Date: 2026-10-01
  • For graduate students, early-career researchers and research engineers
  • Scope: True SmolLM2-135M-Instruct, English emotional classification, two fixed output conventions; no training, not complete benchmark evaluation.

Summary: First know where "improvement" may come from

The accuracy improvement after fine-tuning may be that the model will judge the emotion more, or it may just finally output as required. If two things are mixed into one score before training, it is difficult to explain the meaning of loss after training. This article first establishes the starting point that can be disassembled and checked: the same movie review, the same weight, and only the answer is a single tag or a JSON object. JSON is a text format that expresses structured information with key-value pairs.

In this actual verification, the pure label condition is correct 57/64, and all of them meet the format; the JSON condition can extract the correct emotional words 50/64, but none of them meet the complete format convention. This gap is worth studying, but it cannot be written as "small models will not structure the output": we only measured one checkpoint, two prompts and a small group of development samples.

1. Reproducing value: leave a comparable starting point for subsequent fine-tuning

The previous unit studied the ranking and migration of the search method, and 087 has completed the cross-collection evaluation of the frozen configuration. Starting from this article, the task of the research object is changed to a small model Fine-tuning; the multiplexing environment, version locking and one-by-case auditing methods, and the search score is not connected to the classification score. The new question is: How will answering format changes affect the task ability we measured? The new product is an original answering ledger that can be replayed.

The Instruct checkpoint has already undergone the developers’ instruction training. Here “before fine-tuning” means before our own task-specific fine-tuning. The model card says that the series was instruction-tuned on public and curated data and points to smol-smoltalk. [1] Our experiment neither reproduces that training process nor establishes that the data are free of contamination.

The Smol-Smoltalk in the outline is suitable for subsequent research data mixing, but the open answer lacks the simple and definite correctness labels required in this article. Therefore, this article adopts the SST-2 two-category film review with source tags, and Smol-Smoltalk has not been downloaded or trained. SST papers study emotional combinability; we only use its open dicate to export, and do not reproduce the original paper tree model. [2][3] English experiments cannot be extrapolated as Chinese ability.

2. The core idea: to dismantle compliance, correct answer and joint success

For the I-sample, let F_i indicate whether the answer is in the format, and C_i means whether the unique emotional tag drawn from the answer is equal to the source tag:

F=1N∑iFi,C=1N∑iCi,J=1N∑iFiCi.F=\frac{1}{N}\sum_i F_i,\qquad C=\frac{1}{N}\sum_i C_i,\qquad J=\frac{1}{N}\sum_i F_iC_i.

Here n is the number of all evaluated samples, and each condition is 64 this time; F, C, and J are the format compliance rate, label Accuracy and joint success rate. Failure answers cannot disappear from the denominator. In addition, accuracy in the compliance answer should be recorded, but "undefined" should be recorded when the number of compliance is zero, and the division of zeros cannot be written as zero points.

Pure labeling requires that the blanks at both ends are removed just to be lowercase positive or negative. JSON requires complete parsing successfully, only one key of Sentiment, and the value is the above lowercase label; repeated keys, additional fields, and code fences are not qualified. For content extraction, the case is ignored and the complete words are found; only one emotional word is given for prediction, and both missing or both are wrong.

This C is an automatic proxy metric for "expressed labeling correctly", not a manual semantic judgment. For example, a negative explanation may mislead the extractor. Therefore, the four combinations are worth keeping: the format is paired but the label is wrong, the format is wrong, but the label is paired, both are right and both are wrong. The evaluator itself also needs to be checked, and cannot be automatically upgraded to true value.

Frozen model and format, content separation evaluation process

3. Guide to the official implementation: chase from the character boundary to the next word

First read the config and tokenizer_config of the fixed model, and then read the three entrances of Transformers, the specific line number and license to see the source map. [1][4] Tokenizer cuts the text into token; the chat template organizes the roles of System, User, Assistant, etc. into a model familiar to the model. This article explicitly gives the same SYSTEM message, and sets ADD_GENERATION_PROMPT, so that the input ends at the starting point of the assistant's answer. The SYSTEM text that the template is filled by default is therefore not implicitly adopted.

Then LlamaForCausalLM: the input token ID is hidden by the model, and then projected from the language model output header to the vocabulary. The actual shape of the first case is [1,72] → [1,72,576] → [1,72,49152], and the three axes are the batch, sequence length and hidden dimension or the size of the table. Shapes come from real forward records, not speculated by the model name.

Finally read the greedy branch in Generate: each step selects the token with the largest score at the end, until the end mark or the upper limit is generated. This time the sampling is turned off, a bunch of searches are set, and the first generation token is also consistent with the direct forward argmax check. Although the function name in the official source code is called _sample, the argmax branch is still taken when the sampling is closed, and the protocol cannot be judged by the function name only. [4]

The local program directly calls the official implementation; the three key installation source codes are consistent with the pinned commit byte. Scoring, division and auditing are independent teaching implementations, and the source map does not wrap these custom rules into official evaluators.

4. Minimum Experiment and Evaluation protocol: Freeze first, then see the answer

The model is fixed on the same revision as the tokenizer 12fd25f77366fa6b3b4b768ec3050bf629380bac; SST-2 fixed in 8d51e7e4887a4caaa95b3fbebbf53c0490b58bbb. Download the file to record SHA-256 one by one. The original validation of the data has a total of 872 pieces, sorted by the hash of the string formed by the seed 88 and the original IDX, the first 64 items are developed, and the latter 128 are reserved for confirmation, and the rest are not used. Confirm that only the ID is registered, no running model or selection case; official test has not been downloaded.

The two conditions keep the sample, task description, tag order, weight and template unchanged, and only replace the output requirements; there are no examples. CPU, float32, four threads, batch one, up to 32 tokens are newly generated. If the input is not truncated, it will report an error if it exceeds 512 tokens; this time it is not triggered. The model has 134515008 actual parameters, and the weight file is about 269 MB.

The first order to miss the test run switch, and the development set was completed in advance. We keep this record and do two more samples with two correct test runs: a total of about 0.91 seconds, a peak of about 1.25 GB, and a linear estimate of 900 seconds, 4 GiB Within the memory budget, the official re-run. The original and formal predictions and scores are the same, but they are not two independent sampling experiments; all results maintain the identity of "development and exploration".

The official operation record is 13.05 seconds in total, the inference stage is about 12.69 seconds, and the peak is resident in memory 1252720640 bytes. The timing starts after the import of the third-party library, including cache verification and model loading, does not include download and process startup; no GPU memory is measured. The environment, command, exit code and staged state are all saved, and the speed of this small experiment cannot be used to ensure the time-consuming of other machines.

5. This actual verification: zero in the format does not mean that there is no task signal

Indicator, 64 samples in the same set pure label JSON requirements
Format Compliance F 64/64 0/64
Explicit labels are correct C 57/64(89.06%) 50/64(78.13%)
Joint success J 57/64 0/64
Unable to pull out unique labels 0 1
Maximum length of generation length 0 1

Constantly predicting positive classes can get 33/64. This is not a strong model control, but prevents the class ratio as an available signal. The pure label is significantly higher than the reference within this sample, indicating that there is a behavior worthy of continued measurement; Confidence interval has not been calculated, and the overall significance is not declared accordingly.

One case pairing is more explanatory: the JSON condition has 4 improvements to the correctness of the label, 49 unchanged, and 11 deteriorates. The number of pairs of negative classes changes from 26/31 to 29/31, and the alignment class changes from 31/33 to 21/33. The decline in the mean is not all the samples falling together, nor should it be called neat "ability degradation". These groups are from keeping records, and there are no re-select prompts accordingly.

The total number of tokens generated in the two groups is 128 and 422, respectively, with end marks; the total input is 4546 and 5250. JSON has more than 11 tokens per prompt, so this time it is estimated to change the overall effect of the output convention, not strictly to wait for the comparison of token costs, and there is no causal contribution to isolate word order, length and grammatical complexity.

In particular, it is necessary to avoid only resolving answers that can be resolved and then count Accuracy: that will quietly replace the measured population, and it may also make the method that is more often refused to answer. All the original samples in this article enter the denominator, and the joint successfully measures whether the entire answer can be used directly, and if the label is correct, it provides a more relaxed diagnostic perspective. The questions answered are different and should be presented in parallel. Even if the automatic repairer is added in the future, the answer before the repair should be saved, and the repair cost and evaluation rules will be clearly written, otherwise it will not be able to recognize and improve from the model or post processing.

6. Failure investigation: keep the answer, and then put forward an explanation

The case selects the smallest case of IDx in each non-empty cell according to the pre-rules. IDX 34 answers an English-language explanation containing Negative under JSON conditions: the label is consistent with the source, but the entire string is not the required object. IDX 123 answers uppercase Negative, neither the structure is adhered to and the label does not match the source of the sample. Pure tag IDX 322 outputs legal positive, but misjudgments the category. The three cases correspond to different failures, and they cannot all be attributed to "the model is too small".

The investigation order should first look at the input role boundary, label mapping, only intercept the newly generated part, end conditions, and then look at the task error. This independent audit checks the label and sample ID from the original data, rebuilds the answer from the output token, recalculates the indicators, and checks the eight boundary use cases such as duplicate keys, fences, and double labels. There is no artificial addition of labels, and no program auditing has been written as an artificial gold standard.

Only one of the 64 JSON failures hit the length limit. A short budget might therefore explain an individual failure, but it cannot explain all of them. Whether the model prefers direct answers or needs an example of the required structure remains conjectural. Without a controlled comparison, neither explanation establishes a mechanism.

7. Propose testable scientific research from the baseline

In the next article, first check the supervision position: Does the training loss cover the prompts, labels, structural symbols or end marks? If only strengthened format supervision will increase F, but C does not improve, the evidence will support the adaptation of the output convention; if the tag Accuracy is also improved after the format is frozen, there is a reason to continue researching task learning. The two interpretations require controlled experiments with different supervised masks to distinguish, and now there is no training result.

These development IDs, templates and evaluators must be used in the future to record any changes; the training sample size and effective supervision token are separately controlled. If according to the error adjustment prompts in this article, the adjusted scores are development choices, and cannot be tested repeatedly by retaining the confirmation set. The 128 confirmation samples can only be used after the final plan is frozen; the restriction of public data that may enter through upstream training still exists.

Code and instructions for reproducing the results

This article has been released in a fixed version: Full source code directory、README and run instructions、SOURCE_MAP official source map. The 47 files have been pulled anonymously and checked with the local byte-by-byte hash, and the audit and real trial run of the public copy are passed; these links are fixed on the same submission and are not updated with the master branch.

After installing fixed dependencies in README and running the downloader, the minimum command is python run.py --cache ./cache --out smoke-new --smoke. It runs two real samples, two answer conventions, and generates one-by-case answer, aggregation, shape and resource records; remove the test run switch to redo all 64 development experiments. Independent audit entry is audit.py. This time, the actual resource acquisition, trial run, complete development reasoning and original output recalculation are completed, and there is no training or reserved set reasoning.

The SST-2 data card is marked as unknown, and the source code package does not redistribute the original movie review; the download path, revision and hash can be checked. The cache identity also includes models, tokenizers, prompts, decoding and executing source code, and the old answer cannot be misused by mistake.

Summary

A Fine-Tuning baseline with research value should make subsequent improvements can be dismantled and questioned. This article has obtained the task signal, and also retains the negative results under the structured requirements. The next step is not to immediately look for more beautiful scores, but to convert "learning format" and "learning judgment" into supervised experiments that can be tested separately.

References

Retrieval and actual access date: 2026-10-01. The paper report, the above actual operation and the author's inference are separate statements; the subsequent training that is not run is subject to manual verification.

  1. HuggingFaceTB: SmolLM2-135M-Instruct Fixed Model Card; smol-smoltalk official data card. It is used to confirm checkpoints and official training materials, but this training has not been reproduced.
  2. Richard Socher、Alex Perelygin、Jean Wu、Jason Chuang、Christopher D. Manning、Andrew Ng、Christopher Potts,2013: Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank,EMNLP.
  3. Stanford NLP: SST-2 fixed data card original text. The fixed original text is actually downloaded and verified; the webpage rendering entrance has failed, and it is not allowed to guess according to this.
  4. Hugging Face,Transformers 4.49.0: Chat template source code、Llama output header、Greedy Generation Branch.