Cover illustration

  • Date: 2026-10-06
  • For graduate students, early-career researchers and research engineers
  • Scope: real small model, five fine-tuning, development set exploration; not a complete benchmark or final confirmation experiment.

Summary: First ask what the seeds have changed

In the previous chapter, it was found that the same checkpoint was changed to a task statement, and the correct total number may remain unchanged, but the specific sample will be exchanged for the outcome. This fixed expression, in turn, asks: Retrain once, can the conclusion be maintained? We continue to use the price of data mixing: learn new tasks, will you forget the old abilities? 》Selected only the target task training configuration, add two fixed sequence controls and three real rearrangement seeds, and retain all the prediction and failure boundaries.

The actual verification of this time: The correct number of JSON joints for the three rearrangements is 58, 57, and 58, all of which can output legal formats; the correct number of correct output of the original words has dropped from 57 to 33. The fluctuation between training is small and does not eliminate the cost between task agreements. What is more vigilant is: only the seeds are changed without changing any random process, and even the numerical weights are the same for the two trainings. Calling such repetitions as multiple random experiments exaggerates evidence.

Why reproduce this work and where it fits

Random seeds are the starting point of the pseudo-random number generator; it is not a complete definition of experimental variables by itself. Initialization, data sequence, dropout, and sample decoding may all be affected by randomness, but the seed may not have the opportunity to work when freezing pretrained weights, closing dropout, fixed order and decoding with greedy. PyTorch's reproducing documentation also distinguishes between random number control and other non-determinism, and does not guarantee that different platform and version bits are consistent. [3]

Paper report: Reimers and Gurevych pointed out in the 2017 sequence labeling study that a single score is not enough to represent random training methods, and advocates the distribution of multiple runs. [1] In this paper, this research problem is borrowed, and its LSTM experimental numbers are not transplanted, and the three small models are not used as a large-scale reproduction of the original paper. In 2018, Dror et al. discussed the matching relationship between statistical test and task, indicators and experimental design. [2] Here, choose a simple and auditable pairing difference, and deliberately do not chase the significance label.

Compared with 092, the new question is "whether the gain and loss are reproduced at the same time after the training sequence is changed; the change variable is the sequence and the pairing, freeze the model, data, hints, supervised volume and scorerer. The new product is the seed-sample result matrix, training trajectory, weight hashing, and two different statistical summaries.

Core idea: two uncertainties cannot be mixed into an error line

Let the binary result of the baseline on the sample be bib_i, first ss The result of the training is asia_{si}, then

dsi=asi−bi,Δs=1n∑i=1ndsi.d_{si}=a_{si}-b_i,\qquad \Delta_s=\frac{1}{n}\sum_{i=1}^{n}d_{si}.

here ii is the same evaluation sample, ss is a trained checkpoint, dd It can only be negative one, zero or one; the matrix shape of this SST result is [3,64][3,64], three rearrange runs share the same baseline. Pairing requires comparison of the same row, rather than subtracting the two total Accuracy after each resampling.

An intuitive example is: Model A and B both answered fifty-eight correct, and it is impossible to infer that they answered the same batch of questions correctly. If Jia Duo saves one example and loses another example, the total score is completely invisible. The progressive results can be saved at the same time, and the prompt hash, tag, stop cause and generate tag can be distinguished, and the real behavior change, input mismatch and scorer change can be distinguished. When comparing, first confirm these identities are consistent, and then calculate the difference, which can avoid mistakes in mistakes.

The first layer fixed these 64 lines and reported three Δs\Delta_s Mean, range and sample standard deviation:

Δˉ=1S∑sΔs,SD=∑s(Δs−Δˉ)2S−1.\bar\Delta=\frac1S\sum_s\Delta_s,\qquad SD=\sqrt{\frac{\sum_s(\Delta_s-\bar\Delta)^2}{S-1}}.

S=3S=3, the denominator is two; this standard deviation describes the fluctuation of this training sequence, which is not the overall mean, and cannot contain the model initialization or data sampling factors that have not been tried. Two fixed sequence controls do not mix these three runs.

The second layer is fixed after a training checkpoint and baseline, and there is a sample from the same row of indexes. Calculate the mean of each pairing, and then take the percentile interval. This kind of bootstrap, that is, resampling, is done 10,000 times this time. It only shows the sensitivity of the fixed model to the current sample composition, and does not represent the uncertainty of retraining. It is not possible to spread the three predictions into 192 independent test cases; the news tasks are balanced in four categories, and samples are sampled according to the label category.

Multiple seeds and pairing evaluation methods

In the figure, Seed Spread only summarizes three rearrangements to run; Sample Bootstrap is for a single fixed model pair. The shared evaluation Sample only enters the result calculation, and does not reflow to the checkpoint selection.

Minimum Experiment and Freeze Protocol

We use SmolLM2-135M-Instruct, an instruction-tuned model with about 134.5 million parameters. The model and tokenizer are pinned to 12fd25f77366fa6b3b4b768ec3050bf629380bac, not from random weight training. SST-2 fixed revision is 8d51e7e4887a4caaa95b3fbebbf53c0490b58bbb; ag News is eb185aade064a813bc0b7f42de02595523103ca4. The actual SHA-256 and download address of the twenty files are saved with the source code. [4–6]

We reused 32 sentiment training items, 64 sentiment development items, and 32 news development items. The established hash rule still selected the training rows, without a fresh search for easy examples. We ran no inference on the 128 sentiment or 64 news confirmation items. Some news items from the official test split had already informed development choices in 091, so those items no longer qualify as unseen test data. The normalized exact-duplicate check passed. It did not establish that the reviews came from independent films, and we did not screen for pretraining contamination.

The five groups all start with the same original weight, and the full parameter SGD learning rate is 0.001, gradient cropping 1, batch 2, 64 steps. The JSON answer in each line has a total of 8 supervised tokens, 1024 supervised tokens in each group, and 10556 input tokens, and each row appears four times in the four rounds. Prompt and fill locations do not count the loss, and the real terminator is reserved. CPU four threads, single precision, Eager attention, retain gradient but do not enable dropout; greedy to generate up to 32 new tokens.

Fixed groups 93 and 94 are in the original negative orthal alternating sequence, and only the frame seeds are changed. Rearrangement 93, 94, and 95 each round to disrupt the 16 lines of the two categories respectively, and then combine them one by one to ensure that each batch is positive and one negative. Therefore, the order is paired with the batch, and the category ratio and exposure are not changed. The total number of tokens with fill is 11252, 11298, and 11338, and the control group is 11344; the same input amount is still not equal to the same calculation amount.

The agreement is registered before reasoning, and the best seeds are not selected, nor are they satisfactory until the results are satisfied. The five groups of small test runs were generated in a total of 10 steps and 48 times, taking 18.31 seconds, and the peak memory was 2.62GB; the estimated conservative multiple was estimated at 439.32 seconds, which was officially implemented after the budget was lower than the 1200-second budget.

A guide to the official implementation

Read the fixed version first apply_chat_template, confirm the boundary between the user input and the Assistant answer; then look at the local build and collate, check that the real EOS and fill share the number 2, but are distinguished by length. follow-up LlamaForCausalLM Have found ForCausalLMLoss: input as [B,L][B,L] and output logits as [B,L,49152][B,L,49152], where the last dimension is the vocabulary, and the label needs to be aligned with the previous position prediction. [7]

In each step, the official loss and manual shift cross-entropy are checked, and the maximum error is about 1.19×10−71.19\times10^{-7}. Finally, read the official SGD update, and then go back to the independent teaching scheduler to confirm that the seeds actually enter the Python sequence generator. The five official source files have been compared bytes with the installed version; permanent files, function line numbers, submissions and licenses are concentrated in the source code map, without floating main positioning.

Statistical code is also read in addition: read first d=a-b Whether to align according to the same sample, then check whether the heavy sampling matrix acts on the two models at the same time, and finally check the denominator of the standard deviation. Correct reasoning and statistical interpretation are two checks, which cannot be replaced by each other.

The actual verification and matching results

The formal experiment was updated in a total of 320 steps and generated 960 times, all of which were successful; it took 267.90 seconds, and the peak RSS of the process was 2961309696 bytes. After importing from a third party, it includes verification, loading, training, saving, reasoning and statistics; GPU memory is not measured. There is no concealed training failure, and no acknowledgment set parameter is used.

Status Word output joint correct/64 JSON format is successful/64 JSON content and joint correct/64 News Joint Correct/32
Original checkpoint 57 0 Content 50, joint 0 8
Fixed 93 33 64 58 8
fixed 94 33 64 58 8
Rearrange 93 33 64 58 8
Rearrange 94 33 64 57 8
rearrangement 95 33 64 58 8

Format success refers to the agent that satisfies the strict output convention; the correct content is only a proxy that is the only explicit label in the output that matches the data label, not an artificial semantic gold label. The joint correct requirements of the two are met at the same time. The average value of the three rearranged JSON combined with Accuracy is 90.10%, and the sample standard deviation is 0.90 percentage points.

Just looking at the content, the rearrangement of 93 relative to the baseline is 12 wins, 48 draws, and 4 losses, with a net increase of 12.5 percentage points. The interval is [0,25] a percentage point; rearrange 94 is 11 wins, 49 draws, 4 losses, and the interval [0, 21.875]. The output of the two is only different: the output of the number 481 is different: the former answers positive, the latter answers negative, and the data label is positive. Keeping this index is sufficient to locate the difference, and there is no need to reprint unknown movie reviews in the public package.

On the contrary, all the training words after training are 5 wins, 30 draws, and 29 losses, with a net drop of 37.5 percentage points, and the conditional interval is [-53.125, -21.875]. The JSON gets better and the word answer gets worse and reproduces at the same time. The joint right and wrong before and after the news will remain unchanged, and the interval is [0,0]; it only shows that the difference value of this batch is completely zero, and the performance of the original 8/32 floor is not enough to support "preserving the ability of news understanding".

Troubleshooting and limits of the evidence

First check the "fake multi seeds": the 64-step loss and gradient of the fixed group are the same as 091, and the two numerical weights have the same hashing; the three rearrangement weights are different. Saving the file name will change the PyTorch container hash, so the tensor name, shape and numerical bytes are also calculated to calculate the identity, and avoid serialization difference as a model change.

In addition, the rearrangement did not re-draw the training samples, so the fluctuation of the training set selection was not measured. The three seeds also did not form three new baselines: the original model is only inferred once, and then the progressive results are shared. This shared relationship must be kept in tables and statistical scripts. In the future, if the sampling of data sets is added, new random factors and independent units should be registered separately, and the current standard deviation cannot be used to explain.

Check the "Operation Proof of Interval Promotion": 64 development samples have been repeatedly used for selection and explanation, and resampling cannot change them back to unseen data. Interval endpoints cannot be simplified to "completely invalid" when they encounter zeros; the design here only supports ranged descriptions. Extrapolation is restricted for cross-sample correlation, few training seeds, and a single small model.

How much can a reader recompute from the public evidence? All 960 raw outputs were independently parsed locally. The public release redacts 11 long answers that might reproduce news text, retaining each answer’s hash, length and derived scores. The other 949 can be reparsed in full. The original audit and distribution notes document the redactions, and readers can generate a complete set from their own run. No human scores were fabricated.

Testable Research Questions and Summary

**My interpretation:** The model appears to have adapted strongly to one output convention while shifting its label preferences under another. The experiment does not isolate the internal mechanism. One follow-up would freeze the configuration and test once on the held-out confirmation set whether the effect has the same direction. A separate development experiment could compare class-balanced shuffling with ordinary random batches to see whether stability depends on the pairing scheme. Both require registering the rules before seeing the confirmation results.

This article is not recommended to pick a 58-point seed to publish. It is worth delivering: the random source of real change, all operations, the winner on the same line, and the ability to be without the ability to be masked by the error bars. At this point, the second unit has obtained a set of auditable fine-tuning assets; the next article will check how the official evaluation task configuration itself defines the score.

Code and instructions for reproducing the results

The complete browsing source code has been published to the public repository Directory at the pinned commit. From README Install dependencies, obtain fixed resources, and then press SOURCE_MAP Check the official implementation with the local module. This version contains 57 files; the public copy has been anonymously pulled and SHA-256 is checked with the local source code and archives.

After preparing to depend and download fixed resources, the minimum command is:

HF_HUB_OFFLINE=1 TOKENIZERS_PARALLELISM=false python run.py --cache ./cache --out smoke-new --private private-new --smoke

It actually trains five groups and preserves gradual losses, sample output, budget, pairing statistics and weight identities; the official operation is removed --smoke. The complete five groups and the original audit have been completed this time, and the five sets of test runs after the archives are decompressed in the temporary directory are also passed. The source code comes with all the self-written modules, and the multiplexed verified cache is not required to open the author's local path.

References

Search date: 2026-10-06; review before release: 2026-10-07. The following papers support method motivation or data source, and the numbers in this article are all from the accompanying logs.

  1. Nils Reimers、Iryna Gurevych,2017,EMNLP: Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging.
  2. Rotem Dror、Gili Baumer、Segev Shlomov、Roi Reichart,2018,ACL: The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing.
  3. PyTorch 2.6: Reproducibility.
  4. HuggingFaceTB: Fixed SmolLM2-135M-Instruct model card.
  5. Richard Socher et al., 2013, EMNLP: Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. Fixed data files and hash see source code resource lock.
  6. Xiang Zhang, Junbo Zhang, Yann LeCun, 2015, NIPS, page revisions for V3: Character-level Convolutional Networks for Text Classification; This fixed AG News data card.
  7. Hugging Face Transformers 4.49.0: Implementation of Fixed Causal Language Model Loss. For the rest of the official functions and licenses, see SOURCE_MAP.