Teachers deliver soft distribution and hard goals to students on the laboratory cover

  • Date: 2026-09-07
  • For graduate students, early-career researchers and research engineers

Abstract

If a student model behaves more like its teacher, does it become more accurate? Drawing on the papers and official implementations for classic knowledge distillation, sequence-level distillation, and MiniLLM, we trained a small teacher and four groups of students. We examined how the supervision signal, temperature, and choice of evaluation metric affect the conclusion. Soft targets lowered negative log-likelihood but failed to beat the accuracy of training on true labels. Students also inherited some teacher errors. This checks the loss mechanism and experimental protocol without claiming to reproduce the papers’ large-language-model results.

Reproducing value: first ask what the teacher has handed over

Knowledge distillation refers to the supervised training student model provided by the teacher model. Teachers can hand over the full output probability, or they can only hand over the generated text. The former allows students to see the gap between the "second possible answer" and the highest score answer, and the latter saves a specific behavior trajectory. The student's capacity is small, the teacher has deviations, and the migration samples are not covered, which may limit the income; the few parameters are only structural attributes, and the effect must still be independently tested.

Paper report: Hinton, Vinyals and Dean discuss soft goals and temperatures in the 2015 preprint, and show the knowledge transfer in images and voice tasks; this is not the theorem that all students can compress teachers without loss. Original paper

Kim and Rush's EMNLP 2016 work has elevated oversight to the full output sequence, training students with the teacher bundle search. Beam search is an approximate decoding of multiple candidate prefixes at each step; it does not guarantee to find the most globally probable sentence. Modern response distillation, that is, in response to distillation, also often uses teachers to generate texts as training labels, but the specific sampling and filtering strategies cannot be automatically equivalent to the original paper. Sequence stage distillation

Core ideas and formulas: probability, sequence and loss direction

Logit is the score before normalization. For the same input, the logit of the teacher and the student is zt,zs∈RB×Cz^t,z^s\in\mathbb R^{B\times C}, in BB is the batch size, CC is the number of categories. Define:

pT=softmax⁡(zt/T),qT=softmax⁡(zs/T),L=(1−α)CE⁡(y,q1)+αT2KL⁡(pT∥qT).p_T=\operatorname{softmax}(z^t/T),\quad q_T=\operatorname{softmax}(z^s/T),\qquad L=(1-\alpha)\operatorname{CE}(y,q_1)+\alpha T^2\operatorname{KL}(p_T\|q_T).

T>0T>0 is the temperature, which increases it usually makes the probability more gentle; yy is the length of BB true label, ce is cross entropy, α\alpha Control the weight of teachers' supervision. KL is the measure of the difference between the two probability distributions, where the category is summed and the sample is averaged. When the teacher is fixed, the soft cross entropy is the same as the KL in this direction, and the student's gradient is the same. Multiply T2T^2 It is used to compensate the gradient scale at high temperature, and it cannot be understood that the higher the temperature, the more knowledge. Formulas and Temperature Discussion

In response to distillation, the teacher-generated y^=(y^1,…,y^L)\hat y=(\hat y_1,\ldots,\hat y_L):

Lresp=−1M∑b,jmbjlog⁡qθ(y^bj∣xb,y^b,<j),M=∑b,jmbj.L_{\rm resp}=-\frac{1}{M}\sum_{b,j}m_{bj}\log q_\theta(\hat y_{bj}\mid x_b,\hat y_{b,<j}),\quad M=\sum_{b,j}m_{bj}.

LL is the upper limit of the sequence length, m∈{0,1}B×Lm\in\{0,1\}^{B\times L} Identify valid response tokens, the prompt and the fill position do not participate in the average, and whether the end tokens is reserved must be clear; MM cannot be zero. The model output shape is [B,L,V], VV for the vocabulary size. The input is already prefixed and predicted for the next tokens during training, and the causal shift must be aligned. The complete text can be re-coded by the student's own tokenizer; the tokens match the probability of the two models directly by tokens requires the semantics and context to be aligned.

MiniILLM studies reverse KL, i.e. KL⁡(qθ∥p)\operatorname{KL}(q_\theta\|p), and supporting design sampling and optimization. Its sequence expects to depend on the distribution of students, and the function parameters cannot be exchanged on the fixed teacher response, so that the whole method is claimed. This article uses 2024 v4; the current arXiv v6 has changed the title to the version with On-Policy. MiniLLM v4、Version history

Soft Goal and Response Two Supervised Routes

The two routes in the figure are alternative choices: the probability enters KL, and the response enters the tokens cross entropy; the loss points to the students to indicate training supervision, and the teacher keeps freezing. Independent Test Set checks the students after training and are not used to generate training targets. The conceptual relationship of graphic shows does not represent the language model that has been trained this time.

Guide to the official implementation: find the denominator along the effective token

Read the official tutorial of PyTorch first train_knowledge_distillation: The teacher enters the evaluation mode, forwards in the context that does not record the gradient, and the students keep the gradient; then check the temperature on both sides and the mixed weight. Evaluation mode only changes module behavior such as dropout, and does not block gradient composition in itself. This is the framework teaching code, not the original implementation of the Hinton paper. Official tutorial

Read LMOps again minillm/finetune.py. get_distil_loss Calculate the soft-cross entropy of the teacher softmax and the student log-softmax in parallel branches in non-model parallel, using label != -100 Exclude invalid targets, and finally divide by the number of valid positions; this function does not have an explicit temperature scaling, and the temperature three experiments in this paper cannot be pretended to be the official default value. training entry kd_ratio Mix the real label with teacher loss. KD source code

The response route first generates and processes the pseudo-response by the SeqKD section of the repository README, and then runs fine-tuning . miniIllm route from another trainer.py Trace Rollout to collect, respond to logit slices and model updates, read again losses.py Migrate loss and regularity; these modules describe the difference between it and the offline response fit. This article only reads, and does not start distributed training; the repository main is not locked for submission, and the compatibility of the follow-up operation is waiting for manual verification. Official repository、trainer、Loss Realization

Minimum experiment: real training, clearly narrowing the scope

code/minimal_distillation.py Only rely on the Python standard library. The data is generated by the two-dimensional coordinates and fixed nonlinear scoring rules to generate three-category labels. Teachers fit eighteen parameters on six-dimensional polynomial features, and students fit nine parameters on three-dimensional linear features; this is a real-optimized classification model, but not a pre-trained neural language model.

Teacher Training Set 360 articles, 90 migration sets, and 240 test sets, which are generated with different data seeds. The teacher trains 400 steps and freezes, and the students all train 250 steps, and the learning rate is 0.12; the three running seeds are 66, 67, and 68, and only the initialization is changed. Under the same seed, four groups of students started from the exact same weight. Temperature 1 and 3, soft supervision weight 0.7 are pre-fixed, no test set is required.

PYTHONPYCACHEPREFIX=/private/tmp/codex-ai-066-pycache python3 -m py_compile code/minimal_distillation.py code/torch_loss_check.py
python3 code/minimal_distillation.py > code/smoke_test.txt

What we tested: Python 3.9.6, CPU; teacher characteristics [360,6], student characteristics [90,3], test logit [240,3]. The following table is the mean of three initializations, NLL is the negative log-likelihood of the real label, the lower the better; all evaluation probability use temperature one.

Student Supervision accuracy Real Tags NLL KL with teachers
real labels only 0.9542 0.2100 0.0438
Only the highest score for teachers 0.9333 0.2165 0.0485
Real label plus soft target, temperature one 0.9500 0.2223 0.0495
Real label plus soft target, temperature three 0.9500 0.1887 0.0347

Teacher Accuracy is 0.9792. The standard deviation of Accuracy in each group of students is zero, and the NLL still has tiny fluctuations, and the complete value and weight are stored in code/results.json. Zero fluctuation only shows that these three initializations get the same number of correct predictions, and do not indicate that the results have no statistical uncertainty.

Temperature 3 improves the probability indicator, but does not improve the relatively true label baseline Accuracy . The hard teacher branch does not have real label mixes, so the table cannot individually prove that the "soft probability form" is better than the hard target; strict ablation should additionally match the mixing ratio of the label. There are five errors in the teacher's test, and each student copied four of the errors. This slice is too small and is suitable as an alarm rather than a stable error rate estimation.

This time, we also passed KL self-comparison, direction asymmetry, large logit stability and finite difference gradient check; the maximum gradient error is about 7.94×10−127.94\times10^{-12}. torch_loss_check.py Provides effective tokens, teacher detach and mask gradient control, but both the two Python environments of this machine lack torch, only the syntax check is passed, and the operation is waited for manual verification. Classification hard label branches are not equal to complete Response distillation, there is no text generation, sequence training or original paper Benchmark.

Evaluation protocol: similarity cannot replace correctness

For a full replication, freeze the teacher checkpoint, data sources, tokenizer, and chat template. Split by question family into training, development, and test sets, then deduplicate and check for contamination. Use the development set to select the temperature, weight, and stopping step. Report test results only for the frozen configuration. Our experiment has no development set and uses the final step specified in advance. That suits a mechanism check and leaves the optimal configuration open.

At the same time, report task indicators, NLL under real reference, teacher consensus rate and error copying of teacher error questions. The language task is to fix the decoding temperature, the maximum number of new tokens, and the stop condition, and save the original response; Only the teacher's wording should be repeated, and high scores should not be automatically scored.

There are also two types of fluctuations: repeated initialization measurement optimization sensitivity, and repeated sampling test problems involve sample uncertainty. In this article, the three seeds share the same test set, so the standard deviation cannot act as the Confidence interval of the overall accuracy. The real report can be paired and resampled at the topic level, and retain the corresponding relationship between the baseline and the distillation model on the same question; multiple rewritings of the same family should be used as a sampling unit to avoid using relevant records as independent evidence.

The cost is also fair: in addition to the number of student training steps, it is also necessary to record the teacher's pre-training or fine-tuning cost, the number of tokens, the number of soft logit caches and peak memory. The number of fixed samples is not equal to the number of fixed training tokens. Both the hard response and the soft distribution line should be compared with the real label baseline of the same architecture and the same initialization, and then make a fixed tokens budget comparison; record the data volume and loss form separately, and avoid giving the extra supervised income to the formula.

Failure investigation: check the direction first, then check the data

When the KL value is abnormal, first confirm the function input requirements. The input of PyTorch is usually the student's logarithmic probability, and the goal is the teacher probability; batchmean Divide the two-dimensional categorical tensor by batch size, but the direct use of the three-dimensional tensor will not be automatically divided by the active tokens. The supporting implementation first selects the effective position to become [M,V], then calculate the average; the empty response directly reports an error. KLDivLoss Documentation

Caches also cause concealment errors. When saving soft targets, the sample number, tokens number, mask, teacher version and temperature should be recorded at the same time; only the input and probability are connected according to the file line number, and after reordering, it is possible to train other people's targets. If only the maximum probability is cached, and re-normalization will change the original distribution, it must be explained how to deal with the quality of the missing probability. Response data keep at least prompt, original generated text, generation parameters and filtering reasons, and you can't just leave the cleaned answer.

When the training is smooth but generates a degradation, print a complete input, shift label and mask, and check whether the prompt is counted as a loss, the end mark is dropped, or the repeated shift is repeated. Teachers use the evaluation mode and freeze, and random cropping should also be consistent with pre-processing. If students are more like teachers but are more incorrect, check error copying, data coverage and teachers' performance in the target field, and do not rush to increase the weight of distillation.

Questions for further research

My hypothesis is that when student capacity is limited, the most valuable supervision may come from just some of the regions where the teacher is uncertain. Group examples by teacher confidence, compare bin-specific temperatures with a fixed temperature at the same budget, and flag the teacher’s errors separately. Separate ablations could show whether any gain comes from filtering out errors or from retaining the information in soft targets.

Another problem is the prefix distribution. The offline response only covers the path that the teacher has taken, and the free generation of students will encounter their own deviations. In the next step, the supervision of the teacher's prefix and the student's prefix can be compared, the budget is fixed and the error occurs; this is gradually approaching the problem of online distillation, instead of directly transplanting the conclusion of the classification experiment to the long text.

Summary

Distillation reproduction needs to explain at the same time what the teacher provides, what input the student learns, and who judges the result. This time, a replayable soft and hard supervision control was established, and the fact that Accuracy did not improve and the teacher's error spread was preserved. In the next step, the tensor operation verification and real sequence experiments should be supplemented, and the independent task evidence should continue to be used to judge the income.

References

Search date: 2026-09-07. The above links are all directed to the first-hand information opened this time. The core papers are Geoffrey Hinton, Oriol Vinyals, Jeff Dean's "Distilling the Knowledge in a Neural Network" (2015, arXiv v1); Yoon Kim, Alexander M. Rush's "Sequence-Level Knowledge Distillation" (EMNLP 2016 finalization); Yuxian gu, li "MiniLLM: Knowledge Distillation of Large Language Models" by Dong, Furu Wei, Minlie Huang (ICLR 2024, this article read arXiv v4). The title and version changes of the current v6 of miniIllm are also recorded in the body version. The code link is the retrieval day main, and the PyTorch loss interface adopts a fixed 2.12 document; the official tutorial is a dynamic page, which depends on the compatibility with the model running for manual verification.

  1. Hinton et al., 2015, original paper PDF.
  2. Kim and Rush, EMNLP 2016 conference page and Full text.
  3. Gu et al., MiniLLM 2024 V4 and official code.
  4. PyTorch official distillation tutorial and loss function documentation.