
Who to lend to the same temperature
There are a number of short movie reviews with tags. The classifier learns a temperature in this batch of data, which can soften the probability that is too sharp. If the language model is allowed to read the same batch of film reviews and answer the same positive and negative multiple-choice questions, can it be omitted to re-fit and directly borrow this number? This audit got a directional answer: lend the temperature of the classifier to the language model, and the loss of both templates has dropped. Borrow the temperature of the language model back to the classifier, both times are worse than maintaining the original probability. Even if the borrowing is effective, the problem of using it to put a negative label on all the samples is still intact.
The previous article changes the category composition of the calibration sample. This time, all calibration identities and labels are retained, and only the interface that generates the score is changed, and which interface the temperature comes from. For each target score, you can ask two questions one by one: Is the borrowed temperature better than not dealing with it? How does it compare to fitting the temperature on the target interface? The two references cannot be mixed into a sentence "migration is successful". The most recognizable evidence is that the same parameter will be evaluated differently under two references.
What are the three interfaces given?
The classifier is C. It is a dissilbert that has already been aimed at SST-2 fine-tuning, and each movie review generates two unnormalized scores, and the label order is fixed as negative and positive. The other two interfaces A and B use the same SmolLM2 instruction checkpoint. We give it an original English prompt, the end is fixed to Answer:, and only two sequential scorings are Negative and Positive with leading spaces. There is no sampling, no chat wrapping, and no terminator is included in the Answer. Therefore, the temperature here is to post-process the fractions that have been obtained, and adjust the sampling temperature during generation are two different experimental operations.
The two templates have only different orders of options in the question. A asked "Is this review positive or negative?" The actual time to take the time is always to take the negative and then take the positive. This detail is worth writing into the configuration: if the position of the output array is exchanged at the same time, but the old label mapping is still used, the errors in the opposite direction may be offset, and get a summary score that looks like a normal value.
For a sample, let two scores be , the difference is . The classifier provides these two scores directly. The language model gives the logarithmic probability of the two candidate words under the full vocabulary. Both can use the following formula to calculate the normalization probability of the front side:
here It is a positive temperature, and the index in the denominator only uses the relative fraction of two candidates. For the language model, the common full vocabulary normalization term disappears when it is subtracted; the probability of the two words is re-normalized, and the information about other sequels is given up. Even if the probability of two candidates in the vocabulary is very small, one of them may still achieve a very high conditional probability. Therefore, the correctness of the full text always refers to the two-choice result under the public label. This number cannot be used to estimate the fact that arbitrary open answers are Accuracy .
For the positive temperature, the sign of the difference will not change. If the two types of scores are equal, the same draw rule is used on both sides. If you only change the temperature but observe the predicted label flip, you should stop to check the numerical accuracy, category dimension and label mapping first. On the other hand, when the temperature is very high, the probability of the two types will be close to half. It can reduce the cost of "guessing wrong confidentially" and may also weaken the already high-confidence predictions. The two influences are based on the joint distribution of the target interface score and label.
Take the fit and application apart

The pre-registered SST-2 Validation will be divided. Sorted by the hashing of the stable sample number, skip the original 256 exploration samples, use the next 256 articles to fit, and the last 360 articles are used for protocol auditing. The two pools are separated. This public dataset has been used before, and the results of the classifier of the audit pool have also been analyzed before, and this time it is a controlled experiment of multiplexed data. There is no template, boundary, or parameter selected on the audit set, and the IMDb confirmation pool is not read. Whether the model pre-training has been exposed to a certain open text cannot be excluded by this experiment.
Each interface fits a number individually on the same calibration label. The main indicator is the average negative log-likelihood, referred to as NLL. It charges the probability of the real category according to the model, and the lower the probability of the real class, the greater the loss. use inverse temperature Write calibration goals:
here for the number of calibrations, take zero or one, is the -th example’s positive score minus its negative score. Scoring uses natural logarithms. The implementation uses a numerically stable form of the equation above to avoid taking the log of a probability close to zero. There is a single optimization variable. Its derivative is the mean, across examples, of the score difference times the difference between positive-class probability and label. We find the zero by bisection on the prespecified inverse-temperature interval from 0.05 to 20. A solution at either endpoint would be a constrained optimum. All three solutions in this run fell inside the interval.
After the three parameters are fitted, each is applied to the three groups of frozen audit scores, forming nine pairs. Plus the unit temperature of each group is added as the original fraction control. Each grid in the table covers the same 360 film reviews. Do not pick the lowest value among the twelve cells as a new deployment rule. The independent script starts from one record, and searches directly on the temperature axis according to the loss. The gradient or index function of the main program is not called, and then the loss of each grid, the number of positive predictions and the correct number are checked.
A cross table, read along the line first
| Audit interface for receiving temperature | Not calibrated | temperature with C | temperature with A | temperature with b | Number correct |
|---|---|---|---|---|---|
| C classifier | 0.38411 | 0.23714 | 0.53831 | 0.52833 | 330/360 |
| A ask positive first | 1.00483 | 0.73153 | 0.68979 | 0.68966 | 176/360 |
| B Negative first question | 0.92897 | 0.72383 | 0.69136 | 0.69131 | 176/360 |
The three fitting temperatures are 2.48192, 17.45244, and 16.23110. Read along the first line, all the numbers are output from the same classifier. Compared with the uncalibrated, the loss of its own temperature reduces the loss by about 0.147; the temperature of the transplant A is increased by about 0.154. Change the same number horizontally to different goals, and the effect direction is also changed. The classifier has answered 330 correctly in the audit pool. Excessive flattening will damage many samples with a higher probability of the true class. This is the same explanation as the data; to put the effect size to a specific semantic category, it is still necessary to do another group experiment.

The second line describes why both references are required. The classifier temperature improves A's unit temperature by about 0.273, but still pays an average loss of about 0.042 over a self-fitting of about 0.042. The borrowed temperature is in the right direction, but it is not enough to reproduce the improvement of the target. The third line is also in the same direction. The temperature of the two models cannot be used as a general parameter according to this, and the medium value of the compromise has not been fitted or verified by this protocol.
The templates supply a small counterexample. On A’s scores, B’s temperature gave a slightly lower loss than A’s own temperature, by about 0.00013. A fitted optimum applies to the calibration sample. A temperature fitted on the matching template needn’t win on every fresh sample. Nor does such a small difference establish that asking about negative sentiment first is better for calibration. We used one fixed split and made no population-level significance claim. Rounding to two decimals hides the effect, while displaying full precision risks making it seem more important than it is. The table presents the result, and the per-example file lets readers recalculate it.
Failure record that is more worth keeping than loss
The two templates predicted zero positives on 360 audit samples, and the 176 articles corrected correctly came from negative labels. After the temperature increases, the error is still there, and the model just gives some grasp of the wrong answer. As a reference for reading, when each of the two categories is half a half, the average loss is two natural logs, about 0.69315. The homologous calibration readings of the two interfaces are near this value. If only the report drops from more than one to about 0.69, readers can easily mistake the model to learn to distinguish film reviews. Predicted label counts make this explanation untenable.
This also limits how far the cross-template experiment can be said. The two templates share vocabulary tables, parameters and candidate words. The original loss after the question order change is indeed different, but the transplanting temperature is very close. The authors infer that one possible reason is that both interfaces are biased towards negative, and the large temperature pushes them to a near-uniform area. There is no third template that can generate a variety of predictions, and the effect of the same size on other tasks has not been tested. The next test hypothesis is that the gap between transplant loss and homologous loss may widen if another pre-fixed template can predict the two types at the same time when the identity and candidate words are unchanged. To verify that it uses an independent development protocol, it is not possible to flip through the audit samples to pick up prompts.
How to read the official code, how to run the minimum experiment
The background for temperature scaling comes from Guo and colleagues’ work on classifier calibration. They found single-parameter scaling effective in the settings they studied. In their repository, start with temperature_scale, which divides every logit in a row by the same temperature, then look at how set_temperature fits that parameter using validation loss. We independently implemented a bounded one-dimensional search. The paper did not test transfer between the two language-model templates used here. Every number in the table comes from this run.
When reading the model, first look at the projection of the last vocabulary along the LlamaForCausalLM of Transformers. The batch size is one, and the output is batch, context position, and vocabulary three-dimensional, we only read the last input position. Every time you put together the prompt and candidates and re-tokenize, check the prefix is exactly the same and have a token at the tail. If the candidate is split into multiple tokens, the current entry will stop, because at this time, the conditional probability of each token must be accumulated. The classifier reads the two-dimensional output of DistilBertForSequenceClassification. The existing classifier record is locked by hash, and the 6116 sets of real scores are reused this time, and the read cache is not counted as a new forward.
First use eight movie reviews, each two templates to run smoke runs, wait for the identity, length, tokens and output shapes to be checked, and then run the official two pools. The CPU is four-threaded, single-precision, and fixed attention to be realized; the official stage is 1,232 new scores, including resource verification and loading for about 43 seconds, and the resident memory of the process peak is about 1.63 GB. This time starts after the framework is imported and cannot be directly compared to the time-consuming containing download and interpreter startup. The records include identity, tag, context length, shape, candidate tokens and scores, and the original sentence or model weight is not distributed. The data card is not explicitly given a license, and the reader takes the number from the original entrance according to the fixed version.
Code and instructions for reproducing the results
Public fixed submissions include Source directory、README and Formulas and official implementation comparison table SOURCE_MAP.
Run in the source directory python run.py && python audit.py, you can reconstruct the result table and independent check records with the real score of the archive. To check the neural network itself, perform the analysis in the new copy in the order of README download, smoke, and formal scoring, and then run the analysis. Fixed versions and file summary are saved in resource locks and protocols, respectively. The entry for redrawing the heatmap is plot.py, and the input is result.json compiled by log by log. Only the "recalculation indicator" should be reported when running the analysis, and only the entry of the model is executed to be included in the new forward direction.
When troubleshooting, first look at which layer of mismatch. If the sample identities or text summaries have changed, they should go back to the data version and collation. If the loss is different but the labels are exactly the same, check whether the temperature or inverse temperature is used, and whether the order of the positive and negative classes is forgotten. If the outputs of the two templates are exactly the same one by one, first compare the tokens sequence that was finally sent to the model, and then check whether the cache key contains the template and code version. Finally, the optimizer is considered. There may be differences in the floating-point mantissa of different machines, and saving the complete tokens and score can help distinguish numerical perturbation and interface errors.
Summary
In this batch of film reviews, whether it is possible to borrow the temperature, it is necessary to clarify the source of the parameters, receive scores and compare baselines at the same time. The temperature borrowed by the classifier eases the overconfidence of the two prompts, and the temperature of the two prompts borrows back damages the classifier. Almost the same calibration loss between the two prompts, and the fact that they all miss all the positive labels, should be reported together. Next, when studying Abstention, I have to ask separately: what samples are left, and how many errors are left? This probability loss table cannot answer that question.
References
Searched on 2026-11-06.
- Guo, Pleiss, Sun, Weinberger, 2017, the original experiment of classification probability calibration and the temperature scaling method. ICML Papers Page
- Pleiss et al., the author of temperature division and Validation set fits. Fixed source code
- Hugging Face, pinned version of classification checkpoint and instruction model. DistilBERT SST-2、SmolLM2-135M-Instruct
- Stanford NLP, the official label and data file of the film review two categories. SST-2 Data
- Transformers 4.49.0, the language model forward and vocabulary output. Llama implementation
Apps by the author
I also build small tools for everyday life. Here are two of my apps:
- Daily Expense Journal: Track your income and spending, plan a budget and set savings goals. View on the App Store.
- One Small Action: Pick a small action that fits the time and energy you have today, then record how it felt and what you noticed. View on the App Store.
Find more articles and projects on my website: zffjzz.top. I’d love to hear how the apps work for you and what I could improve.