Cover: The short film and the long film are rolled through the same optical reader

A temperature, a letter from a group of readers

Suppose you have a scale, and the "almost certain" of the model in the short review of the movie has been adjusted to "more sure". What I received today is the entire movie review: praise the performance at the beginning, criticize the editing in the middle, and give the general review at the end. The most common experimental mistake at this time is to use a few long reviews to adjust the scale again, and then announce that it can be migrated. This article asks a question that can be used to deny numbers: take away the temperature learned in short sentences without moving, and the probability that the model will allocate the real labels of long film reviews, on average, will it become better?

The answer has two layers. In the 1,000 IMDb exploration film reviews registered, the fixed temperature has made the average Negative Log-likelihood drop from 0.4410 to 0.2863, the direction is the same as the audit pool of short sentences; but the loss in the long evaluation is still higher than that of short sentences, and the proportion of fixed threshold reception has also changed. In other words, what can be taken is the improvement of the sample relative to the original score. What cannot be taken away together is the stronger promise of "change the corpus, and each Confidence still corresponds to the original Accuracy". The following two judgments are opened for inspection.

what was frozen and what changed

In the previous article, the collection structure method was changed on the same batch of short sentences to observe which labels would be included. This article fixes the model as the English dissilbert two classifier of fine-tuning on SST-2, and changes the evaluation corpus without changing the model, tokenizer, reasoning accuracy or truncation upper limit. The tags are still negative and positive; the tags provided by the data author are used, and there is no set of "looks" reviews based on the model's answer. Both sets of corpus talk about movies, which makes the task similar, but it does not mean that the labeling process of short sentences is the same as the grading mechanism of the entire film review.

The source domain refers to the corpus environment used to learn this temperature. In the historical SST-2 verification division, the isolated 256 calibration samples are used to obtain a temperature of 2.481924; the comparison is to use another 360 protocols to audit samples. It is not the first time they have appeared in this series and cannot be packaged as untested. This time, I directly read the frozen temperature and audit output, and does not retrain, and does not count the forward operation of the previous article's model into today's expenses. The target domain refers to the environment to be moved in. This time, only the IMDb model Forward Pass is added.

IMDb's original paper introduces the emotional classification data of film reviews. The official test is divided into 25,000, which does not mean that they need to be sent to the model here. In order to prevent multiple evaluations of the same film from falling into different research stages, we first index the movie URL in the archive and align it with the file serial number; at most one sample of the same film. In addition, the text of the newline marks, case, blank and characters of the text are normalized, and the repetitive text is also reserved at most one copy. Normalization is only used to check for weights, and the original text is really given to the model. It cannot recognize all rewrites, let alone judge whether the two comments express the same meaning.

When implementing, first sort by the salt hash of the file identity, and keep the candidates whose video and text are not occupied in turn. Take three paragraphs in sequence, each with a thousand identities, and conduct development exploration, target domain calibration and final confirmation respectively. Write the same three pools list before any model prediction. The selection basis does not use the model score, nor is it leveled by the target label. The estimated object here is the film review under this sampling rule; according to the film quota, the weight of the popular movie will be changed, and this set of points cannot be used as the official full score. Confirm that the pool only registers the identity, does not predict, does not look at the output, and does not pick cases.

Freeze the source domain settings, explore data only enter the evaluation, and keep the two pools sealed

How to measure "also effective"

The model outputs two unnormalized scores for each input, called logits. temperature scaling refers to dividing both scores by the same positive number, and then normalizing it into two types of probability. Let the shape of a batch of score matrix be B×2B\times 2, in BB is batch size. For a sample, the first kk The probability of a class is:

pk(T)=exp⁡(zk/T)∑j=01exp⁡(zj/T).p_k(T)=\frac{\exp(z_k/T)}{\sum_{j=0}^{1}\exp(z_j/T)}.

Here zkz_k is the original score of this class, TT is the fixed temperature. If the temperature is greater than one time, the score difference will be reduced, and the probability that is close to zero or one will move to the middle. The size and order of the two scores remain unchanged, and the highest score label will remain the same. This gives both an implementation check and explains why there is no need to answer one more question in terms of the quality of probability. For example, for the same wrong prediction, give 1% of the correct label and give it 10% of it, which is only one mistake in Accuracy; in tasks that need to believe this probability, the cost of the two is different.

The main metrics pre-specified are Negative Log-Likelihood, or NLL for short. It is the probability that the model gives the probability of the real label to take the negative natural logarithm, and then averages between the samples. Calculate separately for each corpus:

L(T)=−1n∑i=1nlog⁡pi,yi(T),D=L(T)−L(1).L(T)=-\frac{1}{n}\sum_{i=1}^{n}\log p_{i,y_i}(T),\qquad D=L(T)-L(1).

Among nn is the number of samples of the pool, yiy_i is the ii -th example’s original label, DD It is the difference between the two temperatures on the same sample, and the negative value indicates improvement. First make the difference in each domain, and then compare the direction; do not match the two groups of different film reviews according to the line number. The auxiliary indicator has the sum of the squared errors of two types of probability, that is, the Brier score of the class sum, and the calibration error ECE of the ten equal-width Confidence intervals. The latter is weighted according to the number of samples in the box, and compares the average Confidence and Accuracy in the box, and the weight of the empty box is zero. After the binning is fixed, there may still be a cancellation in the box, and the change after a decimal point cannot be regarded as a calibration certificate.

Minimum Experiment and Guide to the official implementation

First look at how the official classification head is represented from the first position of the last layer, and two scores are obtained through the linear layer, activation and classification layer. There is no "positive" or "negative" word for the model to generate the two words "positive" or "negative", and there is no authenticity score for open answers. Then look at the three-segment code in the supporting entrance: the registration function only generates identity; the forward branch only allows to explore the identity in the list; the scoring function reads a fixed score and label to calculate the probability, loss and number of received. The advantage of this sequence is that it is easy to check: if you only change the temperature of the prediction category, check the correspondence between the column and the label first, and do not rush to explain the field offset.

The model is cut to the evaluation mode, the gradient is turned off, and the four-threaded CPU, single-precision, and right-side truncation are used. Each batch is at most sixteen, and the longest input is made to the maximum, and the upper limit is 512 tokenization units with special symbols. First, use the first eight lines of the exploration pool to do a real test run, check the output of two columns, the value is limited, and then run the entire 1,000. Eight test runs are included in the official list, and cannot be merged into one thousand and eight different samples. The official forward loop contains tokenization and batch processing time of 58.99 seconds; about 65.11 seconds from the reading of the parameters to the saving result, and the process peak resident memory is about 2.14 GB. The order of magnitude is given here to allow readers to estimate the budget, not cross-hardware speed comparison.

Results: Improved migration, the meaning of the threshold has changed

corpus and temperature Number correct NLL Brier ECE Ninety percent threshold: number of errors / number of receptions
SST-2 original 330/360 0.3841 0.1540 0.0733 23/343
SST-2 Fixed temperature 330/360 0.2371 0.1352 0.0199 10/279
IMDb original 890/1000 0.4410 0.1955 0.0891 89/945
IMDb fixed temperature 890/1000 0.2863 0.1693 0.0285 23/634

Compare the average of original and fixed temperatures in the same domain Negative log-likelihood

Each pair of bars comes from the same actual predictions, calculated from the two saved score columns rather than taken from a published curve. The change in loss was about −0.1470 for short sentences and −0.1547 for long reviews. All three probability metrics improved in both domains, while accuracy held steady in each. The fixed temperature corrected some of the original model’s overconfidence. Whether it is optimal for IMDb is still open, as is whether the gains were equal across domains. We did not run an equivalence test.

Then look at the last column, it only answers "how much is wrong when the model is at least 90% of the selected label". In the long-term commentary, there were 945 articles, and only 634 articles were received after zooming; there are 66 errors in the 311 articles removed, and there are also 245 correct answers. This is more useful than a single report "missing from 89 to twenty-three" because it lists the cost of avoiding mistakes. After the calibration, the reception rate of the short sentence is 77.5%, and the length is 63.4%; the error rate in the receiving set is about 3.6% and 3.6% respectively. Approximately the same conditional error rate masks the difference in reception scale and cannot become a risk commitment on a new corpus.

Look at the highest-confidence bin for long reviews. Before scaling, it held 945 reviews, with mean confidence of about 99.2% and accuracy of about 90.6%. After scaling, it held 634, with mean confidence of about 94.5% and accuracy of about 96.4%. The model was even slightly underconfident in the latter bin. It would be misleading to say that the same group of reviews went from overconfidence to underconfidence. A bin can stay in the same place on a reliability diagram while the reviews inside it change.

Next Steps for Failure Troubleshooting and Verifiable

The first test run did not generate any logits: the interpreter can still be found in the temporary environment, but the import framework fails. We have reserved the failed exit code, and after the independent environment is reinstalled by the locked version, we have reused the written identity list. Another obstacle occurred during the download phase: the author site failed to establish a connection, and the other sorting version did not have the desired video index. Finally, use the fixed image of the complete original package, and the byte summary is the same as the official data tool record. If the video index is missing, only check whether there are the same file name in the two pools, it cannot prove that there is no same video; this is also the reason why this experiment cannot be directly replaced with any two tables.

For the study explanation, the most tempting statement is to attribute the difference to the long text. The record can only support a narrower judgment: the median of the tokenization units actually retained by the short sentence and the long review are 23 and 229 respectively, and the long-term review has 112 of the long-term limit, and the short sentence does not. To reach the upper limit does not mean that the ending must be lost; just looking at the reserved length, there is no way to distinguish "just put it down" and "really cut off". Similarly, although the official high and low scores can be mapped to two columns, although the difficulty is not necessarily the same. There is no random allocation of the sample length here, and there is no real person to re-judgment which sentence determines the direction of the relationship.

A follow-up assumption that can be overturned is that shortening the readable prefix after a fixed film review identity will change the score difference of some samples, and may change the probability loss before the highest score label is flipped. When verifying, use the same film review as a pair, first register the length budget, and then record the actual tokenization number, whether the answer is overturned and the loss changes. If the probability change occurs only in a few critical samples, it should also be retained as a matter of fact. The current cross-library comparison raises this question, but the intervention has not yet been implemented. Public texts may also appear in pre-training; the "sealing" here refers to the experimental discipline of this project, not a zero-pollution certification for training history.

Code and instructions for reproducing the results

Open The source code of this fixed version, from Reproducing the results enter the experiment, or in Source map Check the paper formulas, official classification headers and local implementations one by one.

After installing the lock dependency, execute it in the source code directory:

python run.py analyze && python audit.py

This minimum command reconstructs the result table from the true scores of the package, and independently checks the identity, two column probability and all counts. For full download, registration, eight test runs and formal reasoning commands, please refer to the reproduction instructions. This time, the formal reasoning of the exploration pool was indeed performed, and the temperature was not fitted in the IMDb; the source domain only reused the recorded predictions. The recalculation script adopts two columns of exponential normalization, which is different from the binary score difference of the main script, and checks forty boxes and four sets of indicators. It can find that the subscript is wrong or the denominator is mixed, and it cannot replace the reader to judge whether the research hypothesis is reasonable.

Summary

The temperature of the short sentences to learn can still alleviate the probability penalty of mis predictions. The more useful evidence is that there are three things left at the same time: a thousand fixed identities, two original scores for each, and how many errors and correct answers are in and out of the threshold. They allow the next step to only change the reading length, test the differences that cannot be explained today, and leave the final confirmation pool for the real confirmation experiment.

References

Search date: 2026-11-17.


Apps by the author

I also build small tools for everyday life. Here are two of my apps:

  • Daily Expense Journal: Track your income and spending, plan a budget and set savings goals. View on the App Store.
  • One Small Action: Pick a small action that fits the time and energy you have today, then record how it felt and what you noticed. View on the App Store.

Find more articles and projects on my website: zffjzz.top. I’d love to hear how the apps work for you and what I could improve.