Experimental Metaphor for Temperature Calibration

Reproducing Value: From Manual Cooling to Isolation Fit

The previous experiments have shown that the same set of classification answers can be completely constant, with different probability quality. But try several temperatures, and then pick out the best-looking results, and do not answer the real research question: If only one calibration data that is isolated in advance is allowed, can the temperature learned improve the probability of the other data? This article only changes this part. The model weights, tokenizers, label mapping and inference configurations are all frozen, and the newly added reviewable products are the matching indicators on the calibration pool prediction, the temperature optimization trajectory and the audit pool.

Let's talk about this actual verification first: the fixed disilbert emotion classifier, get the temperature of about 2.48 on the 256 calibration samples. Subsequently, this number was applied to the 360 protocol audit samples. Negative log-likelihood and calibration errors decreased, and the correct classification number was maintained for 330. This result supports "one isolation fit can improve the probability quality of this division", and does not support "the universally reliable Confidence has been obtained", and it does not mean that the model has learned to recognize all errors.

Core idea: only change the sharpness of probability

The classifier first outputs two unnormalized scores, called logits, and then uses softmax to convert to probability. temperature scaling divides all scores by the same positive number before normalization:

pik(T)=exp⁡(zik/T)∑j=1Kexp⁡(zij/T).p_{ik}(T)=\frac{\exp(z_{ik}/T)}{\sum_{j=1}^{K}\exp(z_{ij}/T)}.

Here the sample number is ii, the category number is kk, the number of categories K=2K=2. The score shape of a batch entered is [B,2][B,2], temperature TT is a positive scalar. If the temperature is greater than one time, the probability of the two types of probability is evenly distributed; if it is less than one time, the probability is more sharp. It changes the strength of the model's speech, and does not change the magnitude relationship of the class score.

Therefore, as long as the positive temperature and consistent parallel processing rules are maintained, the label corresponding to the maximum probability will not change. The original wrong sentence is still wrong. What we expect to fix is "the wrong answer reports the probability that it is close to one", rather than emotional understanding itself. The probability here is for a fixed positive and negative two-classification task, which is different from the limited candidate normalized score of the generative model discussed in the previous article, and it cannot be directly interpreted as the correct probability of open generated content.

The fitting target is Negative Log-Likelihood, or NLL for short:

T^=arg⁡min⁡T∈[0.05,20]1nc∑i=1nc−log⁡pi,yi(T).\hat T=\arg\min_{T\in[0.05,20]}\frac{1}{n_c}\sum_{i=1}^{n_c}-\log p_{i,y_i}(T).

Among ncn_c is the number of calibration samples, yiy_i is the original label of the public data. The smaller the probability that the real label is assigned, the greater the penalty. The selection rule is written to the protocol before the new pool inference; the temperature is not selected with the audit error, the optimal range is retained after trying multiple ranges, and there is no optimization of the boxing error.

Paper report and guide to the official implementation

The 2017 paper of Guo, Pleiss, Sun and Weinberger discusses the Confidence calibation of modern neural networks and reports Temperature in the data and models of its research the effectiveness of scaling. That is the scope of evidence for the paper report. This article uses another fixed text classification checkpoint and a smaller exploration division, so it does not repeat the original paper value, and does not refer to its own results as a complete paper benchmark reproduction. Papers and Publishing Information

When reading the author's code, you can find it first temperature_scale, confirm that the logits are removed, and then look at set_temperature: Collect the verification score and label first, and then optimize a single temperature with cross entropy. The author implements the use of GPU and LBFGS; this article does not copy the entire training stack, but writes an independent teaching solver for the two classification goals. The specific fixed commit and function positioning are recorded in the source map. Author realize

The reason for local rewriting is reviewability: let the inverse temperature be β=1/T\beta=1/T, and the difference between the two types of original scores is di=zi1−zi0d_i=z_{i1}-z_{i0}, then the first derivative of the target is:

g(β)=1nc∑idi[σ(βdi)−yi].g(\beta)=\frac{1}{n_c}\sum_i d_i\left[\sigma(\beta d_i)-y_i\right].

Among σ\sigma is a sigmoid function with a label code of zero or one. The derivative is monotonous with the inverse temperature, so you can use the dichotomy method to find the zero point in the pre-registered interval; if there is no zero point in the interval, the boundary solution is reported. This fixed execution of 100 times and two points and each step is saved, and the number of iterations is not regarded as the number of model training rounds. The temperature does not touch the boundary, and the final gradient is about zero under floating point accuracy.

Minimum experiment: freeze the identity first, then open the score

The data follows the 872 records of the official SST-2 Validation. There are registered hashes according to the sample number and fixed prefix, and then sorted into 256 Articles, 256 calibrations, and 360 agreement audits. This time there was no reshuffle, and no "more typical" calibration samples were selected from the exploration pool. The identity list of the three pools has been saved during the baseline experiment, and this one-by-one verification does not overlap.

It must be disclosed: SST-2 has been used in historical research. The new division can only isolate the information flow that this round is fitted, and cannot erase historical observations, nor can it be called a test set that has never been seen before. In the future, the agreement audit pool is also an observing development evidence, and can no longer be packaged as an independent final confirmation. IMDb Sanchi did not download or predict this time, and the final confirmation pool must still abide by the follow-up registration and use rules.

The model is the official Distinbert SST-2 checkpoint of the official revision, and the weights, configuration and vocabulary are checked through the file hash; the data is also checked for the fixed parquet file hash. Label zero is negative, one is positive. Inferences use CPU, single precision, four threads, eager attention, batch 16, up to 512 tokens, and turn off the training mode. Do not update any network weights, only a number of post-processing is optimized.

Isolate Fit and Audit Process

The two pools in the figure represent the set of fractions for the corresponding data to be frozen. The upper lane is fitted with temperature and freezing, and the bottom lane only applies this number to calculate the indicators; the audit results do not return to the fitting node. First perform the first eight test runs of the calibration Pool, and then run the complete calibration Pool, and then execute the audit Pool after the temperature file is dropped. Each sample holds the number, text hash, original tag, two-dimensional logits and probability, and no longer distribute the original statement.

Evaluation protocol and this actual verification

The main indicator is the audit pool NLL, and the auxiliary indicators include the Brier score in the double-category summation form, the expected calibration error ECE of ten equal-width boxes, the average Confidence and Accuracy . Brier is the sum of the two types of squared errors, and the dimension range is zero to two; it cannot be mixed with single-item versions calculated only for the positive class. ECE uses the left and right out of the box, the last box contains the right end point, and saves each box of the denominator.

Audit pool indicator, sample number 360 raw probability Calibration Probability
Number correct 330 330
accuracy 0.916667 0.916667
Average Confidence 0.982405 0.918592
NLL 0.384113 0.237136
bi-class Brier 0.154001 0.135151
ECE10 0.073274 0.019900

The fitted temperature is T^=2.4819242151\hat T=2.4819242151. The NLL of the calibration pool itself has dropped from 0.4037 to 0.2426, but this is not an independent effect verification; the real need to pay attention to the audit results after freezing the temperature. The difference between the audit average Confidence and Accuracy is reduced to about 0.019, and it is still not possible to claim that all probability intervals are calibrated just by this overall mean.

The pre-fixed 90% threshold also revealed the boundary: the original probability of 343 of the original probability reached the threshold, of which 23 were wrong; after the calibration, only 279 were left to reach the threshold, and ten of them were wrong. Accept the collection to become smaller and the error ratio is reduced, but the reserved error does not disappear. This is only a descriptive statistics of a fixed threshold, which has not yet constituted an abstention threshold selection experiment, and there is no correction to the selection deviation caused by repeated observation.

The calibration stage of this complete reasoning is about 1.15 seconds, and the auditing stage is about 1.37 seconds; Depends on download, initial import and environment repair are not included in this time-consuming; no measured energy consumption, service throughput, or hardware peak video memory. All results are from the real checkpoint forward pass, and the beam in the figure is not a measurement curve.

Independent recalculation and failure investigation

The independent audit script does not import the main experimental module, and recalculates all the indicators from Logits with sigmoid, stable SoftPlus and the two-class Brier identity; and also uses the gold segment directly for temperature to search for review and optimization results. The temperature difference between the two solutions is about two times the negative eighth power, and the maximum difference between the indicators is about eight times the negative seventeenth power of ten. It verifies deterministic arithmetic consistency, is not called a real person review, and cannot replace the semantic annotation quality judgment.

The environment recovery left a real failure record: the temporary environment was missing PyTorch; the default dependency cache write is limited, and the first network installation fails. The import of the old environment of the work area stagnates, and then the task process is stopped, and the fixed dependencies are installed in the isolated temporary environment and complete the test run. Another startup to parse the soft link of the virtual environment interpreter into a basic interpreter, resulting in invisible dependencies; continue after the execution path is corrected, and there is no model experiment with synthetic data to replace the model experiment.

When the reproduction fails, first check whether the resource hashes are consistent, and then look at label mapping, inference pattern and logits shape. If the temperature becomes negative, the classification answer changes or the optimization result falls on the boundary, the implementation and data should be checked first, and should not be directly explained as a calibration finding. If the calibration goal is improved and the audit goal is deteriorated, the result should also be retained; after the re-adjustment of the scope, it will be reported to the same audit pool, and the audit data will be gradually turned into a reference data.

Code and instructions for reproducing the results

The full source code has been made public: Directory at the pinned commit、README、SOURCE_MAP. Twenty-six files downloaded anonymously are consistent with the local and compressed packages one by one, and the independent audit of the open copy and the real eight test runs have passed. The package includes a complete entry, dependencies, resource locks, division lists, protocols, source maps, two-pool-by-sample results and independent audit procedures. After the reader obtains the public source code, install a fixed dependency in the source code directory, first execute the resource acquisition, and then run the real model:

python run.py prepare --cache .cache117
python run.py smoke --cache .cache117
python run.py run --cache .cache117
python audit.py

The expected products are two-cell prediction file, temperature and optimization trajectory, aggregation indicators, shape and environmental recording, and independent recalculation results. Existing fixed resources can be reused after the verification cache, no need to download the model repeatedly. The weight of about 270 megabytes of the model is obtained through the official entrance, and the source code package is not entered;

There is also a confusion-prone check: the independent solver obtains a similar temperature, which does not mean that the temperature is also stable after resampling. The former checks the calculation, and the latter checks the estimated fluctuation of data changes. This article only completes the former, and does not use repeated sampling to select better results. If readers want to compare different machines, they should first align the input identities, model files and probability calculation accuracy, and then compare the sample-by-sample scores; only compare the final rounded indicators, which may cover up label dislocations or mix files. The meaning of saving the failure log is also here: it allows the environment repair and scientific selection to be tracked separately, avoiding writing a command that has not been run into a validated entry.

Research questions and summary

Separating fitting from evaluation tells us more here than the particular temperature estimate. On this fixed split, one scalar clearly improved probability quality without changing a single predicted label. That makes the benefit of calibration a concrete observation others can recalculate. The incorrect semantic decisions and a few high-confidence errors remain, and the experiment cannot establish reliability on an unseen distribution.

A follow-up could test whether the temperature remains stable as the calibration sample shrinks. Fix the nested samples, seeds, and aggregation rule in advance, keeping the audit set out of temperature selection. Another hypothesis is that a few severe errors dominate the NLL fit. That calls for a preregistered influence analysis. The overall mean alone cannot settle it. We ran no significance test, offer no population coverage guarantee, and draw no rule for all models from this single result.

References

Retrieve and actual access date: 2026-11-02.

  1. Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger. 2017. On Calibration of Modern Neural Networks,ICML,PMLR 70,1321–1330.
  2. Author codes such as Geoff Pleiss: Temperature_scaling.py fixed commit, MIT; this reading control, the solver is independently implemented.
  3. Hugging Face: DistilBERT SST-2 Fixed Model Card.
  4. Stanford NLP: SST-2 Fixed Data Card.
  5. Hugging Face: Transformers fixed classifier implementation,Apache-2.0.

Apps by the author

The following is the author's own product promotion. In addition to writing scientific research blogs, I also independently develop some daily tools, welcome to understand according to your needs: Daily Expense Journal Used to record revenue and expenditure, arrange budgets and save goals, Go to the App Store; One Small Action Help you choose small actions according to time and energy, and record your feelings after completion, Go to the App Store. See more technical articles and personal works Author website. These introductions are independent of the experimental conclusions of this article and do not represent the recommendation of the author of the paper.