This article is actually completed: fixed public DistilBERT checkpoint, re-inference for 256 development samples of SST-2, save one by one and independently recalculate. There is no training, no fitting calibration parameters, and the fact that the classification probability is not interpreted as an open question and answer is the factual reliability.

1. Why is the answer correctly, it is not the complete answer
Seeing more than 90% of a model Accuracy is easy to think it is reliable enough. But there is another question in the research: is the degree of grasp given by the model itself consistent with the actual performance?
If a model is close to 100% confident in a large number of answers, and in practice, it still often makes mistakes, then its classification ability and probability expression are two things that need to be checked separately. The former answered "Is it the right one to choose", and the latter cares about "whether the score can be regarded as a credible probability". A beautiful Accuracy, can't automatically prove the second point.
This is not to label the model "untrusted" first. This article first establishes a set of observations that can be recalculated by others: fixed model, fixed input, retained original score, and then compares Confidence and correctness. After the baseline is firmly established, calibration, abstention and statistical guarantees will be discussed later. Otherwise, even if the problem is not determined at which level, it is easy to mistake the change of the scoring method into the ability improvement.
Today's results have a contrast worthy of questioning: the model has answered 233 correctly, and the accuracy is 91.02%; the average Confidence of the same batch of samples is 98.17%. These two items are the results of this real operation, not the numbers on the model card, nor are they concluded from other people's paper tables.
2. The research object should be small, and the boundary of the problem should be clear
We choose the Distinbert SST-2 emotional classification checkpoint released by Hugging Face. The input is an English film review text, and the output is the score of the two categories of negative and positive. It is a pre-trained encoder through the emotional task fine-tuning, not a chat model; the model card clearly describes the task and language scope. Model card
It was not chosen to chase the latest rankings, but to control the research variables. There are only two label spaces, the official tags can be directly checked, and the model can run on the machine. We first study a probabilistic problem that can be audited clearly, and then consider complex generation scenarios.
This also draws a conclusion to the boundary: the overconfidence in the English film review classification cannot be directly launched in Chinese research questions and answers, and it is impossible to estimate the hallge language model's hallucination rate according to this. Open-generated answer spaces, correctness definitions and scoring methods are all different. When comparing the generative model later, it is necessary to re-explain what the score represents, and not just put them into the same ranking just because both numbers fall between zero and one.
The data comes from the SST-2 verification division released by Stanford NLP, with a total of 872 pieces. Its research background can be traced back to the emotion tree library, but this article uses the off-the-shelf two-class classification task, which does not reproduce the recursive model of the original paper. data entry; original paper
Another must be disclosed in advance: this dataset has already appeared in the previous blog, and the checkpoint author has also reported the performance of the Validation Set. Therefore, this article is an exploratory experiment of reusing public development data, not "Test Set that has never been seen", nor has it proved that the training data is zero pollution. Re-scrambling the data numbers will not erase this history.
3. Where does confidence come from
One batch reasoning gets the shape as logits, It is the number of batch samples, and the two components correspond to negative and positive directions. Logit is a model score that is not normalized. We use softmax to convert two probability, and then take the larger one as the prediction confidence :
Here, is the -th example’s score for class score, is the normalized probability, is the prediction category, is the maximum category probability. The actual calculation uses a stable log-softmax to avoid numerical problems caused by exponentialization first.
Note that softmax guarantees two non-negative and added one, and there is no additional evidence that "this probability must match the real accuracy". The model outputs 0.99, which does not mean that the researcher has verified the reliability of 99%.
The classic calibration paper of Guo et al. expresses the problem as: whether the correct proportion of the same Confidence is matched with the correct proportion of the Confidence. This paper borrows this measurement framework and does not claim to reproduce the full model, dataset or temperature scaling results of the paper. Section 2 of paper
4. Freeze the agreement first, then see the results

The above figure is a schematic of the process and does not contain experimental measurements. In particular, pay attention to the label path at the bottom: the standard answer only enters the evaluation link, and is not input as a model inference. The probability container in the figure is only a concept map and does not represent any actual ratio.
This round is sorted according to the original number of the sample, and the rule is to calculate SHA-256 for the string "cycle7: numbering" and arrange in ascending order. The first 256 articles were used for this exploration, the next 256 were reserved for the calibration pool, and the remaining 360 were used as agreement audit pools. The numbers of the three sets are saved before the first reasoning; no model predictions for the two pools are generated today.
This is done first for traceability, rather than letting random division bring magical independence. It can prevent picking samples after seeing the results, but it cannot solve the problems of data history reuse, model authors may have used Validation Set, and the correlation of homologous texts. Subsequent domain experiments will still indicate the exploratory nature.
The freezing conditions of this article include: Model and Tokenizer Fixed revision, maximum length 512, batch 16, CPU single precision, four calculation threads, fixed random seed 112, turn off the training state dropout, and use eger attention. No different prompts were tried, no new weights were trained, and no thresholds were adjusted to the result.
Do 8 test runs before the official operation, and only check the links and resources. The test run is from the same development pool, so it does not count as an independent test. After that, the pre-selected 256 pieces will be completely processed, and the sample will not be replaced just because the test run looks good or bad.
5. Real results: the score is very high, and the error still exists
| Metric | This result | how to understand |
|---|---|---|
| Correct number and Accuracy | 233/256,91.02% | The classification performance on these 256 items |
| Average Maximum Category Probability | 98.17% | The model's average grasp of the category you choose |
| Average Confidence minus Accuracy | 7.16 percentage points | The overall directional gap is not a complete calibration measure |
| Ten equal width bins ECE | 0.083705 | The amount of description under the fixed binning caliber in this paper |
| Average Negative Log-Likelihood NLL | 0.385259 | Probability Loss Calculated Using Natural Logarithms |
| Double-Category Sum Brier Score | 0.169913 | The sum of the squared errors of the two categories is then averaged |
The easiest to misread the section of "more than 90%". The fixed threshold value of 0.9 was 242, of which 223 were correct, and Accuracy was 92.15%. At first glance, it seems to have exceeded 90%, why is it said that there is overconfidence?
Because the average Confidence of this set of samples is not 90%, but 99.45%. To judge whether the probability is matched, the probability and actual correctness of the actual report should be compared, instead of just taking the left end point of the sub-box as a reference. There are still 19 errors in these 242 items, and they cannot be ignored just because the "high confidence" label is uniformly labeled.
Low Confidence interval reminds us not to rush to tell stories. There is only 1 between 0.5 and 0.6, only 3 between 0.6 and 0.7, and they are all correct. This is not to prove that "the more the model is not grasped, the more correct it is", but the sample is too small to support this law. The number of people per box must be reported with Accuracy, and the empty box cannot be drawn as a zero error rate.
All samples have no trigger length truncation. The official batch cycle takes about 1.13 seconds; it depends on the imported experimental process about 1.38 seconds, including resource verification and model loading; the total time of the process with Python startup and import is about 3.29 seconds. The peak resident memory is about 831MiB. The download and first environment preparation are not included in these timings, and there is no measurement of GPU video memory. These are the measurements of this machine, this batch and the Input Length, and are not universal performance commitments.
6. Why can't you just report one ECE?
The ECE is divided into Confidence, and then the absolute value of the difference between the average Confidence and Accuracy is calculated according to the weight of the number of people per case:
is the number of boxes, is falling into the first sample sets of boxes, is the total number of samples. In this paper, ten equal-width boxes are pre-fixed, and the empty box contribution is zero, but its accuracy remains null. The full box table is saved with the result file, and the reader does not have to believe an isolated summary number.
When we subtract accuracy from mean confidence, overconfidence in one range can cancel out underconfidence in another. ECE takes the absolute gap within each bin before averaging. That explains how the same predictions can yield gaps of 7.16 and 8.37 percentage points, respectively. The next article holds the predictions fixed and varies only the binning scheme to see how sensitive the measurement is, without retraining the model in pursuit of a better-looking result.
We also keep NLL and Brier. The former focuses on the probability of allocating the real category, and the punishment is heavier for the situation that is very sure but the answer is wrong; the latter uses the square error to check the complete probability vector. This article is a version of the sum of the two categories, and cannot be directly mixed with the version that only calculates the probability error of the positive class. These amounts complement different perspectives, but none of them can obtain overall reliability guarantees with a single development experiment.
7. Read from official code to independent verification
The official classifier does not generate a "credibility head" out of thin air. Following the implementation of the fixed version, first look at the output of the encoder, and then look at the hidden state of the first position, the linear layer, ReLU, dropout and the final two categories of classification layers. Subsequently, the local script converts Logits into the probability of auditing. official implementation
This time, the actual loaded model source code is compared with the official pinned commit byte, and the model weight also checks the official file hash. Each saved result contains the original sample number, text hash, label, two logits, probability, prediction and token length. The original sentence is not redistributed with the source code, because the data card does not provide a clear license mark; the reader obtains and verifies the data identity from the official entrance.
The independent audit script does not import the main program's indicator functions, but uses the standard library to recalculate the probability and all summed values from Logits. The other input verification re-reads the data, and the label, text hash, and sample sequence do not overlap the three sets. We also intentionally damaged numbers, labels, predictions, correctness, Confidence, and losses, and confirm that all six errors will be rejected. Automatic recalculation is not a manual semantic review, and the two will not be mixed.
The process did encounter failures: the original network proxy of the machine was once unavailable, and the old dependencies on the cloud disk were also stuck in the import stage. The two import runs are terminated before the prediction, and the record is kept. In the end, the experiment was completed with the same version of the independent local environment, and there was no change of the model, the sample change, or the failure result to pretend to run a smooth operation.
8. Source code and reproduction entrance
The supporting source code has been made public, and the document is verified according to the pinned commit. The reader can Full source code directory enter, read first How to run the example, reconsider Formulas and official implementation maps. The 32 public files are the same as the local code package; the independent recalculation of the downloaded copy and the 8 real model test runs have also passed.
After obtaining the source code directory and installing the dependencies, download the fixed resources first, and then perform the real model test run and formal experiment:
python execute.py prepare --cache .cache112
python execute.py smoke --cache .cache112
python execute.py formal --cache .cache112
python execute.py audit
python verify_inputs.py --cache .cache112
Entry will generate one-by-item predictions, aggregate metrics, enter verification and execution logs. The hash is still checked when the cache is reused, and does not depend on the implied path on the author's machine. Readers who only want to check arithmetic can directly audit the results saved with the package; this is a reproduction of two different levels of re-running the neural network, and cannot be confused.
9. What to study next
The current evidence supports a limited judgment: under these 256 development samples and fixed protocols, the average maximum category probability of the model is significantly higher than that of empirical accuracy. It doesn't answer temperature scaling must be useful, abstention must improve reliability, or what will happen after changing the domain.
The follow-up tests address several competing explanations before drawing any verdict about reliability. Does the gap depend on how we define the bins? Can a simple transformation of the probabilities reduce the loss? If so, does it also change how errors rank? Will the fitted parameters still work after transfer?
Recent research on the inference configuration changes and Conformal Prediction also regards the configuration itself as an experimental variable that needs to be checked. This time, only the summary and version of the work have been checked, and the results have not been reproduced, so it is only used as a follow-up reading clue. 2026 research entrance
The most important product of the first article is not a "resolved trusted AI" conclusion, but a baseline that can be fairly revised, independently recalculated, and negative results are retained. With it, calibration and abstention become a problem that can be experimented, not just sounding advice.
References
Retrieval and access date: 2026-10-28. Method papers, model sources, code sources and experimental results are respectively indicated; the following paper numbers are not used as the measurement results of this measurement.
- Guo, C., Pleiss, G., Sun, Y., Weinberger, K. Q. 2017. On Calibration of Modern Neural Networks. ICML, this article checks the definition and formula, and does not claim to complete the entire reproduction.
- Geoff Pleiss. Author calibration code, pinned commit. Read ECE implementation; this paper uses clear and different binning endpoint conventions, and this sample has no exact boundary hit.
- Hugging Face. DistilBERT SST-2 fixed model version.
- Stanford NLP. SST-2 data card; Socher et al., 2013, Original Emotional Tree Library Paper.
- Hugging Face. Transformers classifier fixed source code.
- Doula, A., Mühlhäuser, M. 2026. Auditing Conformal Prediction for Language Models Under Inference-Time Model Shift. arXiv V1, 2026-10-20; this time only read the abstract, and did not repeat the experimental details according to the abstract.
Apps by the author
I also build small tools for everyday life. Here are two of my apps:
- Daily Expense Journal: Track your income and spending, plan a budget and set savings goals. View on the App Store.
- One Small Action: Pick a small action that fits the time and energy you have today, then record how it felt and what you noticed. View on the App Store.
Find more articles and projects on my website: zffjzz.top. I’d love to hear how the apps work for you and what I could improve.