The confidence lab: from model confidence to testing when to abstain

If only three-quarters of the film reviews are allowed to be answered, which ones should be left first? In the previous article, the maximum probability of using the classifier was queuing, and 270 were reserved, of which eight were wrong. A natural improvement is a different scoring rule. For example, how hesitant is to use entropy to measure the model between several answers, or directly see how much the first place leads the second place. Today's recurrence starts with a cheaper question: In this model, is it possible to replace even one accepted film review?
What we tested. In the fixed 360 SST-2 predictions, the list given by the three rules is exactly the same from the first to the last. This result can be rolled out from the mathematical structure of two classifications, but the program is still worth running. Prove the exact number of discussions, and the experiment should face finite precision, fraction direction, line sequence and label mapping. We have saved the three scores of each sample, we can find two real records next to the threshold, and see what seems to be lost if a seemingly harmless round is lost.
The order of freedom of sorting from the two outputs is clear
The categorical header is an input that returns two unnormalized real numbers, called logits. respectively and , corresponding to negative and positive. When two values are added to the same constant, when doing softmax, the denominator of the molecule will be multiplied by the same factor at the same time, and the probability remains unchanged. Swapping two values will flip the prediction category without changing the model's grasp of the winning category. Get rid of these two factors that have nothing to do with grasping, and the rest is how far the two are separated.
here It is the "score difference" in this article, and the absolute difference of the original output is taken. are the two probability of normalization. is the larger one. When the gap is zero, the two categories are equally divided; the larger the gap, the closer the winning probability is. Therefore, the queuing according to the absolute difference is the same as the maximum probability from large to small, and the relationship between any two records is the same. If you are used to defining the "score difference" as the probability difference between the top two, you can also directly check the calculation. In the second classification, it is equal to . This program only implements the former definition.
Entropy brings the entire row probability into the calculation. Just in the two types of problems, as long as you know the probability of winning, the probability of another type is determined. Using natural logs, binary prediction entropy and the derivative of the winning probability are
Describes the concentration of a predicted distribution of a row. if only , the derivative is less than zero; one-half of the end point corresponds to the maximum entropy, and the right-going function is strictly reduced. Therefore, to answer the samples that seem to be the most definite, the entropy should be arranged in ascending order. The units and values of the three scores are different, but the same one time can be converted into each other. This proof does not use emotional tags at all. Even if the model confidently judges a certain film review very confidently, this sentence will deceive these three rules at the same time.
Rewrite the proof as a possible failure experiment

What the paper reports. Geifman and El-Yaniv write selective classifications as a combination of predictors and selection functions, and study how to preserve more answers under risk constraints. We continue to use the coverage and selective risk definitions. The additional fixed number of this article is to track the identity of the sample left under this number. The risk control in the paper also requires corresponding estimation procedures and assumptions, and only actual errors on the limited sample are reported here.
The only variable for the experiment is the sort key. Lock DistilBERT's SST-2 classification checkpoint, tokenizer, original sentence identity, inference configuration and output. I have already observed this batch of data, know that there are a total of 30 mistakes, and have also seen the curve of the largest probability. This article writes down the comparison rules before calculating the three-way sorting, so it is a method verification on a known background. SST-2 has also been used in earlier series, and this article will not estimate the guarantee of never touching data from this batch of records. IMDb development, calibration and final confirmation pools are not used.
Each channel is sorted on the original precision score, and only the exact equality is processed in ascending order of the public sample number. The original emotion tag is counted incorrectly after the sorting is over. Let the total number of samples be , before taking article, The error indication for a ranking is , the prediction is wrong, and the correct is taken to zero. The coverage rate and the acceptance risk are respectively
The denominator varies with the number of acceptances. When all samples are rejected, the coverage is zero, and the number of errors is also zero, but the conditional error rate cannot be calculated; the program writes null values in this grid. The main test is to compare the complete identity sequence and each prefix from zero to all. Only one error is missed only than the risk value: exchange positions of the two samples that are correct, and the risk curve can be fixed. Only a percentile will miss the dislocation outside the threshold value. Therefore, it is agreed in advance to check all positions and keep the first disagreement position, which is convenient for direct positioning when it fails.
Official Code Reading Routes and Minimum Experiment
First look at the task and classification label of the model card, and then read the forwarding of DissilbertForSequenceClassification in the pinned version of Transformers. It takes the representation of the first position from the output of the encoder, and then returns the original score of each class through the pre-classification layer, activation, discard layer and final classification layer. Recurrent is set to the evaluation mode, and the randomness of the discarding layer is turned off. There are two steps from here to the abstention list, which are normalized along the category and sorted along the sample dimension. The former is compared between the two outputs of the same sentence, and the latter decides who will answer the first two sentences. Confusing these two dimensions, the obtained array may still be very neat in shape.
From the fixed resource cache, the summary of the weights, vocabulary and verification files is taken, and the historical audit pool is taken according to the registered identity list. Extract the first eight pieces for a real test run, check the original label, text abstract, predicted category and output shape, and then replay all 360 articles in a maximum of 16 items per batch. Use padding within the batch, and keep the maximum length of 512; forwards use four CPU threads and single precision, and the sorting calculation is carried out on the Python floating point number. The eight pieces belong to the official collection and cannot be regarded as eight additional observations. No training, temperature fit, or threshold search.
The official test is about 1.25 seconds, and the resident memory of the peak process is 963,903,488 bytes. The forward timing here includes batch-by-batch tokenization and collection records, and the full process also has import and model loading overhead. The maximum absolute difference between eight test runs and archived logits is about , the prediction category is the same; the maximum difference is zero when replaying all samples according to the official batch. In order to make different articles share the same comparison object, the sorting entry reads the locked archive, and the model replays the output separately to save the output, to avoid the new run and quietly rewrite the comparison benchmark.
Two movie reviews next to the line of acceptance
Each of the three measures produced 360 distinct scores, so the tie-breaker never came into play. At rank 270, ID 382 had a maximum probability of about 0.996946, entropy of about 0.020737, and an absolute score gap of about 5.788205. Next came ID 505, with 0.996938, 0.020782, and 5.785638, respectively. ID 382 had the higher probability, lower entropy, and larger gap: it made the cut all three ways. At three decimal places both probabilities read 0.997. Sorting the values copied from the report would give the tie-breaker a say in which example qualifies.
| Accept the number of items | coverage | Number of set errors accepted (three paths are the same) | Accept set error rate |
|---|---|---|---|
| 180 | 50% | 3 | 1.67% |
| 270 | 75% | 8 | 2.96% |
| 324 | 90% | 17 | 5.25% |
| 360 | 100% | 30 | 8.33% |
According to the pre-specified three-quarter coverage rate, the eight errors left are exactly the same, and the other twenty-two errors enter the Abstension set. The ninety articles of Abstention also contain sixty-eight examples that can be answered in the first place. This cost is paid for each representation. When increasing the coverage rate, the number of errors can only be flat or increased, but the error rate may fall: continue to accept the next one from the 270th place, the molecule is still eight, and the denominator is one more. When drawing, it is stated whether the vertical axis is the number of errors or the error rate, otherwise a small downward step may be misread as a data exception.

The left panel staggers the markers so you can see that all three measures trace the same staircase. The right panel asks a stricter question: compared with the ranking by maximum probability, how many positions contain a different example when ranked by entropy or by absolute score gap? The answer is zero in both cases. Reporting the area under each curve adds no independent evidence. Using the spread among those curves as a measure of algorithmic stability would understate uncertainty. What we can infer about errors on future samples is still limited by the number of examples and errors in this one audit.
Failure investigation and independent recalculation
If the entropy is significantly worse for reproduction, the first and last rankings are given priority to confirm which end the record of the highest entropy falls. If there is an absolute difference, the emotion will be concentrated to the back, and check whether the signed positive class score is used to reduce the burden class score. In that calculation, a firm negative prediction and a hesitant negative prediction will get a negative value, and the direction will follow the emotion. If there are only a small number of rankings, first check the complete precision and sorting second keyword, and then compare the text abstract, batch and tokenization settings. It is difficult to distinguish the same division rearrangement from the wrong input according to the shape of the curve.
The independent script re-reads two columns of the original output from the lock file, uses 60-bit decimal operations to calculate the smaller probability, and then obtains a greater probability and entropy with a complementary relationship. It rebuilds the error indication line by line, re-ranks the road line by way, and counts the wrong number for each acceptance set. The main program uses ordinary floating-point index normalization and cumulative counting, and the two calculation paths can expose different program errors. This time, all 1080 scores, three identity sequences, and 1083 prefixes were all checked and approved. Scripts, input summary, commands, exit codes and results are archived together. These are deterministic arithmetic evidence, and the labels are directly taken from the public division without new semantic annotations.
Research questions and summary
Author’s interpretation. If you want to surpass the maximum probability here, you need a signal that there is still the opportunity to provide new information when the winning probability is the same. A verifiable boundary comes from multiple classifications: two three types of predictions are allocated 60% probability for the first category, and the remaining 40% can be almost all given to the second category, or the remaining two categories can be divided equally. At this time, the maximum probability is the same, but the entropy changes with the distribution of the remaining probability. This is a structure to prove what conditions depend on, and it is not the multi-class benefit measured this time. Another situation that is easy to change the list is that low-precision quantization can press many different values into the same points, and then the scale and adjudication rules should be sequentially.
The two questions were disassembled in this experiment. Whoever answers first, on this fixed two-category header, it can be completely determined by an absolute difference. The former problem has algebraic proof and program verification, and the latter problem only observes eight errors under three-quarter coverage. What's next to ask is that so few events allow more accurate judgments on the abstention benefit. Repeating naming the same sort does not provide new observations.
Code and instructions for reproducing the results
The fixed version has been made public, available from Source code directory enter, press Reproducing the results run, or control Where the equations appear in the official code Check the implementation.
Run the following minimum command from the source directory, use the archived real model output to generate rankings and recalculate independently.
python run.py && python audit.py
The result file Ranking.json contains line-by-line scores, triple complete identity lists, and number of errors prefixed. Readers who wish to regain their scores from the sentence can prepare fixed resources according to README, and then perform eight test runs and full audit pool replay. The source map gives the fixed line number, license and local entrance of the official classification header, and the running log retains the input and output shapes of each batch. In this way, the error can be checked separately in the classification head, or after reading the score.
References
The search date is 2026-11-08.
- Geifman and El-Yaniv, Selective Classification for Deep Neural Networks, NeurIPS 2017
- Hugging Face, Distinbert SST-2 Fixed Checkpoint and Glossary
- Stanford NLP, SST-2 Fixed Validation Division
- Transformers 4.49.0, DistilBERT official classification head fixed source code
Apps by the author
I also build small tools for everyday life. Here are two of my apps:
- Daily Expense Journal: Track your income and spending, plan a budget and set savings goals. View on the App Store.
- One Small Action: Pick a small action that fits the time and energy you have today, then record how it felt and what you noticed. View on the App Store.
Find more articles and projects on my website: zffjzz.top. I’d love to hear how the apps work for you and what I could improve.