
The confidence lab: from model confidence to testing when to abstain
The same is 90%, who is the denominator?
Give a sentiment classifier a batch of film reviews and let it return both “positive” and “negative” when it’s unsure. Across 360 sentences, the returned set contained the true label 333 times. Looking only at that figure, you might write “90% coverage” for negative reviews too. Take the negative sentences on their own, though, and the count is 158 out of 176, one hit short of 90%. Both observations come from exactly the same predictions.
Why is it worth doing another experiment? In the previous article, the number of calibration samples was changed, and the finite sample quantum can suddenly expand the return set. This article restores the number of calibration samples to the complete 256, all sentences share a threshold, and even return which tag is returned, only the denominator at the time of the report is changed. Readers can download sentence-sentence results, starting from the cross grid of the category, length and the two, and count back the same general table. The "security" here only examines whether the return collection contains official tags, and the safety in other senses requires respective indicators.
How a threshold becomes a grouped table
The classifier gives two columns of original scores for each sentence, called logits. First do softmax on them, that is, normalize the exponential weights, and get the predicted probability of each of the two tags. The probability of the candidate label is subtracted from one, and the inconsistency is obtained. The lower the probability that the model gives a candidate, the higher the inconsistency that needs to be tolerated to include the answer. This simple scoring method is often referred to as LAC. When calibration, only take the column of each real label, sorted from small to large.
Here is a sentence, Iterate through two candidate labels, is the probability that the model gives the label. is the number of calibration sentences, is the preset probability of leakage cover, Indicates the number from the beginning Small calibration score. take this time 、, so take the 232nd value; the code retains the complete precision, and the juxtaposition is judged less than or equal to the judgment. is the returned collection. During the audit, the membership of the two candidates will be calculated first, and the two decisions will be written before the hits are checked with the official label.

If the calibration examples and the new labeled example are exchangeable, meaning that permuting them leaves their joint distribution unchanged, and the scoring rule is independent of this calibration set, the rank above gives a lower bound on marginal coverage. “Marginal” accounts for randomness in both the calibration set and the new example. For a particular negative sentence the formula returns a definite set. It doesn’t tell us that this sentence’s true label has a 90% chance of being in it. Angelopoulos and Bates treat marginal and group coverage separately in their tutorial. Validity within a group requires additional construction or assumptions.
For a data that has been run, the relationship is more intuitive. As long as the groups are mutually exclusive and exhausted, the total number of hits is the sum of the number of hits in each group, and the same is true for the total number of sentences. The overall ratio is equivalent to the average of the number of sentences in each group. If you want each category to reach a certain lower bound, only constraining this average leaves a lot of freedom. Realistic audits need to answer two things at the same time: which rows are the average consisting of and how much evidence each row has accumulated.
A minimal experiment and evaluation protocol
Use the public English SST-2 to verify the division, and the DistilBERT classification checkpoint for fine-tuning on SST-2. At the beginning of the series, all samples have been registered in the hash order of stable identity, and the first 256 articles are reserved for exploration, and then 256 articles are used for calibration, and the remaining 360 articles are used for agreement auditing. The SST-2 has been used in previous projects, and these audit results have also appeared in this series. Here we examine a reporting method that is limited to historical development data. IMDb's exploration, calibration and final confirmation pools are not involved.
This grouping agreement is first dropped. Category directly take the official positive and negative labels. The length is the number of valid positions in the tokenizer output, including the special symbols at the beginning and the end, and exclude the positions filled in the batch. The length boundary follows the sixteen and thirty-two when the previous study of high confidence errors is used, and the two ends are equal to the shorter one. The valid length of all sentences in this audit is lower than the upper truncation limit, so this batch of records can be compared with the definition of the original tokenization number earlier. When changing tokenizers, the same sentence may span file, and the recurrence should be locked together with the tokenizer version.
First use eight sentences to test runs on the central processor, and then execute the forward direction on the complete audit pool, batch size 16, four threads, single precision, and turn off the training mode. Formal forward tokenization and construction records take about 1.26 seconds, and the process peak memory is 944160768 bytes. All the original scores after the rerun is equal to the value-by-value of the historical official record. The calibrated pool multiplexes the fixed real model score, and the threshold is 0.7186523761 for this reordering. All groups use this threshold and do not pick points, fit temperatures, or reselect models on the audit results.
What we tested
| Denominator | Contains true tags/sentences | coverage | return two tags |
|---|---|---|---|
| Entire | 333/360 | 92.50% | 9 |
| Really negative | 158/176 | 89.77% | 5 |
| real face | 175/184 | 95.11% | 4 |
| short, up to 16 | 83/86 | 96.51% | 1 |
| , 17 to 32 | 179/199 | 89.95% | 5 |
| long, more than 32 | 71/75 | 94.67% | 3 |
Reading the medium length line depends on the integer. If only one decimal is reserved, about 90% of the numbers can easily mask which side of it actually falls online. Here, dividing one hundred and ninety-nine by 199 is slightly lower than 90%, which is less than the step size of a sample. Both the real label dimension and the length dimension can be spelled back to the overall population. The same sentence belongs to a label group and a length group at the same time; adding the last five rows in the table will count each sentence twice.
After crossing the two groups, the molecules and denominators of the six grids are as follows. The column direction is from short to long, and the direction of the row distinguishes true labels. The complete list of each grid can check which grids together constitute a row of the previous table, and also keep the groups with very few samples but a high proportion in the reader's field of view.
| real tag | Short sentence | medium length | long sentence |
|---|---|---|---|
| negative | 42/43 | 81/94 | 35/39 |
| Positive | 41/43 | 98/105 | 36/36 |
Of the twenty medium-length sentences whose true labels were missed, thirteen were negative and seven were positive. Coverage among medium-length negative sentences was 86.17%, and this group accounted for thirteen of the twenty-seven misses overall. The first figure tells us how often the true label was included within the group. The second tells us where an auditor would find much of the work. It could help us decide which examples to read first. Nobody annotated these sentences for sarcasm, negation, or label ambiguity in this audit, so we cannot attribute the thirteen misses to any of those phenomena.
The thirty-six all hits of the front group of the long sentence are also worth stopping for a while. If you change a hit in this grid, the ratio will move by about 2.78 percentage points. In the medium length group, change only about half a percentage point at a time. There is a sampling problem between the zero observation leakage cover and the future leakage cover probability of zero. The six intersecting grids share the same calibration threshold, and the highest and lowest grids observed are also parallel comparisons;
Here’s a calculation you can do by hand to distinguish coverage within a group from the group’s share of the sample. Cover the medium-length group in the length table. The short and long groups together have 161 sentences, of which 154 are covered. Put the medium group back and you have 360 sentences again. Every sentence keeps exactly the same label set, yet the pooled rate changes with the group weights. On a new batch of reviews, the overall rate could shift even if coverage within every length group stayed exactly the same. Before concluding that the model has deteriorated, report the new numerator and denominator for each group so changes in the mix can be separated from changes in performance.
Which line starts with the official code
Starting with the classification scoring file of TorchCP, see how to take the value along the category when specifying the real label, and then see how to keep two columns when all labels are required to be scored. Then read the upward rounding and order statistics in the common function, and finally go to the segmentation predictor to track the process of calibrated threshold into the collection generation function. These three points correspond to "Which column to take", "How many thresholds to determine the threshold value" and "When to add candidates to the collection". The local source map lists the fixed commit and the specific line number, and can align the input dimension and return value item by item.
The classification header uses the official Transformers class, and the output shape is multiplied by two in batches. The teaching script implements post-processing independently, retaining the explicit sorting and label list, which is convenient for reviewing intermediate states. Independently checking the script to take another operation path: under the two classification, the opponent's label score minus the candidate label score, and the difference between the obtained difference and the degree of non-compliance is strictly in the same direction. Count how many calibration differences there are strictly smaller than the candidate difference, and you can judge whether it falls within the threshold. The script starts from the archived Logits, checks 720 memberships, and then independently summarizes all the twelve lines of counts, and the actual execution passes. This is arithmetic recalculation.
Failure investigation and verifiable interpretation
The easiest to change the conclusions silently is grouping with predicted labels. It can be obtained when the sentence is received, and the answer is "how many sentences the model judged to be negative are covered". The negative lines of this article are divided according to the real labels, which can only be calculated during offline evaluation. If it is directly written as an online shunting rule, the conditions for already knowing the answer are implied at the time of deployment. The length group can be obtained with the same tokenizer before reasoning, but the relationship between it and the sentence difficulty needs to be independently verified.
Then check the size of the returned collection. There is no empty set this time, three hundred and fifty-one sentences return one tag, and nine sentences return two. Each double tag collection will contain true tags on binary tasks. Just looking at the hit count will make the reader invisible to this part of the information cost. In turn, the highest score label will make a sentence with "double labels contain true classes but preferred errors" to fail. The reproducer can first check whether the sum of the double labels of each group is equal to nine, and then check the denominator, and finally compare the percentage. This can help locate whether the collection is calculated incorrectly, or the records are lost in the summary stage.
Research questions and summary
A verifiable follow-up hypothesis is that in the new sample of the same source, the medium-length negative sentence will still be easier to leak cover than the same length positive sentence. To verify it needs to fix the vocabulary, boundary, main comparison and sample budget before looking at the new results, and report the respective omissions and uncertainties of the two groups. Another explanation is that the current two groups of difficult cases happen to be concentrated in this batch of sentences. This single segment cannot distinguish two explanations. If each group is calibrated, it will also give the small group to fewer calibration samples, and encounter the problem of rounding up the limited sample rounding discussed in the previous article. This article retains the original threshold and leaves a common starting point for future comparisons.
Code and instructions for reproducing the results
The fixed version has been made public, available from Source code directory enter, press Reproducing the results run, or control Where the equations appear in the official code Check the implementation.
Enter the source code directory to run python run.py, you can reconstruct each sentence-by-word collection, group count and input summary from the real score in the package, and then run python audit.py independent check. When you need to redo forward, install the lock dependency according to the instructions and prepare a fixed version resource, and run eight test runs and complete audits in sequence. When checking time-consuming, please distinguish the model forward loop and the entire command containing import, load and exit. The two numbers are given in the running record.
References
The search date is 2026-11-15.
- Angelopoulos, Bates, "A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification", 2021 First public in the year. The structure and coverage of the division conformation are shown in Section 1, and the group balance and category conditions are covered in Section 4. Full text from the authors
- SUSTech-ML, TorchCP, pinned version of the limited sample take rank and cross-border processing. Quantitative realization
- Hugging Face, English SST-2 classification header and label configuration. Pinned model revision
- Stanford NLP, SST-2, Validation division and data identity. Pinned dataset revision
Apps by the author
I also build small tools for everyday life. Here are two of my apps:
- Daily Expense Journal: Track your income and spending, plan a budget and set savings goals. View on the App Store.
- One Small Action: Pick a small action that fits the time and energy you have today, then record how it felt and what you noticed. View on the App Store.
Find more articles and projects on my website: zffjzz.top. I’d love to hear how the apps work for you and what I could improve.