The confidence lab: from model confidence to testing when to abstain

Mechanical Limiters next to Individual Sample Boxes

Where did the seventeen more answers come from?

The same emotional classifier handles 360 film reviews, the first report reads "Reply 304 sentences, 14 wrong sentences", and the second one says "Reply 321 sentences and 16 wrong sentences". Both reports are in line with "in the sentences that are answered, the error ratio is not higher than 5%". If only one copy is reserved as an independent review, which one should be kept? Just looking at the denominator of the molecular denominator still lacks a record: when did you read the correct answers of these 360 sentences, and what decision did you make with the answer?

In the first report, use another 256 sentences to select the threshold value, and then count. The second report first spreads the label of 360 sentences, finds a dividing line that can answer more, and then counts it in place. There are 15 of the extra seventeen sentences to answer correctly. Those fifteen sentences are the true and correct answers. The problem is that if the reader moves to the next batch of film reviews with the second report, he will mistakenly think that this dividing line has undergone an independent test.

In the previous article, the number of acceptances was fixed, and the sampling fluctuation of a small number of errors to the profit of abstention was studied. This time, I will replace the fixed thing with a risk budget, and how many sentences to accept is determined by the data. The new experiment only changes the sample pool used to select the threshold. We have saved the counts of all candidates, separate calibration selection files, and a deliberately leaked comparison. The reader can judge along these three records, which step affects the answer.

From the Risk Definition of the Paper to an Executable Rule

Geifman and El-Yaniv in a selective classification paper published in 2017 separate what to do and whether to deliver this forecast. The paper studies how to use statistical upper bounds to control risks and report results on image classification tasks. This article borrows the definition of coverage and selective risk, and implements a simpler empirical rule on English short sentences. The high probability properties given by the paper need corresponding assumptions and calculations, and this experiment does not calculate such upper bounds.

record the first time ii The original confidence score of the sentence is sis_i, the forecast label and the public label are respectively hih_i and yiy_i. The two-class header gives two real numbers. After the softmax is normalized, the larger one in the two probability is taken as a fraction. to candidate thresholds tt, accept collection, coverage and conditional error rate writing

At={i:si≥t},Ct=∣At∣/n,Rt=∑i∈At1[hi≠yi]∣At∣.A_t=\{i:s_i\ge t\},\qquad C_t=|A_t|/n,\qquad R_t=\frac{\sum_{i\in A_t}\mathbf{1}[h_i\ne y_i]}{|A_t|}.

here nn It is the total number of sentences in the current pool, the vertical line represents the size of the set, and the judgment in square brackets is true. The denominator of the coverage rate is always the whole pool; the denominator of the risk varies with the acceptance set. If a sentence is not answered, the coverage rate can be zero, and the conditional error rate is not defined. Scores are only responsible for deciding whether to accept, and predicting negative and predicting positive samples take the same rule. The measurement here is a classification error under the given task and label.

Eight candidate values are written before this scan, which are 0.50, 0.90, 0.95, 0.97, 0.98, 0.99, 0.995, 0.999. For each non-empty acceptance set, multiply the number of wrong sentences by twenty, and compare with the number of accepted numbers, the former is less than or equal to the latter to be eligible to be selected. Those who take the highest number of acceptances from qualified persons, and take a small threshold. If all fails, all abstention all. In this way, even rounding, draws and empty sets do not need to see the audit results before making a decision. Samples that are exactly equal to the threshold also belong to the accept set.

Experimental Settings and Label Reading Order

The data is the SST-2 verification division exposed by the Stanford team. The items are initially calculated and sorted according to the sample number plus a fixed prefix, and the exploration, calibration and protocol audit pools are drawn in turn. In this article, 256 predictions of the second pool are reused, and the 360 articles of the third pool are used for evaluation. The sample numbers of the two pools are mutually exclusive. SST-2 has been used in previous research, and the overall performance of the audit pool has also been made public. The "independent" here limits the data flow selected by this threshold; re-executing the script cannot erase the researcher's previous understanding of these data. There is no access to IMDb, and no new manual tags have been added from the text meaning.

The upper row is locked first and then evaluated, and the lower row is selected and reported with the same audit data.

The upper slider is locked before the audit sample arrives. In the actual operation, the selection process is started separately. The input is the calibration prediction and original division list, and the output file saves the performance of the selected value and eight candidates. The audit pool prediction is then replayed on the frozen model. The lower row reuses the same set of comparison functions, and also uses the same risk budget. The only intervention is to replace the input of the selection function into the audit pool itself. There is a complete label circuit in the lower row, so it is clearly marked as a leak control in the result file.

This control also fixes the computer meeting. There are eight candidates on both sides, and there is no search for different temperatures, text templates, or abstent scores. In the selection process, the model parameters are neither updated nor the sentence content is selected. The number of calibration samples and the number of audit samples is divided into historical division, and the denominator for selection is different on both sides. Therefore, the observed difference describes these two specific processes, and it is impossible to dismantle the sample size, sample composition and contributor to the same pool selection from one pairing.

Minimum Experiment and Guide to the official implementation

Start with the official DistilBERT sequence classification class. It takes the encoder representation at the first position and applies a linear transformation, an activation, dropout, and the classification layer to produce two unnormalized scores per example. Inference ran in evaluation mode, and we verified that label 0 means negative and label 1 positive. The input is an integer tensor of shape batch size by padded sequence length. The output always has two columns. Now follow the model entry point used here to see where it computes the maximum probability and records each sentence’s ID, original label, and whether the prediction was correct.

The environment is fixed to Python 3.12.14, PyTorch 2.6.0, Transformers 4.49.0, and run on the CPU with single precision and four threads. Models, tokenizers, data files and history are verified one by one, using the original cache. First take eight sentences for smoking inspection, and then replay 360 sentences according to sixteen sentences in each batch, with a maximum length of 512 tokens. The logits of the official batch are the same as the historical snapshots, the reasoning loop takes about 130 seconds, and the process peak is resident in memory of about 945 megabytes. This time does not include the time to import, first download and blog. Calibration predicts the real reasoning that has been archived, and this time there is no duplicate model calculation for it.

When only the table is recalculated, the reader does not need to download weights. First run the following minimum command in the matching source code directory, view the selected file and result file, and then run the independent audit. If you want to verify the model environment, prepare fixed resources according to the reproduction instructions, and replay the smoke and official batches. The difference between the small batch and the original record is about 2.4 by the negative seventh power, and the label is the same; the independent official batch checks the influence of this small difference to make the influence of this small difference can be positioned.

python run.py
python audit.py

In which step are the eight candidates to decide the winner?

Each grid in the following table is "Error/Acceptance". The calibration column determines the upper row threshold separately, and the audit column is used to review it, and it also shows how much information can be seen in the deliberately leaked row. Check the qualifications from top to bottom for each column, and then find the largest denominator in the qualified grid, and you can reconstruct two results with a pen and paper.

threshold Calibration pool (256 sentences in total) Audit pool (360 sentences in total)
0.50 25/256 30/360
0.90 22/247 23/343
0.95 16/239 21/334
0.97 15/236 20/329
0.98 12/231 16/321
0.99 10/218 14/304
0.995 5/209 10/282
0.999 1/158 6/225

In this actual verification, the watershed of the calibration column is in two adjacent rows. The threshold value is 0.98, and 231 sentences are left, and twelve mistakes require at least 240 sentences to be the denominator to meet the standard. After increasing to 0.99, ten mistakes correspond to 218 answers, about 4.59%. Continue to raise the threshold and still pass, but will answer less. After locking this line, 304 sentences are accepted in the audit pool, with 84% of the coverage rate, and the risk is 4.61%.

The leak control sees another kind of boundary in the audit column. Threshold 0.98 accepts 321 sentences, there are sixteen errors, just more than the 320 answers required for eligibility. Further down to 0.97 will deliver eight more sentences, four of which are wrong, so it exceeds the budget. The grid in the leak rules gives a coverage rate of 89.17 percent, and the risk is 4.98 percent. If only the risk is compared, it is also slightly higher than the locking rule. What is optimistic here is "how much can be answered within the budget", and a report with only error rate and no coverage will hide this choice effect.

We can account for the 4.72-percentage-point coverage gap sentence by sentence. The accepted sets are nested, and the larger set contains seventeen extra sentences with scores between the thresholds. The model predicted the same class for each sentence under either rule. What changed was which answers counted in the reported result. Notice how close 16 out of 321 is to the cutoff: with the same denominator, one more error would have disqualified that cell. This arithmetic check of the selection rule involved no relabeling or changes to the protocol.

Troubleshooting and limits of the evidence

If the reproduction result is only one row, check the comparison first or after the rounding. If a line that just crossed the line in the audit table is displayed as an integer percentage, the reader can't see how far it is offline. Formatting all probabilities first into two decimals also moves sentences near the boundary. The independent script recalculates a high probability from the absolute difference of the original two-column scores, uses the original label to redo the category judgment, and checks the molecules, denominator, coverage and selection results of the two pools and sixteen rows one by one. This process verifies deterministic arithmetic and does not involve a real person's ruling on film review emotions.

Another difficult error is that although the data is divided into files, the selection function accepts a spliced general table. In your own reproduction, you can put the selection stage separately in the directory available only for calibration prediction, keep the source code, division list and related lock files, and then execute a separate selection command. The same threshold should still be obtained under normal circumstances. The full assessment will then require audit forecasts. Even if the two tables have never been physically stitched together, it is important to pay attention to whether people have encrypted the candidate list again after reading the audit results. That would give researchers an opportunity to adjust the ginseng, leaving the final threshold itself that cannot reveal this history.

Research questions and summary

This time, the author infers that when you see the "standard" when you see the "standard", you can judge the strength of the evidence better than reporting one decimal to one more decimal. The locking rules meet the standard on these historical short sentences, which is an observation that can be reviewed. The leak control happened to find more thresholds to cover, and the difference gives a visible consequence of a protocol error. We did not estimate the mean of this difference during resampling, nor were we guaranteed to maintain the same risk in the future text. If the two pools happen to choose the same line, the two reports will be exactly the same in numbers, and whether the selection process is independent or not depends on the running record to answer.

A verifiable follow-up question is, after the freezing pool is divided and accepted rules, can positive temperature scaling change the overall risk curve of the entire ranking? Numerical thresholds will move due to fractional scales, but the number of fixed acceptances only depends on sorting. Only by distinguishing these two experimental objects can we judge whether the coverage change seen next time will come from the model, or whether it will move from the scale on the same ruler.

Code and instructions for reproducing the results

The fixed version has been made public, available from Source code directory enter, press Reproducing the results run, or control Where the equations appear in the official code Check the implementation.

The minimum entry output Selected.json and results.json, respectively, save the two sets of audit results for calibration selection, lock and leak respectively. Sentence prediction, protocol, resource version, actual running log and independent recalculation script are provided, which can be checked directly from the denominator of two adjacent thresholds.

References

Retrieve vs version core for 2026-11-10. The following links correspond to the paper version, actual model and data, framework implementation and meeting minutes respectively.


Apps by the author

I also build small tools for everyday life. Here are two of my apps:

  • Daily Expense Journal: Track your income and spending, plan a budget and set savings goals. View on the App Store.
  • One Small Action: Pick a small action that fits the time and energy you have today, then record how it felt and what you noticed. View on the App Store.

Find more articles and projects on my website: zffjzz.top. I’d love to hear how the apps work for you and what I could improve.