
The confidence lab: from model confidence to testing when to abstain
For the same film review, the classifier gives "positive", but the prediction set can keep both "positive" and "negative". Leave one more option, it may save a wrong sentence, or it may only let the reader see an unusable answer. In the same batch of 360 short comments, we have changed the rules of constructing sets. The new rules are given seventy-four more labels, and the net more covers seven sentences. When these seven sentences are dismantled, the actual ten sentences are actually ten sentences from missing to hits, and the other three sentences change in the opposite direction. The coverage goals agreed upon by the two methods are exactly the same, but the sets handed over have different costs.
Reproduce Objectives and Evaluation protocol
In the previous article, keep the rules unchanged, check whether the overall coverage covers the group difference. This time, the sample, model and target remain the same, and only replace the scoring function that determines whether a label can be selected. We ask two questions that can be checked separately. First, which method actually lists fewer labels? Second, which sentences do the saved labels fall on, and does it contain the correct answers? If the average set size is only compared, the reader can't even determine the direction of the second question.
Conformal Prediction uses a set of calibration samples with official labels to set the threshold. The override here means that the true label belongs to the return set, and the denominator is all audited sentences, including sentences with empty output. Switchability means that the swap position of the calibration sample and the future sample in the joint distribution does not change the distribution. Under this condition and fixed scoring rules, the split conformal method gives the marginal guarantee for the probability of co-taking the Calibration Set and the new sample. It does not require two finite sample measurements to yield the same hits. When writing the "same coverage rate" into the experimental table, it is necessary to explain whether the same is the 90% of the pre-requested or the percentage you see afterwards. This time the former is fixed.
The data comes from the Public SST-2 verification division. The starting point of the series has been sorted according to the hashing of fixed prefixes and official sample numbers, the first 256 sentences are used for exploration, the next 256 sentences are used for calibration, and the remaining 360 sentences are used for protocol auditing. This batch of data has a history of use in both the old series and this series, especially the performance of the audit pool has been seen, and this article is a method exploration. The two rules share the calibration pool, which are divided on the true label. The audit label is only used to count the final hit, and never use it to adjust the two lines of numbers to the same. No new semantic judgments, no predictions to access IMDb.
Core thought: Decide who to compare with a label
LAC can be understood as a separate probability threshold for each candidate label. set the input as , one of the positive and negative labels is , the probability that the model is normalized is . The incompatibility of the candidates that do not conform to the score measurement is incompatible with the model output. The smaller the value, the easier it is to enter the group. LAC use . For example, the probability of the two candidates is 0.7 and 0.3 respectively. As long as the minimum probability required to be selected is lower than 0.3, both can enter. This example describes the rules, and the actual threshold is given by the calibration data.
APS is an adaptive prediction set. It first sorts the probability of the same sentence from large to small, and then accumulates in this order. This paper uses the deterministic configuration explicitly provided by TorchCP to obtain the cumulative probability mass of the candidate itself. If the candidate's ranking is , after sorting Item probability is , then there is
Among is the number of calibration sentences, is the allowable marginal leakage level, It is the rank from small to large, is the current method ranks the first on the calibration true class score. can be replaced with any predetermined scoring function, All candidates who have passed the threshold through this method are accepted. There are two directions in the ordering: the probability of the sentence is from large to small, and the true class score of the cross sentence is from small to large. The second sorting is also written in descending order, which will make the same set of code produce completely different overrides.
The original structure of the paper report allows processing boundaries with uniform random numbers, discussing marginal validity and adaptability to samples of different difficulties. We turn off the randomization in the implementation, so that every tag change can catch a specific comparison. This also limits the scope of application of this conclusion. Especially in the second category, the precise cumulative mass of the second place is always equal to one. There may also be uncounted categories after the second place in the multi-category, and the numerical structure of the two is substantially different.

Each branch in the figure first uses the calibration input to determine its own threshold, and then generates a collection for the same audit input. The two indicators on the right use this collection together, and the blocks in the figure only describe the calculation order.
Official Code Reading Routes and Minimum Experiment
Read from the model interface. The fixed English DistilBERT emotional classification checkpoint gives two unnormalized scores for each sentence, the first column is negative, and the second column is positive. The code first checks the label mapping, and then asserts that the number of rows output in each batch is equal to the actual number of sentences and the number of columns is equal to two. The official batch is sixteen, and the last eight sentences are in batches. Fill in the maximum length within the batch, the upper limit of truncation is 512 tokens; the attention mask retains the real tokens. This time, use the same CPU environment to perform eight test runs, and then re-tokenize the 360 sentences and move forward. The formal reasoning is about 1.36 seconds, and the peak of the process is about 952 megabytes. The original output is consistent with the historical official record item by item. The calibration side is directly reused the freeze fraction.
Follow the source map to TorchCP’s LAC class and compare the two call paths. With labels, it takes one score per sentence to set the threshold. Without labels, it returns the full matrix of candidate scores for constructing sets. APS uses the same interface but sorts and accumulates scores internally, then maps ranks back to the original label columns. Skip that last step and you might record the top-ranked label as class zero. To isolate other dependencies, the entry point recreates the package layout the official module expects and runs the original files from the pinned commit. Both methods use the library’s finite-sample rank function. The model’s forward pass uses single precision. Postprocessing converts the same pair of raw scores to double precision before normalization. Casting only the final result cannot reproduce the boundary effect below.
The error level of pre-registration is 0.1, and the calibration scale is 256, which corresponds to the 232 true class score. The output includes each identity, official tag, and two sets of candidates, and then calculates the number of empty sets, single tags, double tags and matched hits from these records. There are also entry conditions for the spare multi-classification experiment: only two schemes are given different collections on less than 18 sentences, and the registration of handwritten digital tasks is enabled. In fact, there are ninety-two different sentences. The two-classified experiments are completed according to the agreement, and there is no task to pursue a larger gap.
What we tested: where did the saved labels go?
| Fixed 90% target | actual hit | Empty/Single/Double Label Sentences | Total Labels | average size |
|---|---|---|---|---|
| LAC | 333/360(92.50%) | 0/351/9 | 369 | 1.025 |
| Deterministic APS | 340/360(94.44%) | 0/277/83 | 443 | 1.231 |
Both methods retain the first-ranked category. The three hundred and thirty sentences that are correctly classified in a single answer can naturally be covered. The remaining thirty sentences can better explain the problem: LAC saves three sentences alone, APS saves ten sentences alone, and the other seventeen sentences are missed on both sides. The "rescue" here only means that the correct label reappears in the alternative, and the first choice and correct number of the model have not changed. This distinction is practical for the reader: if the system still adopts the first item after the system displays two tags, then the collection coverage upgrade will not automatically become the final decision Accuracy boost.
When reading a sentence-by-sentence file, you can take the least official number in the two changing directions. The official label number 145 is negative, but the model gives a positive probability of about 86%. LAC only lists the front, APS reserves two items. Number 184 is also negative, and the model is also positive, but the preferred probability is about 66%. This time, the LAC accepts the next two items, and the APS only lists the front. Both examples can be independently restored from numbering and two-column scores without the reader's consent to any subjective explanation of the sentence's tone. They also point out why "large collections contain smaller sets" are easy to read incorrectly: the size of this table is averaging across sentences, and it is impossible to apply the inequalities between the averages line by line.
You can also see different meanings by restoring the two lines to the workload. If you return one more candidate, you need to check it once, the LAC has only nine sentences that need to be checked, and the APS has 83 sentences. If each original hit sentence is required to be kept, the increase of the average set size cannot meet this requirement, and the three pairs of pairing examples in opposite directions are enough to deny it. The article does not specify a relative price for missing the real class and looking at one more candidate, so it is impossible to deduce a general winner from this table. These two costs can be reported in the same field control of fixed goals; if you want to demonstrate who is more economical when the same measured coverage is to be demonstrated, new calibration design and independent evaluation are required.
Failure Troubleshooting: Thresholds are shown as one, why are there only 83 double labels
The easiest detail to miss is in the last decimal place. The calibration pool contains 25 incorrect first choices and 231 correct ones. Under deterministic APS, correctly classified examples rank the true class first, with cumulative mass less than one. On mistakes the true class ranks second, so its cumulative mass is theoretically exactly one. The 232nd score lands in the latter group. In exact arithmetic the resulting threshold would include both candidates. The reference implementation actually prints 0.9999999999999999, one representable step below one.
The independent script kept the same double-precision normalization but reconstructed each set using row-wise addition, a list sort, and an explicit comparison. In the audit pool, the two probabilities summed to the representable number immediately below one for 83 sentences, to exactly one for 257, and to the representable number immediately above one for 20. Only in the first group did the runner-up qualify under the inclusive threshold. Printing the threshold to two decimal places and copying it back into the code moves the cutoff. Even if the displayed coverage figures agree, check the actual operands, the comparison operator, and the precision of both calculations.
To test this explanation, we have added a diagnosis: the last cumulative probability mass is clearly set to one on both sides of the calibration and audit, and then the rank is taken from scratch. As a result, the three hundred and sixty sentences are all returned to double labels, all covered, with a total of 720 tags. This control is only used to locate the numerical effect, and the original registration result is retained as the main table. It shows that 83 double tags can be recalculated one by one on this machine, but calling these sentences a difficult sample specially identified by the algorithm is beyond the evidence. When changing the computing device or probability calculation sequence, the count of the last item should be compared first, and then the coverage difference should be explained.
Code and instructions for reproducing the results
The fixed version has been made public, available from Source code directory enter, press Reproducing the results run, or control Where the equations appear in the official code Check the implementation.
After installing the fixed dependencies, run in the source code directory:
python run.py
This minimum command reads saved real predictions, and outputs sentence-by-word collections and summary tables. Then run python audit.py, the diagnosis of each candidate's attribution, pairing count and the last item can be checked. If the model is replayed from the original text, the fixed weight and data are obtained according to the instructions, and then the test run and formal reasoning are executed in turn. Publishing packages only carry distributable derivatives; data identities, tokenization configurations, and dependency versions are all recorded. The initial development execution called it before the independent script was dropped and exited, and after completing the script, the four steps were re-executed and the exit record was kept twice.
Research questions and summary
The next time you need to check whether the determination rule can use the difficulty of the sample, you can freeze the numerical endpoint agreement in advance on the new multi-class development task, and record the scores of the last and last digits at the same time. The former still carries a sorting structure, and the latter is always one mathematical. In this way, even if the size change of the set is observed, it is possible to ask which item the change comes from. In the current group of short sentences, it can be verified that there are seventy-four candidates with less than seventy-four candidates to cover seven sentences, and the floating-point boundary changes which second candidates are selected. Interpretations of difficult texts, other models, or future long film reviews still require separate independent evidence.
References
Searched on 2026-11-16.
- Romano, Sesia and Candès, Classification with Valid and Adaptive Coverage, NeurIPS 2020
- TorchCP pinned commit, APS official implementation
- TorchCP pinned commit, LAC official implementation
- DistilBERT English Emotional Classification Model and Fixed Version
- Stanford NLP, SST-2 Fixed Validation Division
Apps by the author
I also build small tools for everyday life. Here are two of my apps:
- Daily Expense Journal: Track your income and spending, plan a budget and set savings goals. View on the App Store.
- One Small Action: Pick a small action that fits the time and energy you have today, then record how it felt and what you noticed. View on the App Store.
Find more articles and projects on my website: zffjzz.top. I’d love to hear how the apps work for you and what I could improve.