
Reproduce the goal: first ask what range the probability is normalized
It also shows "80% grasp", and the two models may be answering completely different questions. The emotional classifier outputs positive and negative probability; the language model first allocates the probability of sequential writes in the vocabulary, and then we only keep two labels and re-normalize them. The values on the interface are between zero and one, which does not mean that they have become the same reliability evidence. This article fixed the same group of public short comments, and established a contrast that can be tracked one by one, focusing on the meaning of checking the score, rather than the ranking of the competition model.
Compared with the previous high-confidence error grouping, the new variables in this article are the prediction interface: while using the saved binary header results, the limited candidate scoring of the real generative model is added. The sample identities, original labels, and comparison denominators are consistent, do not fit the temperature, do not pick a prompt template, and do not adjust the abstention threshold. The new product includes the tokens for each candidate, the logarithmic probability, the normalization probability, the total candidate mass, and the independent recalculation record. Beginners can understand the difference along the "denominator"; engineering readers can execute the entrance; study readers should pay attention to what conditions the conclusion is true.
Core thought: two numbers can be evaluated on the same table, not automatically synonymous
The categorical header gives each tag a logit, i.e. the real score that has not been normalized. After performing softmax on two scores, you get the class probability:
here is a movie review, is the label number, It is the output of the classification head; a single output shape is two numbers. Prediction takes the label corresponding to the maximum probability, and the Confidence takes the maximum value. This probability may still be inaccurate, and it cannot be regarded as a real accuracy just because the classification head has been specially trained. The calibration paper of Guo, Pleiss, Sun and Weinberger discusses the relationship between predicting Confidence and the actual correctness; it does not guarantee that a small model in this article must be well calibrated.
The output of the language model covers the entire vocabulary. Given a freeze prompt , candidate string of:
Among is the number of token candidates, Indicates fixed model parameters. First, the logarithmic probability of each token is calculated in the order of cause and effect, and then summed; there is no average length, and there is no additional terminator. This time, the two candidates each occupy one token, so the specific confounding factor of the difference in candidate length is ruled out, but it does not mean that other tag words are also the same. Only represents the relative weight after redistribution between these two strings.
If both candidates are different single tokens, further calculations can be made:
is the total probability mass of the next token that falls in these two candidates. it with Answering different questions: Even if one side between the two candidates has an absolute advantage, they may still retain a large number of vocabulary probability. Multiple token candidates, prefix overlaps, or length normalization cannot be used without checking this simple explanation; this single token check is part of the conclusion.

Data, Budget and Freeze Agreement
The data comes from the SST-2 validation pinned version of Stanford NLP, and sorts the splicing results of the string "cycle7:" and the sample number according to SHA256, and takes the first 256 items. In this round, the other 256 calibration pools and 360 agreement audit pools have not participated in the forecast, and IMDb has not been used. The SST-2 has historical contact in the old wheel, so this is an exploration experiment of reuse public data, and cannot be packaged as an unseen test. The new model may also be exposed to public text in pre-training, and there is no zero pollution proof in this article.
The classifier is a fixed disilbert SST-2 checkpoint, and the generative model is fixed SmolLM2-135M-Instruct; precise revision, file hash and dependency are listed in configuration and lock files. The former uses the 256-line records of the first realistic reasoning, and the latter does not write the time of the reuse result as a new measurement. The training objectives, scale, tokenizer and input formats of the two are different, so this experiment is not just the causal ablation of the architecture.
The generation side uses a fixed English question, which requires the judgment of emotions between positive and negative, and the Answer is prefixed as Answer:, and the candidate contains a leading space. There is no chat template, a few-shot example, an additional start character, or an open generation; this is a frozen original renewal interface, which does not represent the optimal effect of the model card recommended chat. Each first validate the complete string tokenization retains the prompt prefix, and if it exceeds 512 tokens, it will report an error, and it will not be truncated. Run eight tests first, and then execute all the samples on the predetermined budget; the template has not been changed due to the test Accuracy.
A guide to the official implementation
First look at the two official model cards to identify tasks and licenses; then look at the DistilBERT category header and LlamaForCausalLM in the Transformers pinned commit. The former maps hidden representations into two label scores; the latter maps to the entire vocabulary via lm_head. This time SmolLM2 is implemented using this language model. The actual installed source code and pinned commit files are checked byte for byte, and the same version is not replaced by the same version of the class name.
When reading and running a script, the easiest thing to go wrong is not softmax, but the predicted position: the output on the position predicts the next token, and the output of the location of the candidate token itself cannot be regarded as its conditional probability. The code first removes the token at the end of the complete sequence, and then intercepts the output from the end of the prompt. The log saves the Logits tensor shape each time; the review script also uses the path of "Only prompt input, take the last position, and the full vocabulary SoftMax" to check the eight real scores. Only the two implementation paths are aligned to allow discussion of model performance.
The local probability formula and arithmetic auditing are independent teaching implementations; the neural network actually calls the official library forward, and there is no copy or replace the model layer. The source map gives pinned commit, specific functions, licenses and local correspondence. The data card license marks unknown, so the distribution package only saves identity, hash and derived results, does not contain original movie reviews; readers obtain data through a fixed download address.
A minimal experiment and its results
Officially run using CPU, Float32, Eager Attention and four threads, no training. The new generation side executes five hundred and twelve candidates forward, with two candidates for each; the single token situation can share the same prompt forward direction, this article retains the intuitive implementation, and clarifies its calculation redundancy. The final version of the official forward direction is about 19.55 seconds, the total time after importing the dependency is about 20.78 seconds, and the peak memory of the process is about 1.59 GB. The download is not re-measured, and the resources are from the caching of the caching; these numbers are not cross-hardware throughput benchmarks.
| Indicators, 256 items in the same group | DistilBERT Category Header | SmolLM2 candidate normalization |
|---|---|---|
| Number correct | 233 | 130 |
| accuracy | 91.02% | 50.78% |
| average maximum probability | 98.17% | 87.16% |
| Average Negative Log-Likelihood | 0.385259 | 0.996743 |
| Two Category Sum Brier | 0.169913 | 0.718332 |
Negative log-likelihood takes the negative logarithm of the corresponding probability of the original label, and Brier averages the probability square error of the two labels after summing; the lower the two items, the better. They can be used for comparison of the same limited classification task, but the build side must be explicitly used , not a full vocabulary probability. There is no form to reproduce a paper, and no more than 200 exploration samples are regarded as complete Benchmark.
The most obvious failure is: the generation side 256 articles are all selected Negative, and the correct number is exactly equal to the number of negative labels in the sample. The two models have 133 predictions inconsistent. The average mass of the two candidates is only about 46.84%, and the rest of the mass is assigned to the other next token. This is enough to show the missing link between "confidence between two candidates" and "free generation of answers are reliable", but the quality of the remaining vocabulary is not equal to the error rate: other tokens It may be a new line, an explanation or other legal expressions, and this article does not make a manual judgment on its semantics.
Evaluation protocol and troubleshooting
Check the identity first, then check the location, and finally look at the indicators. Independent audit of the original number, text hash and label, reconstruct candidate summation, binomialization, maximum value, correct number and average value; the classification side and the generation side are strictly paired according to the identity. The other real forward review confirms that the candidate decoding is Negative and Positive with spaces, respectively, which excludes the two interpretations of label order reversal and wrong one. Automatic recalculation only states that these deterministic steps are consistent and are not called manual review.
The full negative phenomenon still cannot be directly attributed to the model "just guess the negative". The original continuation interface, the word order in the question, the label word prior, and the chat template are not used, all may affect this checkpoint. The above is a competitive explanation in the author's inference, and the current experiments are not isolated one by one. Keeping failures is more valuable than replacing a better template after the fact: changing the template will change this article from a clearly defined measurement experiment to an unregistered search process. In the future, if the template migration is studied, the candidates, budget and selection data should be registered first, and then the control should be run.
It is also necessary to distinguish between correct arithmetic and scientific and effective. Independent scripts can find that the probability is not normalized, the sample is misplaced or the correct number of calculation is wrong, but it cannot judge whether the true emotions of the film review are ambiguous. This article follows the official label, does not add artificial semantic labels, so there is no need to forge new real person confirmation; in the future, if the text is rewritten and the label is maintained, there must still be a real review. The cache keys also need to include the model, tokenizer file, prompt, candidate, data identity, precision, code and protocol hashing, not just sample number.
Code and instructions for reproducing the results
This article has been published to a pinned commit: 31476a32d0a4 source code directory、README and run instructions、SOURCE_MAP source map. The remote file list and hashing have been checked with the local archive, and the arithmetic audit of the open copy and the eight real model test runs have been passed.
After installing fixed dependencies, run first python prepare.py --cache .cache Obtain and verify the resources, and then run the following unified entrance. Will end --smoke If it is removed, the official 256 articles will be executed; the expected output is a sample-by-sample candidate record, a comparative summary, and a configuration hash at the beginning.
python run.py --model-dir .cache/model --data .cache/validation.parquet --smoke
python audit.py
The arithmetic audit default checks the official results saved in the package, and does not regard the eight Smokes as an official evaluation. Real forward review use check_forward.py the same name resource parameters. Support README explains the amount of resources, environment, archived re-run and actual execution scope; there is no implied module that depends on the author's machine, and the reader does not need to open the local blog path. The official result remains frozen, and please do the re-run in the independent dungeon.
On a rerun, compare results example by example before comparing the aggregates. Similar means can hide mispaired samples, and the same number correct can conceal completely different sets of errors. We saved text hashes so that others can verify sample identities while respecting limits on redistributing the original text. If you change dependencies or hardware, report the largest numerical discrepancy and whether any decision changed. Running without an error is insufficient evidence of a byte-for-byte reproduction.
It can help to display the candidates’ total probability mass beside the largest normalized candidate probability. Their product, for the single-word candidates used here, is simply the original next-token probability of the selected token. Calling that “true accuracy” would ignore the meaning of the answer, the generation stopping rule, and how correctness is assessed. Keep those conditions explicit in metric names so later plots and abstention experiments continue to estimate the same quantity.
Testable and summary
This evidence supports a limited conclusion: under the same short review and label, the two types of scores can accept the same classification evaluation, but the candidate normalization changes the range of conditions of probability and cannot be directly interpreted as the correct probability of open generation. The failure interface of the generative model also reminds us that the lower average Confidence itself does not mean that the calibration is better, and it cannot replace the correct number and probability loss.
The next three different problems can be tested: the fixed model only changes the template, whether it is still completely negative; the fixed template replaces the equal-length label words, whether the candidate quality has changed; only the fitting temperature of the calibration set is isolated, and whether the probability loss can be improved. They need to be registered separately, and they can't modify all factors in one go and choose the best results. In this series, the next step is to enter the temperature calibration, and first establish a legal data usage agreement;
References
Search and access date: 2026-11-01.
- Chuan Guo、Geoff Pleiss、Yu Sun、Kilian Q. Weinberger,2017,ICML: On Calibration of Modern Neural Networks. It is used to calibrate the concept, and does not claim to reproduce the original paper value.
- HuggingFaceTB: SmolLM2-135M-Instruct Fixed Model Card.
- DistilBERT: SST-2 Classification Checkpoint Model Card.
- Stanford NLP: SST-2 data card.
- Hugging Face: Transformers pinned commit language model forward.
Apps by the author
The following is the author's own product promotion. In addition to recording scientific research experiments, I have also independently developed some daily gadgets. Welcome to understand: Daily Expense Journal to record revenues and expenditures, arranging budgets and savings objectives; One Small Action Help choose small actions according to the current time and energy, and record the feelings after completion. More technical articles and personal works can be accessed Author website. These products have no endorsing relationship with the conclusion of this study, and you are welcome to choose according to your own needs.