
- Date: 2026-10-09
- For graduate students, early-career researchers and research engineers
- Scope: Freeze small model, 64 development samples of SST-2, five presentation and scoring conditions; not a complete benchmark or no test.
Summary: Only more than one answer is correct, maybe 63 answers may have been changed
This actual verification: When the same model is scored in alphabets, after exchanging the content of the options, the correct number has changed from 32 to 33, which seems to have almost unchanged; but after mapping back to positive and negative emotions, questions 63/64 have changed the prediction. If the answer text is scored, only 1/64 of the change order is changed. Average Accuracy describes how many correct answers, but does not tell us whether the model is still making the same judgment.
In the previous study example selection and order, this article fixed the number of examples to zero, and the new question is: Can the content, position, letter and scoring string of the options be regarded as the same answer? We continue to use the model, data version and development question number, and add an auditable replacement mapping, candidate token record and question-by-question difference, and do not choose a highest score from five settings to improve it.
Why reproduce this work and where it fits
Multiple-choice evaluation often turns open generation into limited candidate comparisons, which is easy to execute, but introduces additional representations. In 2024, Pezeshkpour and Hruschka studied the influence of option order on the large model; Wang et al. [1][2] These papers support the necessity of research questions, and do not support moving the magnitude of their reports to this small model.
The difference between the paper report and this experiment: This paper does not reproduce the complete model matrix, free generation or de-biasing algorithm in the paper. SST-2 was originally a two-class emotional task; we added A/B list to the official text candidate protocol, which is a deliberately constructed measurement control. It can test the stability under this prompt, and cannot represent the full reading comprehension of the model. [3][4]
The core idea: first distinguish three "answers"
The original label zero means negative, and one represents positive. The positive sequence list is a corresponding negative and b corresponds to positive; in the desequence, a corresponds to positive and b corresponds to negative. The position letters are still arranged in A and B, and the content is moved. If a is still regarded as a negative after the reverse sequence, even if the model score is completely correct, the evaluation will be wrong.
For question , let the context be , candidate location the answer string contains token, the conditional log likelihood is
is the frozen weight, is the candidate token; only the answer part is accumulated, the closer the negative value is, the better. The model is not free to say the answer, but is compared between the two continuation strings we specify.
Map the display position as the original semantic label, is a data gold label, is the correct candidate position required for the official rating. The positive sequence mapping is zero, one, and the reverse sequence is one, zero; the semantic prediction must be restored first Then do cross-condition matching. The juxtaposed maximum value takes the first item, and the rule freezes before reasoning.
Compare the A/B probabilities under the same list, and estimate the alphabet continuation preferences; compare the negative/positive probability, and estimate the textual continuation preferences. We don't have additional instructions for "only answering the letters" or "only text only", so the two groups of contexts are exactly the same question. This isolates the change of the candidate space, but does not measure the obedience of the output format, and cannot assert that the two restricted probability are the choices when answering freely. [2]

The colors in the figure represent semantic content, and the letters indicate the display position. The two branching branches are mapped first, and then compared one by one; the length check reads the two candidates, and the icon in the schematic diagram cannot be regarded as a measurement result.
Beyond the formula: who is the denominator of length normalization
There are at least two definitions of common "normalization": Divide by the actual number of tokens to continue writing , or divided by the number of candidate string characters . This fixed harness acc_norm Use the latter; the pre-separated spaces participate in the tokenization, but do not count the number of candidate characters. [5] The same name does not guarantee that the mathematical object is the same.
In the actual tokenization, Negative, Positive, A, and B with spaces correspond to a single token, No. 3524, 2731, 330, and 389. The two emotional words are eight characters each, and the two letters are one character. Therefore, within each condition, the two denominators are equal: the three scores must maintain argmax, and the actual measurement is also the same. This is the algebraic nature of this setting, not the empirical conclusion of "normalization will never affect". We didn't change the synonym of different lengths to create effects after the fact, because that would change the wording of the answer at the same time.
A guide to the official implementation
First read the SST-2 configuration of the pinned commit, and confirm the question stem, Answer: End, Negative/positive order, and original labels. [4] Continue reading api/model.py of _encode_pair: Tokenizing the joined text can yield a different sequence from tokenizing the two parts separately and concatenating their tokens. The independent implementation checks that the context tokens form a prefix of the full sequence. It fails explicitly if the length exceeds 512, preventing silent truncation.
Look at the backend of Hugging Face _loglikelihood_tokens. The length of the question is , the complete sequence length , the input shape is , logits is ; the first answer token read position distribution. Final check process_results The labels, argmax and characters are normalized in. [5][6] This article directly calls the official likelihood and task scoring interface. The outer loop is a self-written teaching code, and there is no default task that claims to run completely unchanged.
The six actual installation files have been compared by byte with the fixed official source code, and downloaded for verification anonymously. SOURCE_MAP.md List functions, line numbers, licenses and local correspondences to avoid making "official implementation" an untraceable slogan.
A minimal experiment and evaluation protocol
The model is frozen with tokenizer for SmolLM2-135M-Instruct 12fd25f…. The complete revision is saved in the resource lock. [7] GLUE data is frozen as bcdcba79…, read the Train and Validation of SST2, only 64 development questions are inferred; 128 SST-2 confirmation questions and 64 news confirmation questions have not entered the model. [8] The same development set has been observed before, and this article belongs to exploratory protocol research.
The five conditions are the original text baseline, the positive and negative order of the list text, and the positive and negative order of the list letter. There are two candidates in each group, with a total of 640 candidate scores; each is calculated by independent forward and official back-end dual-paths. The main indicators are pre-determined as the number of semantic prediction flips and the number of wins and losses after the sequence change, and the accuracy and length diagnosis are auxiliary indicators. The statistical unit is still 64 questions, and 320 conditions cannot be described as 320 independent samples.
CPU single precision, four threads, EAGER attention, seed 96, close chat template, training and inference cache, no BOS or EOS. Cache identity save model, tokenizer, data, prompt construction code, configuration and scoring protocol. The context of the four list conditions is the same length, 27 to 69 tokens; the total number of candidate inputs is 5530, and the original baseline is 4506. There are more tokens in the list than the baseline, and this comparison cannot be called strict and so on.
What we tested: what the total score covers up
| Condition | Number correct | Predict the number of positives | Select the first position |
|---|---|---|---|
| original text baseline | 32/64 | 3 | 61 |
| List text order | 31/64 | 0 | 64 |
| List text deorder | 32/64 | 1 | 1 |
| List alphabets | 32/64 | 1 | 63 |
| list letter desequential | 33/64 | 64 | 64 |
Only the question number 170 of the text is flipped, and a new question is added to the correct order. Compare the two candidate spaces under the reverse sequence, and there are also 63 questions with different semantic predictions. The positive sequence is only different in question number 287. If you only write "slightly", you will miss the main evidence of this group.
This table also has an easily overlooked background: the development label contains 31 negatives and 33 positives. The desequential letters are all positively predicted, and 33 questions are correct; this number has been explained by the distribution of the category, and it cannot be said that this setting will understand emotions more. Correspondingly, if all the positive orders of the text are predicted negatively, 31 questions can also be obtained correctly. Reporting the constant category baseline can prevent us from miswriting one or two percent as a task ability improvement.
Candidate scores themselves also have boundaries. A string has a high probability in the two, which does not mean that the model assigns most of the probability mass to these two strings; other words, newlines or interpreted prefixes may still be more likely to appear. In particular, the negative logarithm probability of the text branch cannot be directly subtracted from the alphabet branch, which is interpreted as "how much confidence is improved": the two sequential spaces are different and lack a common calibration definition. This article only compares the semantic results after the selection and restored within each space, and does not report the cross-space Confidence gain.
Question number 400 provides an example of recalculation: the positive order letters A and B score about −5.079, −5.904, Select A and map it to negative; the reverse-order score is about −3.946, −4.844, and A is still selected, but it is mapped to the front. The original label is positive, so it becomes a gain. Question No. 615 also keeps the A selection of A, but it changes from the right to the wrong one. The two cases do not need to write a model psychological story, which can explain why the benefits and losses are offset by each other.
The author's inference: The letter branch shows a strong A or first position preference, and the text branch mostly retains negative predictions; but A is always in the first position, the two are mixed in this experiment, and it is impossible to distinguish symbol priors and position preferences. More stable text doesn't mean it's more correct: it's almost always negative. This is a counter-example of stability and ability must be reported separately.
The independent recalculation covers 320 records and 640 candidates, and the maximum difference between the official score is zero; the label mapping, the correct number, the paired gain loss and shape inspection are all passed. It is estimated that the official operation of the two questions will run for about 280 seconds, which is lower than the budget of 1200 seconds; it takes 43.56 seconds after the official process is imported, the peak RSS is 1568227328 bytes, and the external command wall clock is 47.12 seconds. The GPU memory is not measured, and the resource number does not contain dependencies downloads.
Failure investigation: first doubt the agreement, then explain the model
The first test run uses the newly added label field as a string configuration, the framework understands it as a template text, and the official indicator consistency assertion fails. After changing to a function that reads this field, the missing field error is triggered because the task initializes the original sample to detect the original sample. Eventually, the replacement position is calculated directly from the original Label, and the initialization and formal score are satisfied at the same time. The two failures and the repair before the repair were reserved, and only the report was successful after not being deleted.
If an abnormality is encountered, the sentence hash and question number, original label and display position, pre-candidate space, causal shift, and upper limit of the length should be checked in turn, and then checked cumulative and normalization. Independent scripts do not import reasoning programs; six mutation tests actively destroy labels, scores, numbers, duplicate records, token numbers and predictions, all of which are rejected. This is automatic arithmetic re-calculation, not manual review. The OneDrive environment starts blocking and restores the restoration of the same version of the temporary environment, and the model and data hashes have not changed.
Code and instructions for reproducing the results
The supporting source code has been disclosed: Source tree at the pinned commit、README and run instructions、SOURCE_MAP source map. The 43 files have been anonymously downloaded, and the file hash is consistent with the local archive; all links are fixed to the same commit.
Install fixed dependencies and use by README prepare.py After obtaining the hash-locked resource, the minimum real running command is:
python run.py --cache ./cache --out ./replay-smoke --smoke
It is expected to generate question-by-question candidate records, summary, environment, stage status and identity files; remove --smoke Execute 64 questions to the official scope. audit.py The result of the saved can be recalculated offline. This real operation covers all development questions covering five conditions, excluding generating, training or artificial gold labels. The public package does not distribute the original text of the film review, the question stem token and the model weight that is not clear, and retains the fixed download portal, index and derived score; the local ZIP is used for filing and acceptance.
Testable Research Questions and Summary
In the next step, the rest of the settings can be frozen, so that A/B maintain their own semantics, only change the display row order, and then design the control composition factor design with the current exchange content, distinguishing the letter identity and physical location. If the preference follows A to move, it supports symbol interpretation; if the first line is always followed, position interpretation is supported. The budget should be registered before the official implementation, and the story cannot be added after reading the results.
An open question is whether probabilities restricted to the available choices agree with what the model actually generates. The next article turns to answer extraction and preparing a small, human-annotated gold standard. Checking arithmetic here cannot settle questions of meaning. One finding is already reproducible: overall scores just one question apart can conceal sixty-three reversals in the meaning of the selected answer. A research report should let readers trace those changes behind the ranking.
References
Retrieve and open the date: 2026-10-09; save the anonymous download and hash verification records for the fixed source code.
- Pezeshkpour, Pouya; Hruschka, Estevam. 2024. Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions. Findings of NAACL, Conference Edition.
- Wang, Xinpeng et al. 2024. “My Answer is C”: First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models. Findings of ACL, conference version.
- Socher, Richard et al. 2013. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. EMNLP.
- EleutherAI. Fixed SST-2 task configuration,harness 0.4.9.2.
- EleutherAI. process_results and character length normalization.
- EleutherAI. Hugging Face candidate like then end.
- HuggingFaceTB. SmolLM2-135M-Instruct Fixed Model Card.
- NYU. GLUE fixed data card, see resource lock for SST2 configuration and data version.