
- Date: 2026-10-07
- For graduate students, early-career researchers and research engineers
- Scope: Fixed small model and official evaluator, 64 development samples; not a complete GLUE benchmark or no test set .
Summary: First ask how the score is produced
The first six articles discuss the cost of format and content around Fine-Tuning. This article enters the third unit, first does not change the model, but disassembles the machine responsible for scoring: also called "SST-2 Accuracy", input how to splicing, how to score the answer, and how to number the sample, it may be different. Writing only "Use Official Reviewer" is not enough to allow another researcher to reproduce the same measurement.
The actual verification this time: Using the original SmolLM2-135M-Instruct, you get 32/64 under the official task configuration; the score of 128 candidate answers is independently recalculated, and the difference is zero. What's more valuable is that although the first complete operation returns the same total score, the log question number has 61 mismatches and is stopped by the identity assertion. This is not a story of "how much the model improves", but an experiment that "whether the evidence corresponds to the correct question".
Why reproduce this work and where it fits
Task is an executable description of a task in the evaluation framework, which usually includes data, division, hints, candidates, answers and indicators. YAML saves the configuration format of these fields, which is easy to read, but does not mean that all behaviors are written on this page; unfilled fields will still get the default value from the frame.
Paper report: Socher et al. proposed the emotional tree library in 2013, and GLUE paper then incorporates SST-2 into the multitasking language understanding evaluation. [1][2] This supports the source and research position of the task, and does not support the use of the 64 subsets of this article as a complete GLUE conclusion. LM-Evaluation-Harness is a software project; this time it uses the 0.4.9.2 fixed version released in 2025, which does not claim that it is the latest version, and does not make a new paper with the same name as the software. [3]
Compared with 093, the new question in this article is "Whether the official score can be recalculated item by item"; freeze the original model, existing development samples and data labels, and the new variable is a measurement protocol. We didn't mix the checkpoints after 093 training. New products include valid configuration, candidate token scores, official question-by-question results, identity checks and traceable failure records.
Core idea: There is also a computing chain before Accuracy
The official SST-2 task connects the original sentence to the fixed problem, and the end is Answer:, the candidates are negative、positive.[4] multiple_choice It is represented here for scoring a given candidate, not requiring the model to freely generate an option letter. let the context be , first Candidates are divided into token, then
corresponding to the same sample, corresponding candidate positions, It is the weight of the frozen model. The larger the candidate log, the better the better; the negative number should choose the one that is closer to zero. The formula only accumulates the candidate token, and does not mix the probability of the entire question stem, and does not automatically divide the candidate length.
is a data tag, It is the actual number of evaluation lines, this time is 64. The two candidate positions are mapped to zero and one respectively, and the first candidate is the first candidate when the score is exactly the same. The final average is the correctness of the question from one to the other, and the score of all negative words cannot be averaged first and then compared with the mean of positive words.
The model input is a line of token numbers, and the shape is after removing the last candidate token and output logits as . The vocabulary dimension is 49152. The length of the question stem is , the candidate's first token is determined by the position prediction. The shifting here is the same as the training cross entropy, but no backpropagation or weight update.

The original sample in the figure provides both the question stem and gold labels at the same time; the gold label only enters the calculation and review of right and wrong, and does not enter the path of the model's answer. Manual Check said that the Manual review link in the original design has not been confirmed by real people; this time, the independent script arithmetic re-calculation using the independent script disclosed below is adopted.
Guide to the official implementation: from short configuration to default behavior
The first step is to read the pinned commit glue/sst2/default.yaml, rewrite each field into a checkable question: which data to read, which division to take, what exactly does the model see, and which candidate is zero? Don't read the task name only.
| level | This valid setting | meaning to check |
|---|---|---|
| Data | GLUE's SST2 configuration, validation | Validation set with tags, which is used as a development set by this project |
| Prompt | Original sentence, fixed problem, Answer colon | No chat template, no additional system prompts |
| Example | num_fewshot=0 | The actual addition of zero examples does not mean that the model has never been trained in command |
| candidate | negative、positive | Map the labels in this order, and fill in a space in front |
| Export | multiple_choice | Calculate candidate likelihood, there is no free generation and answer extraction |
| Polymerization | acc、mean | The mean value comes from indicator registration, YAML is not explicitly written |
read the second step TaskConfig、fewshot_docs and construct_requests.[5] target_delimiter The default is a space; it is not conspicuous, but belongs to the actual rating string. The training is divided into train, and the corresponding sample source will be used when adding examples in the future, but the number of examples in this prompt is indeed zero. Not setting the FEW-SHOT division does not mean that you can find any examples from the evaluation questions.
In the third step, follow _encode_pair into the Hugging Face backend. [6] For a causal model, the backend tokenizes the context and candidate as a single string, then locates the boundary. The independent implementation also asserts that the prompt tokens form a prefix of the joined sequence. If a token crosses the boundary, encoding the parts separately and joining them can give a different result. Here the two candidates, each with a leading space, take one token apiece, with IDs 3524 and 2731 respectively. No EOS is appended and no length normalization is applied. Other models or answers may tokenize differently.
Final reading process_results argmax with metrics.mean, read the official simple_evaluate entrance. [5][7] The seven installation files have been checked byte bytes with the fixed official submissions. The source map gives specific functions, line numbers, licenses and local corresponding modules, and readers can locate the responsibilities of each layer.
Minimum Experiment and Freeze Protocol
The model and tokenizer continue to use revision 12fd25f77366fa6b3b4b768ec3050bf629380bac. [8] The frame is fixed to ad3f4d0cad1cfcdb815f1e795f7947e49ed9f2e9. Configure the GLUE data pointing to bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c, and check its 67349 training and 872 verification records are consistent with the previous Stanford copy; the sentence, label and number are confirmed, and the two compressed files must not have the same hash.
64 pieces of development and 128 confirmations are still obtained in the hash sequence of 088; the 64 pieces of news confirmation reserved by the previous unit have not been reasoned. Reading confirmation rows to save the division identities does not mean sending them into the model. The data source license tag is unknown, so the public package saves index, hash and derived scores, and does not distribute the original text of the movie review, complete prompts or the token number of the question.
official support custom_dataset The interface, locally use it to read the verified fixed Parquet, which changes the download and cache entry; the original task name, prompt, label, candidate and scoring logic are consistent. The difference must be explicitly stated that the custom transfer cannot be described as a default command with no coverage.
CPU single precision, four threads, EAGER attention, candidate batch 1, random seed 94, length upper limit 512. The independent implementation encounters a super-long input and fails directly; this time the context is 19 to 61 tokens, and there is no truncation. The request cache and the model response cache are closed, and the saved identity keys still include the model, tokenizer, data, hints, scoring settings and code hashing, to avoid future reuse of wrong results.
What we tested: For the total score, you must also
First, the four questions are independent and real forward pass, which takes 0.93 seconds, and the peak memory is about 1.31GB; it is conservatively estimated that the formal process is about 119 seconds, which is lower than the 600 seconds and 4GiB budget. After the official SMOKE of the two questions is passed, the 64 questions will be officially verified. After successfully running from a third party, the time is 12.10 seconds, of which the independent score is 4.81 seconds, and the official call is 5.81 seconds; the peak RSS is 1310195712 bytes, and the GPU memory is not measured.
The final official and independent implementation is 32/64, and the maximum absolute difference between 128 candidates is zero; all labels, prompt hashes, candidate boundaries and final question numbers are the same. The model predicts 61 negatives and 3 positives, and the development labels are 31 negatives and 33 positives. The total score is close to the accidental level of the second classification, and it does not automatically prove that the model can only guess; there is also a obvious label bias observed here.
The four-item review packet lists the actual likelihood scores. On item 400 they are -3.8741 and -4.7651. The gold label is positive, but the scoring rule selects negative and marks the answer wrong. On item 615, the selected negative label agrees with the gold label. Every candidate for these four items consists of one token, so the sum is simply that token’s score. This makes the items a convenient first check on score direction, label mapping, and the mean.
Review status (updated on 2026-10-08): The original plan requires real people to recalculate four questions, but the actual results have not been received, and the official verification has been completed first; this sequence deviates from reservations. The pure arithmetic inspection has been changed to independent script recalculation: all four questions are predictive, and the correct number is 2/4; the complete 64 question recalculation is 32/64, which is consistent with the official question-by-question record. Input, script hashes, and execution results are archived. The current acceptance has passed, but this is not a manual review, and there is no new semantic gold label.
Failure investigation: the correct total score may also leave wrong evidence
The first 64 question invokes a list of question numbers passed into the hash order. This fixed version filters the questions in the original order of the data, but backs the logs in the order of the caller list doc_id, so 61 log question numbers and original data idx inconsistent. Independent identity assertion lets the program stop with the exit code. Just looking at 50% of the total score will completely miss this problem.
Fix only the incoming question number is sorted, and the 64 question set and all candidate scores have not changed; run again. The initial return, failure logs, repaired pre-source code and identity differences are reserved. This is for the normalization of input for this version, not claiming that the latest framework still has the same problem, nor is it to pick up test questions through debugging.
There are also hierarchies for review. To recalculate the mean of existing scores, you can only find aggregate errors; re-code the question stem to check the identities of the prompts and samples; the independent execution of the model forward and the probability to be covered by token, can cover the scoring path. These three layers of records are saved in this time, and there is no replacement for the previous inspection with the last total score. Even if the two implementations are exactly the same, they may still share the same pre-trained model, tokenizer, or wrong task assumptions, so consistency cannot prove that semantic labels are absolutely correct.
In addition, when the official tries to read the Git information of the calling directory, the non-repository prompt is output, and the process is still returned normally; this project uses the actual pinned commit and the hash of the installation file to confirm the source, and does not regard the version number of the calling directory as an evaluation device version. Warnings, true failures, and verified successes must be recorded separately.
Also, don't interpret the 57/64 of 32/64 and 088 directly as a decline in ability. The weights are the same, but the prompt form, chat template and scoring method are changed at the same time; the former is a candidate likelihood selection, and the latter is free generation and parsing. The gap poses a protocol sensitivity problem, but there is no separate isolation of which factor causes the difference.
Testable Research Questions and Summary
The author's inference: The current negative bias may come from the co-action of the cues form with the candidate prior. In the follow-up, a set of scoring rules should be fixed first, and then the example selection and order in the training division should be changed respectively, and the matching results of the same question should be saved; All factors cannot be changed at the same time, and then label a single cause.
The starting point of the third unit is therefore not a new ranking, but a measurement description: data version, title number, complete prompt, candidate boundary, scoring formula, aggregation and failure can all be traced. The agreement review only states that "these numbers are calculated according to the declared method", and do not indicate that the model has a stable ability, and does not restore the confirmatory evidence lost by the repeated use of the development set.
Code and instructions for reproducing the results
The source code has been made public and completed the pinned commit verification: complete directory、README and run instructions、SOURCE_MAP official implementation map. The 53 distribution files are the same as the local byte; the copy exported from the public version also runs through the real model. The fixed source code package retains the manual review history of October 7; on October 8, only the acceptance method is adjusted and independent arithmetic verification is supplemented, and the model, data, code and original results have not been changed. The current completion status does not mean that the real person review has happened.
Install dependencies, use prepare.py After obtaining and verifying the resources, the minimum real entry is:
HF_HUB_OFFLINE=1 TOKENIZERS_PARALLELISM=false python run.py --cache ./cache --out smoke-new --private private-new --smoke
The command is rated two and four candidates, and keeps official and independent records; removes --smoke Run the same 64 development samples. follow-up python audit.py --cache ./cache --out smoke-new, offline check identity and arithmetic. At present, four independent calculations, two official test runs of two questions, a complete operation that has been intercepted, and a complete operation after sorting and repairing have been performed, and there is no training model or evaluation reserved confirmation set.
References
Search date: 2026-10-07. The software version, data and model are accurately hashed to see the source code resource lock; the paper only supports its corresponding research background.
- Richard Socher et al., 2013, EMNLP: Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank.
- Alex Wang et al., 2018 first edition, ICLR 2019, arXiv v3: GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding.
- Lintang Sutawika et al., EleutherAI, 2025: LM-EVAL v0.4.9.2 software release record.
- EleutherAI: Fixed SST-2 task configuration.
- EleutherAI: Fixed Task implementation.
- EleutherAI: Fixed token boundary implementation、Hugging Face backend.
- EleutherAI: Fixed Evaluation Entrance、Indicator registration and aggregation.
- HuggingFaceTB: Model card at the pinned revision;NYU: Fixed GLUE data card.