Cover illustration

  • Date: 2026-10-10
  • For graduate students, early-career researchers and research engineers
  • Scope: 64 of the fixed small models and SST-2 have existing development questions; 16 pre-selected review samples are pre-selected. At present, there are only overall review and confirmation, and no one-by-text labels have been submitted; no recalculation-to-issue labor gold labels are provided.

Summary: The same batch of answers, why did it change from zero to 50 points

The previous experiment found that the change option presentation and scoring space may change the answer even if the model remains unchanged. Continue to ask this time: The model has been answered, and how many judgments have been changed in the post-processing program? We freeze the real model and the output prompt, and only hand the same answer to the complete string matching, strict format parsing and regular extractor. The actual result is: the complete matching of zero questions is correct, and the first emotional word is selected for fifty questions correctly.

The model’s ability has not changed, and it would be premature to say we recovered fifty answers that were wrongly marked. The two programs may measure different things. A permissive extractor can pick up an answer word inside a quotation, a negation, or an unnecessary explanation. This article provides the reproducible automated procedure and tools for drawing a review sample in advance. Only an overall review confirmation survives, with no item-level labels from which to recompute the result. The “small gold standard” in the title describes how to build one. The original blank form supplies no item-level semantic evidence.

1. Reproducing value: give the scorer an independent reference

Exact match, that is, Exact Match, is the method to determine whether the output is consistent with the reference string. It is easy to recalculate, but relies on how to write the reference answer. Kamaloo, Dziri, Clarke and Rafiei In ACL 2023 paper, the open domain question answering output was manually evaluated, and the question of lexical matching of omission semantic equivalent answers was discussed. This is What the paper reports; This article does not copy its tasks, models or increase the range, nor does it directly extrapolate the question and answer conclusions to emotional classification. Entrance to the original paper

The object of this research is SST-2 film review two categories, following the previous development questions and the original SmolLM2-135M-Instruct. The advantage of a simple task is that it can be audited one by one, but "positive" is still not the same as "response to judge the film as a positive". The original data label answers the emotion of the film review; the new artificial label answers what the model output expresses. These two layers of labels cannot be substituted for each other. SST original paper

We have seen the JSON format failure in existing experiments, so choosing this prompt has the exploratory motivation. The current sample is not a new blind test, and the follow-up cannot be called a scorer that has been independently verified because a certain rule has a higher score.

2. Core ideas: separate strings, formats and meanings

For question ii , let the full generated response be yiy_i, the dataset emotional tag is gig_i, the specs JSON string is c(gi)c(g_i). The complete match is defined as:

Ei=1[strip⁡(yi)=c(gi)]E_i=\mathbf{1}[\operatorname{strip}(y_i)=c(g_i)]

here EiE_i is zero or one, indicating that the function only determines whether the condition is true; strip Remove the blanks at both ends. The canonical string is fixed to a JSON with only the Sentiment key and a space after the colon, so the internal layout also affects the result. This strict definition is a local diagnostic rule, not assertion that it is the exact match used by all tasks by default.

In order to avoid failing to mix the layout difference into the format, we also do JSON parsing: the entire answer must be a legal object, only one Sentiment key, no duplicate key, and the value is lowercase positive or negative. The analysis is successful and gig_i Consistent, the format and content are correct. Code fences, extra keys or natural language interpretations do not satisfy this output convention.

third path order r(yi)r(y_i) The first full sentiment word captured for a regular expression, ignoring the case; an invalid value is returned when not found. Extract the correct indication as Ri=1[r(yi)=gi]R_i=\mathbf{1}[r(y_i)=g_i], the tag is unified as an emotional name before comparison. It may tolerate interpreted text, or may catch the wrong word in "Not Positive". This example is used to illustrate the limitations of the rules, which is a teaching counter-example, not a false claim from a formal sample.

Only after obtaining the tags of real people can they introduce content judgment HiH_i, and undisputed sample collections in the judging and reference labels DD Calculate on:

e^=1∣D∣∑i∈D1[Ri≠Hi]\widehat e=\frac{1}{|D|}\sum_{i\in D}\mathbf{1}[R_i\ne H_i]

HiH_i Indicates whether the content is correct, ∣D∣|D| It is an actual number of questions that can be judged and referenced without controversy, rather than always written as sixteen. False positive is also reported separately, that is, the program judges the right one and the person is wrong; False negative is the opposite. Both the program and the real person can make mistakes, and different answers may be drawn, so check whether the extracted label is consistent with the final position of the real person. Currently, not received a question HiH_i, even if there is an overall review confirmation, the misjudgment rate remains null.

3. The guide to the official implementation: the two "firsts" are not the same

Read the fixed version first RegexFilter.apply: it does regular lookup for each generation, group_select=0 Select the first match in this configuration, and if there is no hit, a fallback value is given. Read again TakeFirstFilter.apply: The "first" here refers to the generation of the first item of the same question, not the first emotional word in the text. We only generate each question once, but still retain the input and output convention of the official nested list. Official extraction implementation

Finally, the chat template and generation function check along Transformers: system messages, user prompts, where to join the start of the answer, and which tokens are new answers. Only score newly generated and avoid drawing positive or negative from the problem. The five key files have been downloaded from the pinned commit, and are checked by byte with the installation version; the specific functions, line numbers, licenses and local corresponding paths are shown in the source code map. We call the official filter and the real build library, but we do not claim to run the Harness standard SST-2 full task.

4. Minimum experiment: first select artificial samples, then look at the output

Experimental flow

The artificial branches in the figure indicate the original pre-registration workflow; there are currently no manual labels for each question, and the dotted line does not mean that quantitative artificial indicators have been obtained.

The model and tokenizer are fixed in 12fd25f77366fa6b3b4b768ec3050bf629380bac, GLUE data is fixed in bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c. Continue to use the 64 development questions selected according to the existing hash rules, including 31 negatives and 33 positives. 128 SST-2 and previous 64 news confirmation samples did not participate in reasoning. Model card at the pinned revision

The newly added artificial subset is determined before this reasoning: each category is sorted by SHA-256 composed of question numbers and fixed strings, and then staggered and arranged. Sampling scripts, sample lists, movie review hashes and time are saved together, and there are no discrepancies based on generated results or scores. The sampling rates of the two types are eight divided by 31, eight divided by 33; if all the samples can be determined, when these sixty-four questions are pushed back, they are weighted according to the original category, and the average value of the small set that accounts for half of each half cannot be directly estimated as the original distribution estimate.

Generate a prompt to reuse the previous JSON convention, requiring only one Sentiment key to be output. The CPU uses single precision, four threads and greedy decoding, and each question generates up to 32 tokens, and does not train, search for prompts, and do not add examples. If you enter more than 512 tokens, an error will be reported; the actual range of this time is 66–108, and there is no truncation. All scoring units share the same output, and will not get more reasoning budgets because of the extractor.

Run the two questions first, the peak memory is about 1.28 GB, and it is conservatively estimated that the official part is about 23.40 seconds, which is lower than the preset 1200 seconds and 4 GiB upper limit, and then run all the questions. After the import is officially executed, it takes 10.37 seconds, the external process wall clock is 14.69 seconds, and the peak resident memory is 1,280,081,920 bytes. The first question input shape is [1,83][1,83], the vocabulary score shape is [1,83,49152][1,83,49152]: The three dimensions are the batch, input position and vocabulary size in turn. This is a native measurement, not a generic performance benchmark.

5. This actual verification: the difference is established, and the explanation has not yet been established

Fixed scoring rules 64 Questions Actual Count Conclusions currently supported
Specification JSON complete match 0 correct questions No output matches the full reference string
Strict JSON parsing 0 questions are successful, 0 questions are combined correctly Failure is not just space after the colon
Official first emotional word extraction 50 correct questions Label extraction is the same as the dataset reference
Real person audit record Only the overall review and confirmation; the question label is not submitted Can't calculate manual reference misjudgment rate

The result of matching-by-question is that the extraction path is increased by 50 correct marks, but there is no reduction; the other fourteen questions are calculated incorrectly. Divide fifty by 64 as "extractor error correction rate", because there is no question-by-question semantic reference for readers to recalculate. A total of 5250 input tokens are used for all outputs, and 422 tokens are generated (including end marks); the sixty-three questions end normally, and one question encounters the upper limit of length.

In the output, there are separate uppercase emotional words, as well as explanatory sentences, and an answer with code fences and additional rating and outcome keys. These are actual records of the verifiable string phenomenon. At present, there are only overall review confirmation, and these outputs are not submitted for question-by-question positions; the article does not convert the overall confirmation into one-by-question judgment, and does not change the author's intuition to manual results.

The independent script re-parsed all 64 records, and checked the output hash, rule results, correct number, matching question number, total token and official extraction results. Six rule-boundary samples and six types of evidence break tests passed; blank artificial tables were indeed rejected. These inspection and proof software conventions can be recalculated, and cannot prove that the convention has semantic validity.

6. Evaluation protocol and failure investigation: the artificial table must allow "uncertain"

The audit portal displays movie reviews, data set references, complete model answers and end reasons, and hides automatic draw labels and scores. The original design requires that the final position, content correctness, format compliance, reference label dispute and reasons be filled in question by question, and the reviewer and time will be recorded with the exported documents. All fields start with blank space. At present, only the overall confirmation record, recording time and sample hashing are retained; the question-by-question review table is still blank, and no judgment of any question is presumed according to this.

When there is a contradiction in the position, there is no classification conclusion or the meaning of the citation is unclear, you should choose uncertainty, rather than forcing the two to choose one. If you have any questions about dataset references, you should mark them separately and keep the original label; you can't quietly rewrite the benchmark with one review. In the future, the report should be listed for the number and reasons for exclusion. Only the eight questions of each of the two categories can still be determined, and the predetermined weighted description amount can be calculated. Even if the small set is zero, it is not enough to prove that the scorer is reliable in other hints, models or domains.

The investigation sequence starts with identity: first check the question number, output hash and label, and then check whether the entire chat is wrongly scored, and finally distinguish between no matches, multiple matches, format failures and length stops. The only length stop question keeps the original result, and the decoding is extended separately after failure. The old results are preserved when changing rules, and the output that can reveal the negative example is not covered.

7. Checkable follow-up questions

Of the fifty scoring disagreements, how many involve correct content in a noncompliant format? That requires semantic labels for each item. A second question is whether the first-word rule yields false positives on answers containing a negation or expressing several positions. If the presampled subset contains no such answers, the possibility remains untested. Third, do strict parsing and permissive extraction disagree mainly on outputs stopped by the length limit? The current sample contains only one such case, enough to suggest a hypothesis but too little to attribute a cause.

If you read the manual label and then modify the extractor, these sixteen questions will enter the development purpose. The new rules need to be verified separately for samples that are not used for modification; it cannot be corrected while continuing to call the original small set as an independent gold standard test. The manual guide is also frozen when comparing across the model, otherwise the scoring criteria will be disguised as model differences.

Code and instructions for reproducing the results

The source code has been made public and pinned to commit 6d7cdad04b701a4a8e4cdf698d486a47d738f857: Full source code directory、README and run instructions、SOURCE_MAP official source map. All 47 files were retrieved anonymously and checked individually against the local archive. The public copy passed an independent audit, software tests, and a trial run on two real questions. The packaged protocol and results preserve the pending-review status recorded at the time, with a separate note documenting overall confirmation. No item-level labels were submitted, so the quantitative metrics based on human review remain blank.

After preparing the Python 3.12 environment, press README to install a fixed dependency and obtain a hash-locked resource, the minimum running command is:

python run.py --cache ./cache --out ./replay-smoke --smoke
python audit.py ./replay-smoke

It is expected to generate question-based records, aggregation results, environmental information, stage status and input identity files. In this case, the actual execution of the two test runs and all 64 questions; all self-writing modules are provided with the package. The download cache can be reused, but each time a prior hash; does not distribute weights, original movie reviews, full tips or a local review page with movie reviews. The public code can reconstruct the audit portal in the reader's machine.

Summary

The difference in scoring is the first measurement phenomenon, not ability improvement. This time, freeze generation, three scoring definitions, question-by-question difference and pre-sampling materials are preserved, but there is still a lack of manual-to-question manual labeling. The overall audit confirmation cannot replace the recalculated semantic evidence of recalculation. Automatic disagreement can be recalculated; artificial misjudgment rate, false positive and false negative still do not report, which is the boundary of this conclusion.

References

Retrieval and actual access date: 2026-10-10. The fixed source code link that failed to read the web page is also anonymous download and check.

  1. Ehsan Kamalloo、Nouha Dziri、Charles Clarke、Davood Rafiei,2023: Evaluating Open-Domain Question Answering in the Era of Large Language Models,ACL.
  2. Richard Socher et al., 2013: Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank,EMNLP.
  3. EleutherAI, 0.4.9.2 pinned commit: Task Configuration and Filter Guide.
  4. HuggingFaceTB: SmolLM2-135M-Instruct Fixed Model Card.
  5. NYU-MLL: GLUE fixed data card.