Scientific research devices for isolation confirmation from candidate searches

Abstract

The previous experiment calculated a paired interval for two fixed checkpoints, leaving an earlier question unanswered. Why compare those two in the first place? If a researcher tries several prompts, demonstration orders, and answer formats, then puts the highest score in the abstract, even a correctly calculated interval may miss the optimism introduced by selection. Here we show the development results for all ten protocols and choose one before opening the 128 examples held back for confirmation. The aim is to make the search, freeze, and confirmation stages auditable. A gain discovered through search need not disappear.

Reproducing value: also treat the selection process as an experiment

Selection deviation means that after the evaluation object is data-driven filtering, the results during the screening are no longer as easy to explain as the results of the pre-specified object. It does not require training models, nor does it require intentional concealment: change the prompt several times, only remember the most pleasing line, and is already making a choice.

What the paper reports: Cawley and Talbot's 2010 study pointed out that the model selection criteria on limited samples will also be overfitting, and the selection steps must be included in the performance evaluation design. [1] This article borrows this problem framework, and does not reproduce its nuclear method experiments, let alone put the scale of the deviation in the paper on the language model.

What we tested: Reuse the real SmolLM2-135M-Instruct and public SST-2 data, and only contact 64 development questions for ten settings; after selection, only the winner and fixed Zero-shot baseline are evaluated. Compared with the previous article, the selection process of the evaluation protocol is changed, and what is frozen is the original model, data division and semantic Accuracy definition. The new product is a frozen file containing all attempts, time and source code hashing.

Core Ideas and Formulas: The highest score is not free to get

decorate jj Accuracy set on the development set is a^j\hat a_j, the selection rule is

j^=argmax⁡1≤j≤10a^j.\hat j=\operatorname*{argmax}_{1\le j\le10}\hat a_j.

here jj Indicates a set of prompts and answer scoring agreement, j^\hat j It is the selected protocol number; the same points are selected in the order listed in advance. Even if there is no system offset for the estimations of each setting, the maximum value is inclined to retain favorable fluctuations. A basic relationship is

E[max⁡ja^j]≥max⁡jE[a^j].\mathbb E[\max_j\hat a_j]\ge\max_j\mathbb E[\hat a_j].

The expectation is that for repeated sampling, it is not simply averaging the ten numbers; the settings can be related, and this inequality still holds. But it doesn't tell us how optimistic this time is, and it doesn't mean that all the settings are the same. Treating ten tests as ten independent coins is usually not in line with the fact that shared models, questions and templates are used.

How the model gives the answer should also be fixed. to context xx and candidate answers cc, adopt

s(c∣x)=∑t=1∣c∣log⁡pθ(ct∣x,c<t).s(c\mid x)=\sum_{t=1}^{|c|}\log p_\theta(c_t\mid x,c_{<t}).

θ\theta is the frozen weight, ctc_t is the tt-th candidate token, ∣c∣|c| It is its length; only ask for the logarithmic probability of the candidate part, and do not include the token of the question. If the two candidates are larger, they will be mapped back to the original negative or positive label. The letters A/B only display the position, and the label mapping changes accordingly.

The complete record is frozen after search, and only allow the set to enter the final matching report

Guide to the official implementation: first follow the data, then follow the score

Reading from the pinned version of the SST-2 task configuration: it declares the public GLUE data, the original problem template, two text candidates, and Accuracy . [4] Then see _encode_pair How to deal with context and sequential writing boundaries, and then look at _loglikelihood_tokens The probability of which position is taken, and finally enter process_results Check argmax and tags.

This article directly calls the official HFLM.loglikelihood and ConfigurableTask.process_results, did not take the whole simple_evaluate Runners are called reproducing. In addition, the independent path is another real Transformer forward pass to check the candidate token, causal displacement and score. The source map gives the permanent file number of the permanent file that has been actually downloaded and checked; the seven official files are consistent with the installation version by byte. Statistical part reads SciPy's bootstrap: The pairing switch determines whether the two schemes use the same batch of resampling question numbers. [5]

Minimum Experiment: What Are Ten Settings?

Model with tokenizer fixed in 12fd25f…, GLUE is fixed in bcdcba79…. The complete revision and 12 resource file hashes see the resource lock. 64 development samples were still taken according to the original question number hash order, and the latter 128 confirmed samples were not intersected. The data comes from the public verification division, and cannot be called the official test list. [6][7]

The nine historical protocols come from the previous example sequence and option experiments, and the original zero-shot protocol is counted only once. Add the tenth only Review: {sentence}, the next line is written Sentiment:, the candidates are still Negative and Positive. All settings are written to the configuration before this new reasoning, and the candidates are not added after confirming the results. The two groups of four examples only take the Training Set, and follow the fixed question numbers and sequences.

development settings Correct number/64
Original Zero-Shot 32
Four Example A: Positive Sequence/Inverse Sequence 31/37
Four Example B: Positive Sequence/Inverse Sequence 33/44
List options, answer text: positive order/reverse order 31/32
List options, answer letters: positive order/reverse order 32/33
short prompt 52

This table retains all ten items, including results below baseline. Short Prompt wins 52/64. The nine historical achievements have been seen before, so this is not a brand new ten-arm research registered in advance; to be precise, it is a forward-looking confirmation of a retrospective candidate collection. The old model fine-tuning results are not mixed into this table, and the artificial scoringr is not involved in the selection.

Evaluation protocol and Real Results

The running sequence is to test the two questions of the development set, the full evaluation of the development set, and the selection file, and finally confirm it. The freezing record time is October 14, 2026 at 09:16:38, the winner, baseline, all scores, the question number to be confirmed and the execution source code hash. Confirm that the entrance refuses the test run mode, and also refuses the change of the source code identity; the reserved news 64 questions are not used.

What we tested: on the confirmation set, the short prompt answered 89/128 questions correctly and the original baseline answered 65/128. The winner gained 28 correct answers, lost 4, and matched the baseline on the remaining 96. The paired gain was 18.75 percentage points, with a 95% percentile interval of [10.9375, 26.5625] percentage points. The set contained 64 positive and 64 negative examples, so always predicting positive would score 64/128.

The increase in the development set is 31.25 percentage points, and the improvement after confirmation is reduced but still positive. Therefore, it cannot be written as "the choice deviation makes the advantage completely disappear", and the absolute accuracy of 81.25% of the development set cannot be used to the confirmation conclusion; the confirmation that the accuracy is actually 69.53125%. Author’s interpretation: This supports the current freezing model and under the current task, short prompts are more suitable than the original template; it does not recognize whether the improvement comes from the wording, the length or the model pre-training habit.

An easy-to-miss boundary is the meaning of "unused": confirm that the sample has already been downloaded, and the previous article also uses its text to perform precise overlap of tags that do not depend on it; it has not participated in the selection of model scores before, and there is no reasoning record. This is a stay in the local research process, not a secret blind test kept by a third party. Now that it has been consumed, it cannot continue to declare that it has not seen while viewing the wrong question.

Forward Pass uses CPU, float32, EAGER attention, four threads and batch size 1; no chat template, no BOS/EOS, no answers, and no training. Maximum input 512, exceeding the stop rather than silent truncation. The ten-condition development officially includes 640 problem agreement evaluations and 1280 candidates; only two freezing schemes are confirmed, with a total of 256 evaluations and 512 candidates. The maximum difference between independent and official scores is zero.

Resources are also evidence. 131.505 seconds after the development stage, the peak RSS is about 1.91GB; the confirmation stage is 32.995 seconds, about 1.38GB. The shapes of the first topic logits are [1,36,49152] and [1,23,49152], respectively, which are batch, input length and vocabulary size. Time-consuming changes with the load, and cannot be used as a throughput benchmark. The total amount of development candidate input tokens for the original zero-shot, four examples and short prompts are 4506, 26522, and 3588, respectively; the independent recalculation is calculated again, and these numbers are not the total operation amount of the entire process. The budget is different, so this article compares the optional protocol, not the example causal effect under the equal calculation amount.

What to answer in the interval, what not to answer

antiward ii confirmation question, order did_i As the winner is correct minus the baseline is correct, then

Δ^=1n∑i=1ndi,di∈{−1,0,1}.\hat\Delta=\frac1n\sum_{i=1}^{n}d_i,\qquad d_i\in\{-1,0,1\}.

n=128n=128, the first is that the winners are added correctly, the one is the regression, and the zero is the same as the right or the wrong. The correctness of the two columns was resampled at the same time in the unit, fixed random seeds 9900, and calculated the percentage interval of the difference of 20,000 times. All resampling results are consistent with the direct shared index implementation; independent standard library scripts also use discrete distribution convolutional recalculation experience intervals. [2][5]

The interval describes the difference for the already selected, fixed protocols under row-level resampling. It does not repeat the search in each resample, so it cannot capture uncertainty in the entire research strategy. The interface supplies no film or source identifiers for grouping. Independence across rows is therefore an assumption, and we cannot rule out exposure to these reviews during pretraining. Even if the advantage holds on the confirmation set, the evidence does not extend to sentiment analysis in Chinese, other models, or other tasks.

The changes from the development to the confirmation mix the sampling difficulty, category composition and screening effect, and the two improvements cannot be named "accurate estimation of selection deviation". If you want to estimate the search strategy, you should re-complete the inner layer selection in the new outer division, instead of just drawing an extra error bar for the current winner. Repeatedly look at the confirmation score and then change the prompt, and the confirmation set will be changed into a development set again; it will require additional methods and assumptions to reuse data, and this article does not implement that mechanism. [3]

Failure investigation: first arrange the wrong identity, then explain the model

This time, the temporary environment file was cleaned up this time, and then the newly parsed version of PyArrow was not compatible with fixed NumPy. The error occurs before the model runs; after the compatible version is restored, the test run and the official run continue. The test driver also mistakenly identified the local selection script as a standard library of the same name, which has only repaired the loading method, and the source code of the freeze experiment has not been changed. Both the failure command and the exit code are reserved.

If the re-run is inconsistent, it is checked layer by layer according to the resource hash, question number set, example ID, context hash, candidate token and semantic map. The expected hash cannot be modified directly for alignment results. Eight damage tests are checked for errors, duplicate questions, missing questions, error labels and confirmation questions leaking, which can prove that deterministic checks will reject bad records, but they cannot replace real-person judgments about the semantics of natural language.

Checkable follow-up questions

There are at least two explanations for a short hint to win: it is more in line with the well-known complementary stylistics, or less context reduces irrelevant competition. This time, the expression and length are changed at the same time, and it is indistinguishable. In the follow-up, the same length can be pre-arranged and compared with the same word length on the new reserved data, and then the full conditions can be reported; the 128 questions that have been consumed cannot be repackaged into an unconfirmed set.

Another question is whether the winner is stable. After rearranged the development set or changed the random division, the winner may be different. You can do exploratory stability analysis on the existing development data first, but it cannot restore the confirmation set that has been seen. If a resource is only allowed to be confirmed once, it is recognized that there is only one independent confirmation result, and the boundary of failure is retained, which is more valuable for research than after the fact that the conditions are added after the fact.

Code and instructions for reproducing the results

The full public source code has been fixed to submit 4d027f931e0f: This catalog、README and full running steps、SOURCE_MAP official implementation mapping. The 51 public documents have been anonymously downloaded and checked one by one with the local source code and archives, and the audit and real model test runs of the public copy have been passed.

The minimum offline command is python audit.py, which outputs the independent audit of candidate summation, label mapping, selection result and confirmation difference. The real model entry is python run.py --cache ./cache --phase development --smoke --out smoke-new, first press README to install lock dependencies and run prepare.py. It is expected to keep question-by-topic records, stage status, environment and identity files. The package includes complete self-write module, downloader, configuration, license and source map; does not distribute model weights, original comments or implied native paths.

Summary

The search itself is not a mistake, and omitting the search process makes the evidence difficult to judge. This article separates the ten attempts, a frozen winner and a comparison record. The verifiable conclusions come from the confirmed data and their limitations; the highest score on the development set is responsible for proposing candidates, and should not be borne by the final proof alone.

References

Retrieve and actual access date: 2026-10-14.

  1. Gavin C. Cawley、Nicola L. C. Talbot,2010: On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation,JMLR 11,2079–2107; Full text.
  2. Rotem Dror、Gili Baumer、Segev Shlomov、Roi Reichart,2018: The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing,ACL.
  3. Cynthia Dwork et al., 2015: Generalization in Adaptive Data Analysis and Holdout Reuse. This article only cites the background of the problem, and does not run its algorithm.
  4. EleutherAI: Fixed commit SST-2 task configuration.
  5. SciPy: Bootstrap documentation version 1.13.1.
  6. HuggingFaceTB: Fixed version SmolLM2-135M-Instruct model card.
  7. NYU/GLUE: Fixed version data card; Richard Socher et al., 2013: Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank,EMNLP.