
- Date: 2026-10-08
- For graduate students, early-career researchers and research engineers
- Scope: A small model, 64 development samples, two sets of four examples and forward and reverse order; not a complete reference or no test set .
Summary: Giving an example does not mean teaching right
Give the model four examples with answers, and then let it judge the new sentence, which should be more reliable than asking questions directly. But the example not only provides task information, but also changes the input length, label position and context distribution. "Added four examples" is not a sufficiently complete experimental condition.
The actual verification of this time: Fixed 64 development samples of SmolLM2-135M-Instruct and SST-2, with 32 correct answers for zero examples. The two orders of the other group were answered 33 and 44 respectively. We did not find a generally better order, but found that: the same number of examples and the same length, the content and arrangement will still change the measurement.
Reproducing value: from total score to pairing observation
Few-shot refers to context learning in this article: put a small amount of input and answers into the prompt, and the model uses them in at inference time; there is no gradient update, and it is not to use four data fine-tuning . Zero examples also do not mean that the model has never been trained with instructions.
Paper report: Lu et al. in the ACL 2022 study sample arrangement sensitivity, pointing out that a good arrangement on one model may not be migrated to another model. [1] Zhao et al. discuss the answer bias and context calibration of the prompt in ICML 2021. [2] These work support check sequence and answer prior, does not mean that their improvement magnitude will be reproduced in this article.
In recent work, Cobbina and Zhou's EMNLP 2025 paper distinguish "how to arrange the internal examples" and "whether the whole group of examples are placed in the system prompts, where the user messages are located", and at the same time measure the accuracy and prediction changes. [3] This article only deals with the former, and the examples are always before the problem; the two types of variables are not mixed as a position effect, and the complete evaluation of the paper is not reproduced.
Compared with the previous article, we keep the model, topic, candidates and scoring methods, and only add example sets and arrangements. The new product is not the best one-line score, but the corresponding record of the same question under five conditions.
Core idea: the same question is the unit of comparison
Make for the sample collection, in order, For first road problem. The conjunction after the splicing is denoted as , the candidate scores are:
is the candidate position, is the number of candidate tokens, is the frozen weight. Select candidates with a large number of likelihoods, and do not do length normalization; take the first one when they are tied. The candidates are still Negative and Positive, which are not changed to letters, and they are not generated freely, and do not add answer extractors.
pair condition and , first calculate the difference in correctness of each question, and then calculate the average:
zero or one, this time . Five conditions produce 320 predictions, but not 320 independent problems. The pairing record can explain where the multiple answers come from, and repeat measurements cannot be regarded as sample size growth.
Predict the flip rate to answer another question:
is the prediction category. Accuracy is unchanged and there may be many questions that have changed the answer, but the gains and losses are offset. Therefore, at the same time, the number of wrong turns, pairs and forecasts is reported at the same time, and the total score cannot be only looked at.

Each path evaluates a separate condition. The two sets of demonstrations never appear together in a single prompt. The grid on the right shows the pairing, with questions in rows and the five conditions in columns. It contains no measured values.
A guide to the official implementation
Read the SST-2 configuration of the pinned commit first, and confirm the template, candidate sequence, label mapping and ACC indicators. [4] Then read ContextSampler: SAMPLE is responsible for the selection example, and get_context is responsible for splicing the problem, answers and separators. [5] "Random Seed Fixing" cannot replace the saving specific example numbers, especially when the traversal sequence or the number of calls have changed.
The local FixedSampler only overrides the sample method, returns the frozen training row, and the rest of the splicing calls are officially implemented. At the same time, the complete prompt is independently constructed, and compared with FEWHOT_CONTEXT character by character. CUSTOM_DATASET reads fixed Parquet, which changes the offline transmission, not the task content.
Finally, catch up the candidate likelihood paths of simple_evaluate and HFLM. [6] In addition to the official return, use Transformers to separate forward pass to each candidate token. The input shape is , the output is ; the context has When a token, the first candidate token takes the position distribution. The two implementations cover the same batch of 640 candidates, not just the five mean values.
Minimum experiment: freeze first, then look at the results
The model revision is 12fd25f77366fa6b3b4b768ec3050bf629380bac. The data come from a pinned copy of GLUE’s SST-2 at revision bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c. [7][8] The evaluator is pinned to 0.4.9.2, with the commit recorded in the source map. No claim is made that this is the latest version.
The development questions follow the 64 items selected in the established hash order. Another 128 SST-2 confirmation samples will not be sent to the model; the previously reserved news confirmation samples are also not used. The development set has reappeared, so this is an exploratory protocol experiment, which cannot be repackaged into final verification on unseen data.
Examples are only from official train. Exclude the exact same text after either sentence normalization of Validation, and then exclude duplication between eight examples; screening only uses text, not validation tags. Sort by fixed hash, each label takes four training records that meet the length condition; a and b each use two positive and two negatives, the positive order is arranged in negative, positive, negative and positive, and the reverse order is completely inverted.
Each sample block with answer and separator is 42 tokens alone, and the prefix of the four blocks is actually 171 tokens, which cannot be simply multiplied by four. The complete input is tokenization again: the four FEW-SHOT conditions have the same context length for each question, 191 to 233; the zero example is 19 to 61, all below 512, without truncation.
These are deliberately selected short, balanced, and isometric examples, which do not represent the entire training distribution. While controlling costs are mixed, scope limitations are also introduced. The sample identity and all conditions are written to plan.json before reasoning, and the sample is not replaced according to the score, and no permutation search is appended.
CPU single precision, four threads, EAGER attention, candidate batch one, no chat template, no BOS or EOS, no open request or response cache. Four FEW-SHOT conditions each handle 26522 candidate input tokens; zero example is only 4506. Therefore, the order in the group is equally long, but the FEW-SHOT and the zero example are not equivalent calculations.
What we tested: Negative results and improvement are reported together
The timed run took 102.77 seconds after third-party imports completed. Peak resident memory was 1994588160 bytes. GPU memory was not measured. Across five conditions, the run produced 320 predictions. The official and independent implementations agreed exactly on all 640 candidate scores, with a maximum absolute difference of zero. Question IDs, gold labels, prompt hashes, and candidate boundaries also matched.
| Condition | Number correct | accuracy | Predicted as positive |
|---|---|---|---|
| zero example | 32/64 | 50.00% | 3 |
| A positive sequence | 31/64 | 48.44% | 0 |
| A deorder | 37/64 | 57.81% | 6 |
| B sequence | 33/64 | 51.56% | 2 |
| B deorder | 44/64 | 68.75% | 13 |
A is less than one question in the positive order than zero, and there are actually three questions to turn over: one is wrongly turned, and the two questions are turned from right to wrong. This net difference cannot prove that FEW-SHOT is generally harmful, but it shows that the preset that "there are more examples, so this time it is definitely better" do not hold.
There are six questions in the positive order to the reverse order, which are correct and wrong, and there is no right and wrong, which is increased by 9.375 percentage points; in the corresponding eleven questions of B, the wrong turn is paired, which is increased by 17.1875 percentage points. The denominator is always the same 64 questions, and the two changes cannot be added as a new method.
The example selection also affects the results: under the same positive order position, B is more than a pair of two questions; at the same reverse order position, there are seven more questions. This suggests that the set effect is related to the arrangement, but the two sets and two arrangements are far from enough to estimate a stable overall interaction effect. We reserve all the conditions and do not replace the full report with the highest score.
Watch the change along a record
Take the number 400 as an example, and the gold label is the front. At zero example, the log likelihood of Negative and positive is about minus 3.874 and minus 4.765, the model is negative; when B is reversed, the two are about negative 0.763 and negative 0.651, and the positive candidate achieves a higher score. This question has not been corrected in the two orders of A and the positive order of B, and the changes cannot be explained only with "example".
When reading the record, you should first check the question number and gold label, then check the candidate sequence, and finally observe the score. The absolute log likelihood change of a single answer under different prompts is not equal to the change of Accuracy, and it cannot be directly called Confidence Calibration improvement. This article only takes the maximum value between the two candidates of the same question and the same condition, and then counts the prediction changes.
This case is an example of the calculation chain after the result is generated, and is not the main indicator specified in advance. You can't ignore the remaining sixty-three questions just because it just supports a certain story. The complete question-by-question file keeps records of success, failure and no change, and readers can re-aggregate without trusting a few examples selected by the author.
Failure investigation: control the length, but also control the explanation
In the initial test, the initialization time of the task is also enlarged according to the number of samples, triggering a 600-second budget check. After separating the fixed overhead, the conservative prediction is still 816.96 seconds, so the upper limit of the local CPU is clearly changed to 1200 seconds before the official operation, and the memory is still 4GiB. The final test run predicts 944.89 seconds and passes, and the official run is actually about 103 seconds. The two failures, logs and revision reasons were reserved, and there was no condition for poor performance to be used to adjust and replace poor performance.
There are four types of silent errors that need to be prevented: the example leaks from the evaluation question, the label is missed when desequential, the long prompt is automatically truncated, and the result is not aligned according to the real number. The independent audit re-reads the data and tokenizer, verifies the identity, and then recalculates the candidates and pairing statistics. Eight tests include intentional deletion of records, duplicate question numbers, tampering predictions or pairing counts, and confirmation errors will be intercepted.
The arithmetic check is completed by an independent script, and no real person is required to press the calculator by question, and it does not claim that someone has added a semantic gold label. The license mark of the data card is not clear, and the public package only keeps the index, hash and derived scores; does not upload complete movie reviews, prompts, question stem tokens, weights and caches. The fixed version download script is responsible for obtaining resources.
Testable: Don't just look for the best order
**My interpretation:** Changing the order clearly affected the model’s bias toward a class, but the comparison cannot isolate a “last label” mechanism. In the forward ordering of A, the final demonstration was positive, yet the model predicted negative every time. Reversing the order changes the first, last, and intermediate positions together, leaving open which position mattered.
The more verifiable design is to only exchange the middle of the head and tail, or the fixed content only changes the last case, and divides the pre-determined strategy in the independent development. Calibration should be used as a separate variable, leaving a confirmation set that is not selected. Don't try to get the highest score on these 64 questions, and then announce that the sequence problem has been solved.
Screening for exact duplicates after normalization rules out only that particular overlap. Passages may still share a source, be close in meaning, or have appeared during pretraining. An expanded experiment should record how the demonstrations and test questions are related by source, then control relevance, length, and label position separately. If all three change together, even a higher score tells us little about which change helped.
In conclusion, FEW-SHOT is not just a switch with the number of examples. Record who is selected, where it is, how many tokens are used, and how to change the same question, in order to distinguish task information, label bias and evaluation accident. This article is a reviewable local experiment, not a new Prompt formula.
Code and instructions for reproducing the results
Full source code, dependencies, freezing plans and question-by-question records have been made public: Source tree at the pinned revision、README Running Instructions、SOURCE_MAP source map. The 51 files downloaded anonymously are consistent with the local archives one by one; the independent audit of the open copy, the eight tests and the real model test run are all passed.
After installing the dependencies in README, obtaining and verifying the resources with prepare.py, the minimum real entry is:
HF_HUB_OFFLINE=1 TOKENIZERS_PARALLELISM=false python run.py --cache ./cache --out smoke-new --private private-new --smoke
This order executes all five conditions for two development questions, and checks the official and independent scores. Remove --smoke to run the full 64 questions; then execute python audit.py --cache ./cache --out results to check and save the results. The output directory must be given a new name to avoid overwriting evidence. At present, the five conditions have been officially run, and there is no training model, no confirmation set is used, and no number of papers is used as the original result.
References
Search date: 2026-10-08. Recent work and classic papers are selected around this issue, and do not claim to cover all the latest papers.
- Yao Lu et al., ACL 2022: Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity.
- Zihao Zhao et al., ICML 2021: Calibrate Before Use: Improving Few-shot Performance of Language Models.
- Kwesi Adu Cobbina、Tianyi Zhou,EMNLP 2025: Where to show Demos in Your Prompt: A Positional Bias of In-Context Learning.
- EleutherAI: Fixed SST-2 task configuration.
- EleutherAI: Fixed ContextSampler、fewshot_context.
- EleutherAI: simple_evaluate、Hugging Face backend.
- HuggingFaceTB: Fixed SmolLM2-135M-Instruct model card.
- NYU: Fixed GLUE data card.