Self-Consistency Sampling and Voting Experiment Cover

Abstract

Self-Consistency (Self-Consistency) does not train new models, but in AT Inference time Let the same language model sample multiple reasoning paths for the same question, extract the final answer of each path, and then choose the consensus answer with most voting. It transforms an unstable generation into an auditable "sampling-normalized-aggregation" protocol, but also magnifies the inference cost to several times the sample. This article follows Wang et al.'s ICLR 2023 paper, Chain-of-Thought original paper, GSM8K official data code and Google DeepMind follow-up Oneetwo Implementation, dismantling formulas, code contracts and evaluation methods; the attached-free scripts use controlled simulations to verify when voting is valid and when systemic errors will be magnified. It is not a PaLM, GPT-3 or GSM8K score reappearance.

Reproduce the value and goal: turn "ask several times" into an experimental protocol

Chain-of-Thought (CoT, thinking chain) prompts that the model is required to form an intermediate inference and then give the answer. Ordinary CoT is often used greedy decoding, and only retains one path with the highest local probability; Self-Consistency actively introduces randomness, expecting that the correct answer can be reached from a variety of different paths, and the wrong path is more scattered. It's not a line that is really worth repeating Counter(...).most_common(1), but its four contracts before: whether the same question uses the same prompt; whether the samples are independent of each other; how the answer is parsed and normalized; the cost is compared with the accuracy on the same budget.

This article answers four checkable questions: what probability is the marginalized probability of most voting; when the number of samples is increased from 1 to 33, the benefit is the same as token How the cost changes; how the answer format, flat vote and parsing failure affect the result; why "more consistent" does not mean "more correct" when multiple paths share the same misunderstanding.

Core ideas: marginalization of reasoning paths

Given hints (p) and question (x), the (i) time sample generates the inference path (r_i) and answer (a_i):

(ri,ai)∼Pθ(r,a∣p,x),i=1,…,m(r_i,a_i)\sim P_\theta(r,a\mid p,x),\qquad i=1,\ldots,m

where (m) is the number of samples, (r_i) is the token sequence of variable length, (a_i\in\mathcal a) is the discrete answer mapped by the task parser. If a BATCH has (B) questions, each question is sampled (M) times, and logically (B\times m) completions are obtained; the length of each path is different, and the actual token cost should be summed.

Most voting without the right to use the paper is:

a^=arg⁡max⁡a∈A∑i=1m1[ai=a]\hat a=\arg\max_{a\in\mathcal A}\sum_{i=1}^{m}\mathbf 1[a_i=a]

Indicates that the function takes 1 when the answer to the path (i) is equal to (A). Here, the "Edgeized Reasoning Path" does not check whether each step is correct, but discards the final value after (r_i). The paper also compares the polymerization weighted by sequence probability; its table 1 shows that the direct voting directly has no right to vote and the length normalized weighting result is similar, and the unnormalized probability is easy to favor short output. This conclusion belongs to the report of the specific model and task of the paper and should not be the law of all models.

If the answer is only right/wrong, each path is independent and a single accuracy is (q>0.5), the probability of most correct under the odd (m) is:

Pm=∑k=(m+1)/2m(mk)qk(1−q)m−kP_m=\sum_{k=(m+1)/2}^{m}{m\choose k}q^k(1-q)^{m-k}

Formulas are independent, the same distribution and (q>0.5) are the key assumptions. The path of the real model will share the hints, parameters and knowledge gap, and the error is highly related; if a wrong answer is the most common, increasing (M) will only choose the wrong more stably. If multi-category errors are scattered with each other, even if a single Accuracy is slightly higher than each error category and does not exceed 0.5, most modes may still benefit.

Self-Consistency method, cost and systemic error boundaries

Paper and guide to the official implementation

Read Section 2 of the Self-Consistency paper, following the mapping from (r_i\rightarrow a_i), the task-specific answer parser and the majority-vote formula. Section 3.1 gives the sampling settings: temperature 0.5 and top-k 40 for UL2/LaMDA, temperature 0.7 and top-k 40 for PaLM, and temperature 0.7 without top-k for the GPT-3 model used at the time. Each main result used 40 independent samples, and the full experiment was repeated 10 times. When working with a new model, record the temperature, top-p/top-k, maximum tokens, stop strings, model version and random seed in the experiment configuration.

The original paper has not been officially implemented in the repository, and some paper models cannot be publicly available; therefore, it cannot be claimed that "the official code of cloning can be reproduced". Data-side readable GSM8K official repository: each row has question and answer, the final number is located in #### after; dataset.py Extract and remove commas with regulars. This detail is directly determined 1,200、1200 Whether to enter the same voting bucket also reminds us to write the results of parser version and parser failure rate.

For project implementation, please refer to the v0.2.0 description of Google DeepMind's subsequent open source OneTwo: it treats self-consistency as a single parcel policy meta-stategy, estimating the distribution of answers after repeated sampling, and explicitly supports answer normalization, weighted voting, consensus answer and complete distribution evaluation. It is not the original paper matching code, but provides a clearer module boundary: sample() only responsible for generation, parse() Only responsible for the standardization of answers, aggregate() only responsible for voting, evaluate() Record correctness and cost at the same time.

Trace a real implementation through five checks. First, verify that generation returns (m) independent sequences. Copies of one sequence, neighboring beam-search outputs, and server cache hits do not qualify. Save the request ID and raw text for every response. Second, run the answer parser after truncation at the stop string and check that it does not pick up the first number in an intermediate reasoning step. Third, print the normalized answers and counts before aggregation and test negatives, decimals, thousands separators, units, and missing answers. Fourth, specify how ties are broken. Random choice, fixed lexicographic order, and the first sample can give different results on a few items, so use the same rule at every budget. Fifth, retain the entire sample pool. It lets you examine correlations among errors and recompute results under new normalization rules while guarding against leakage from tuning the parser after seeing the results.

Minimum experiment: verify the aggregation and failure boundaries without a large model

run:

python3 code/minimal_self_consistency.py --check-only

Scripts fixed random seeds 59, and do not generate natural language reasoning. It first validates the four answer formats normalization, and then repeats 20 rounds of 2000 simulated questions, and compares (m\in{1,3,5,9,17,33}). diverse_errors The single distribution of the situation is the correct answer 0.54, and three different wrong answers 0.16/0.15/0.15; systematic_error The situation is the correct answer 0.42, the same main error 0.52, the rest 0.04/0.02. The flat vote is broken by the earliest answer, which is written to death in the code to avoid quietly changing the result of the default behavior of the evaluation library.

In this CPU Smoke Test, the decentralized error accuracy has been increased from 0.5435 (m=1) to 0.9520 of (m=17); systemic error is from 0.4219 down to 0.3356. The former also increased the budget estimated at the upper limit of 128 token from 128 to 2176, and the latter proved that high consensus may only be a stable deviation. Binary independent theoretical examination obtained 0.8789 at (q=0.60, m=33). The numbers only validate the voting mechanism and hypothetical boundary and do not represent any language model, dataset or real API cost.

Also be careful to explain the vote_share: With a single sample, the winning vote share is always 1, regardless of the model’s confidence. As samples accumulate, that share approaches the mass of the most frequent answer in the empirical distribution. When errors are dispersed, the correct answer’s share settles near 0.54, ahead of each wrong answer at about 0.15, so voting accuracy approaches 1. The individual paths have gained no ability. More samples simply identify the mode more reliably. With systematic errors, one wrong answer has a share of about 0.52, and more samples confirm that wrong mode. Alongside accuracy versus sample count, plot at least the winning vote share, the number of distinct answers, and the concentration of wrong answers. Together they help distinguish averaging away random noise from amplifying bias.

Minimum path to upgrade to real model

Fix a publicly available model with precise revisions, freeze the fEW-Shot Exemplars, and first save each original completion on the small GSM8K subset. When generating 1, 3, 5, 10, 20, and 40 paths for each question, it is best to generate the maximum pool at one time, and then calculate the different budgets in the subset of the previous, to ensure that the same batch of samples is used for horizontal comparison. The log contains at least qid, sample ID, decoding parameters, original text, normalized answers, parsing status, input/output token, delay and model version.

If the service supports parallel requests, the parallel only reduces the wall clock time, and does not reduce the total token or cost. The cache must use the prompts, model version and decoding parameters as keys; otherwise, the temperature is changed but the old samples are reused, which will create a false reproducibility. The use of 40 papers is not a general advantage, and the frontier of Pareto should be drawn on this model, this data and this budget.

Evaluation protocol: what must be reported outside of Accuracy

First report the Exact Match, and separate the single-path random sampling, single-path greed and self-consistency; the random sample (m=1) is not equal to the greedy baseline of Temperature=0. Secondly, report the mean, standard deviation, or Confidence interval of each (M), and use the same set of problems to make pairs for comparison. Report the total complexion, input/output token, actual cost, p50/p95 delay and resolution failure rate again. The cost formula can be written as:

C(m)=∑i=1m(cintin(i)+couttout(i))C(m)=\sum_{i=1}^{m}(c_{\text{in}}t^{(i)}_{\text{in}}+c_{\text{out}}t^{(i)}_{\text{out}})

where (t_{\text{in/out}}) is the input/output token of the (i) request, (c_{\text{in/out}}) It is corresponding to the unit price; the price and model interface change with time, this article does not write dead dynamic numbers, and waits for manual verification before actual operation.

The final report of Winning Vote Share, the number of unique answers, and the number of buckets in Accuracy by consensus interval. Consensus can be used as a signal of uncertainty, but not directly as a calibration probability; should check whether the "80% consensus" question is really about 80% correct, and evaluate the coverage rate together with the abstention threshold - Accuracy .

The cost curve should also not only use "the number of calls multiplied by the unit price". The output token of long inference questions may be much larger than short questions, and the fixed maximum length will increase the truncation rate with the difficulty. It is recommended to calculate the incremental cost from (m=1) to the target budget and correct the answer for each question, and then summarize "Accuracy gain brought by every additional token". If 20th to 40th The path hardly changes the winning bucket, and the curve has entered the diminishing return area; if it is still flipped frequently, it means that the model is unstable in the family of the problem, and should be compared with adaptive stop, external validator or tool execution, instead of continuing heap sampling by default. The throughput of parallel batch processing is presented separately from the Token cost, as it only changes the wait time and does not reduce the total amount of calculation.

Failure analysis: where is the easiest place to lie

The first type is parsing errors. Different equivalence forms will dilute the correct answers; over-normalization may incorrectly merge answers with different semantics of units, symbols or options. Failure of parsing cannot be discarded silently, otherwise the number of valid samples will change with the difficulty of the question.

The second category is related errors. The same model shares erroneous knowledge, ambiguity understanding and prompt induction, and dozens of paths may just be superficial rewrites of the same misunderstanding. The questions of "high consensus but wrong" should be sampled and audited, and the semantic diversity of inference paths should be compared, not only statistical differences.

The third category is data leakage and non-independent requests. Multiplexing random seeds, deterministic caching of the server, or degradation in a request will make the samples appear to be many, but the effective sample size is small. The fourth category is budget unfairness: use 40 paths to compare once greed but only report Accuracy, and can't answer whether it is worthwhile to add 39 times the reasoning.

The fifth category is open answers. The traditional majority of voting requires stable and normalized discrete answers; abstracts, long texts and multiple equivalents prove that there are no natural barrels. Universal Self-Consistency uses the model to select the most consistent answers between candidates, but introduces a new review model deviation and additional cost, which cannot be confused with string voting.

Questions for further research

The most direct question is the adaptive budget: can the simple question stop early when a small number of samples have formed a stable advantage, and leave the budget to the difficult problem? Adaptive-Consistency and Early-Stopping Self-Consistency have reported significant savings, but still need to redo stop errors and worst-cost analysis on new models and tasks. Second, how to estimate path correlation and effective sample size, rather than default independent? Third, should voting weights come from sequence probability, self-assessment Confidence, external verifier or execution result? Fourth, can we make causal classification of high consensus errors, distinguish between lack of knowledge, misunderstanding of problems, and defects in arithmetic errors and parser? Fifth, under a fixed token budget, how should the number of increased paths and the extension of single-line reasoning should be allocated? These questions are more research value than blindly adding (m) from 40 to 100.

Summary

Self-Consistency is a simple but assuming method of calculation when testing: keep the problem with the prompts fixed, sample multiple paths, normalize the final value and vote. It works when the correct answer is more stable than any wrong pattern, and may backlash when a systemic error is dominant. Reliable reproduction must treat sampling parameters, parsers, tickets, repeated experiments and token costs as part of the protocol. First use the script of this article to verify the aggregation, and then connect to the real model and fixed data subset, which can avoid reproducing "care several times" into a comparable paper.

References

Search date: 2026-08-31.