LLM-as-a-judge Deviation Audit Experiment Cover

Abstract

A language model can evaluate open-ended responses by reading the task, candidate answers, and natural-language criteria, then returning scores, preferences, and reasons. Even a capable judge may change its verdict with the candidate order, answer length, model family, prompt wording, or rating scale. Drawing on the papers and official code for MT-Bench/FastChat, G-Eval, FairEval, Prometheus, and LLMBar, we define an auditable protocol. Write a task-specific rubric, hide model identities, and judge each answer pair in both A/B and B/A order. Map the verdicts back to answer identities, withhold an automatic decision when the verdicts conflict, and evaluate the judge against human labels. The accompanying script tests these steps in a deterministic simulation using synthetic features. It reports no LLM, MT-Bench, or public leaderboard results.

Reproducing value and goals: It is not the answer that evaluates, but the evaluator

Traditional Accuracy requires a determination of Gold Answer; abstracts, dialogues, code explanations, and research questions and answers often have a variety of acceptable expressions. The model referees turn the difficult programmatic "related, correct, clear, and follow the instructions" into a generation task, and the cost is usually lower than the manual review of one by one, and it can also give reasons. However, what researchers really need to answer is the second-order question: whether the same content is still obtained after changing positions; whether the short and correct answer will lose to the long answer; Rubric Whether the abstract words are implemented into a judicious boundary; the consistency between the referee and the person is the ability to stabilize or the data set, prompts and model versions are coupled.

This minimum reproduction does not call the online model, nor does it repeat the dynamic leaderboard. There are four goals: the data structure behind the pointwise (score item) and pairwise (pairwise comparison); the consistency of quantization and sequence; Rubric; show how the automatic rejection and heterogeneous review panels change the coverage with Accuracy . In this way, the evaluation protocol is first verified, and then replaced with the real Judge API, which can turn "look reasonable score" into a diagnostic experiment.

Core idea: Judge is a conditional measurement function

For task (x), candidate answer (y), optional reference answer (z), Rubric (r) and prompt template (P), the score can be written as:

s=Jθ(x,y,z,r,p;ξ)s=J_\theta(x,y,z,r,p;\xi)

(j_\theta) is a fixed version of the referee model, ξ\xi Including sampling randomness and server-side changes. If a batch has BB samples, Rubric have KK dimension, first get the scoring matrix V∈RB×KV\in\mathbb R^{B\times K}, then use the weights w∈RKw\in\mathbb R^K Polymerization s=Vws=Vw. Each dimension must state what 1–5 points each means; only write "helpfulness" without anchor points, and no reproducible measurement is defined.

Pair of review output d=Jθ(x,yA,yB,r,p)∈{A,B,T}d=J_\theta(x,y_A,y_B,r,p)\in\{A,B,T\}, in TT It's a draw. Since inputs are ordered sequences, theoretically exchanging answers and re-mapping the response status should maintain a conclusion. Define:

Cswap=1N∑i=1N1 ⁣[diAB=map⁡(diBA)]C_{swap}=\frac1N\sum_{i=1}^{N}\mathbf 1\!\left[d_i^{AB}=\operatorname{map}(d_i^{BA})\right]

(n) is the number of answer pairs, diABd_i^{AB} and diBAd_i^{BA} from two sequences, map⁡\operatorname{map} Turn the slot label back to the identity of the answer. CswapC_{swap} The lower the position disturbance, the better the results can be turned over. The robust protocol can only be automatically accepted when the judgment is consistent twice; at this time, both Accepted Accuracy and Coverage should be reported at the same time, and the rejection sample cannot be quietly deleted and only reported higher.

Rubric, anonymous order change, deviation check and manual upgrade process

Paper and guide to the official implementation

First read the MT-Bench/Chatbot Arena paper by Zheng et al. In the open multi-round dialogue, the paper studies strong models as referees, and reports that the agreement rate of GPT-4 referees and human preferences is more than 80%, and at the same time specializes in the analysis of the boundaries of location, length, self-enhancing and reasoning ability. This is a report under the specific data, model and 2023 settings, and is not a property that is automatically owned by any new model.

Then follow the official FastChat repository to read the code. data/judge_prompts.jsonl of pair-v2 First ask for a reason for comparison, and then [[A]]、[[B]] or [[C]] Output rulings, and explicitly remind not to be affected by order and length; text reminders are not deviant evidence themselves. gen_judgment.py Disassemble general questions, math questions that need to be referenced, single-round and multi-wheel into different Judge templates. The key is common.py: play_a_match_pair() Run the positive and reverse order twice for the same answer, map back model_1/model_2, and write the original prompt, judgment and winner twice to JSONL. The real reproduction should also be locked for repository commit, referee model snapshot and API parameters, instead of just recording "GPT-4".

G-Eval is generated for summary and dialogue, and form a form-filling prompt to form a task introduction, evaluation criteria and automatically generated Evaluation Steps, and try to use the rating token The probability-weighted discrete scores; the paper reports that its specific abstract experiments are more related to human scores, and it is also clear that the referee may prefer LLM to generate text. Prometheus puts Reference Answer and fine-grained Score Rubric into the input of the open Evaluator Model, emphasizing the controlled version and custom standards. They indicate that "give a total score" is not the only paradigm: tasks, evidence, dimensions, and output formats must be explicit.

Finally, look at the meta review. FairEval directly displays a large number of judgments that can be turned over by only exchanging candidate positions, and proposes multi-evidence, balanced position and manual intervention calibration. LLMBar provides natural and confronting answer pairs with Gold Preference, which tests whether the referee can recognize the output of the actual instructions, rather than being confused by the surface quality. Judge's evaluation set should be independent of the development set of the model under test; otherwise, the referee prompt will be adjusted to "more accurate", or it may only be overfitting for the meta-evaluation set.

Minimum Experiment: Switching is not data enhancement, but measurement audit

run:

python3 code/minimal_llm_judge.py --check-only

The script uses only the Python standard library, fixed torrent 61, and generates 120 answer pairs. Each composite answer has four 0–4 dimensions and token numbers for correctness, instruction follow, evidence, and clarity; gold preference It is determined by Rubric, which is pre-disclosed and has the highest correctness weight, and deletes the approximate draw. The Sim Referee adds position offset, length offset, and noise fixed by the SHA-256 key, so each run is exactly the same.

In this CPU SMOKE TEST, the accuracy of the "overall quality" referee in the AB/BA order is 0.700/0.742, and the consistency of the replacement sequence is only 0.642; only after accepting two-way identical samples, the coverage rate is 0.642 and Accuracy is 0.844. After using explicit Rubric, the two sequences of Accuracy are raised to 0.850/0.867, the bidirectional consistency is 0.717, and the subset Accuracy 1.000 is accepted. Three synthetic referees with different bias directions and noises make the majority vote, the overall accuracy is 1.000, and all the referees have a stable ratio of 0.967.

These figures only prove the nature of the protocol in the script construction: Rubric is closer to the rule of generating GOLD, and the two-way consent is actively rejected, so the result will inevitably benefit. The real research conclusions must be replaced with frozen model output, independent manual labeling and pre-registry analysis.

Bias map: first distinguish the source, then the design comparison can be designed

Position deviation means that the content remains unchanged, and the preference is changed only when the front and rear slots are changed; the minimum cost check is a bidirectional sequence and report a conflict matrix. Long deviation means that additional details, repetitions and rhetoric are mistaken for completeness; a "short and correct" comparison of "long but wrong" should be constructed, and simplicity and fact should be scored separately. The self-enhancing bias is the output of the referee's preference for the homologous model; it is necessary to hide the model name and allow the referees of different families to cross-review. Familiarity and style deviations will reward common phrasing, fluent English or specific templates, and cross-language, rewrite and format disturbances are necessary stress tests.

Three other failures are often grouped under “bias.” First, a judge may lack the mathematical, code-execution, or domain expertise a task requires. When unit tests, reference answers, or retrieved evidence can settle the question, use them. Second, judgments may be sensitive to the prompt. The order of rubric dimensions, whether the judge explains its choice first, and the form of its output tokens can all change the distribution. Third, contamination or leakage may expose the judge to the benchmark, reference answers, or candidate sources. All three require logs and controls, but each calls for a different causal explanation.

Rubric design: write value judgment as falsic rule

One can use Rubric to write the task target first, and then list 3–5 orthogonal dimensions. Facts are available with correctness, evidence coverage, instruction compliance, and clarity; abstracts also need to be covered by loyalty and key information. Each dimension provides fractional anchors and counterexamples, such as "5 points: all key assertions can be supported by given evidence; 3 points: the conclusion is basically correct but a key constraint is omitted; 1 point: core facts and evidence conflict". Weights should be determined before looking at the results and state that Fatal Error: Whether the key factual error should override the advantage of the style of writing.

Pairwise is easier to answer "Which one is better", but only gives relative information; PointWise is easy to summarize across batches, but it is easy to drift the scale. The safe practice is: first use the PointWise dimension to generate a checkable reason, and then make the Pairwise president; give priority to the programmatic dimension of the rule executor; reserve a draw and a rejection for the subjective dimension. Don't ask the referee to expose long-form implicit reasoning, and only keep short evidence excerpts, dimension scores and structured error codes, which is not only convenient for auditing, but also reduces the seemingly reasonable post-event defense.

Evaluation protocol: report validity, reliability and cost simultaneously

The dataset is layered according to the task, difficulty, language, length difference and whether external knowledge is required, the answer is blinded, and the order is random; at least run AB/BA for each pair. Reliability report conversion consistency, repeated sampling consistency, adjudication consistency, and Cohen's κ=(po−pe)/(1−pe)\kappa=(p_o-p_e)/(1-p_e): pop_o is the observation consistent rate, pep_e It is an accidentally consistent expectation according to the marginal distribution of respective labels. High consistency does not mean correct, and two referees can stably make the same mistake, so it is also necessary to report Accuracy, Confusion Matrix and layered errors on the independent artificial GOLD.

Validity checks whether the referee has really measured the target structure: add irrelevant descriptions, change positions, rewrite styles, and insert the name of the model to the answer, and the correct conclusion should remain unchanged; delete key evidence or create factual contradictions, and the score should be reduced. Automatically accept, draw, parse failure, switch conflicts, and manual upgrades are counted respectively; record Prompt Hash, Rubric version, JUDGE version, temperature, original input and output, token, delay and cost. The dynamic API and the list will change, and the current price and ranking are manually verified.

How to diagnose failures

If the two sequences conflict frequently, first confirm that the mapping code does not use the slot A/B as a model identity, and then check the difference in answer length, template position and conclusion token preferences; do not use random again to cover the conflict. If all the scores are crowded in the high partition, check whether the anchor point is missing a low score sample, and prompt whether it is "friendly" by mistake, and check the margin of each dimension. If it is inconsistent with the labor, first review the manual Rubric and the labeler agreement, and then locate the knowledge or reasoning type that the referee does not know according to the task slicing.

If you change the Judge ranking to reverse, you should not choose the "most in line with expectations". Heterogeneous Panels can be used, but each referee result and aggregation rules must be disclosed; multiple family models are not equal to independent evidence. If the reason is correct and the final tag is wrong, check the parser, label mapping and generate truncation. If the reason is fluent but the fact that the candidate does not exist is referenced, the Rationale Fidelity should be sampled separately, and the high-risk samples should be sent to labor.

Questions for further research

The first is how to automatically generate the minimum adversarial disturbance from Rubric to verify the monotonicity of each dimension; the other is the sequence conflict, the judgment disagreement and the language model Confidence, which one is the most predictable and difficult example of labor; the third is how to estimate The family dependence of JUDGE and the model under test without leaking anonymity; the fourth is how the rules, retrieval, executor and LLM referees are combined according to the ability boundary; the fifth is the Panel The diversity should be selected according to the model family, training data or false correlation; sixth is how to detect the drift of the referee version in the long-term Benchmark. An executable topic is to freeze the same batch of answers, conduct a full factor experiment on the Rubric, sequence, language and Judge family, and disclose the original ruling and manual review sets.

Summary

The value of LLM-as-a-judge is not to replace all labor, but to extend open evaluation into repeatable candidate measurements. Reliability comes from task-specific Rubric, identity blinding, two-way sequence, structured output, independent artificial meta-evaluation, conflict rejection, and complete log, rather than a "please stay objective". Use the script of this article to reproduce the position and length deviation first, and then connect the real referee and freeze the version, to know whether the score change comes from the measured model or whether it is from the ruler itself.

References

Search date: 2026-09-02. The following are papers, conference pages or official codes; dynamic API behavior, model availability and manual verification.