LLM-as-a-judge bias comes in four documented forms, each with a cheap fix: score pairwise comparisons in both orders (position bias), grade concrete criteria and control for length (verbosity bias), grade with a model outside the families under test (self-preference), and swap 1-to-10 scores for binary checks (score clustering). Then report a bias-corrected score with a confidence interval.
A model grader is the only practical way to score open-ended output at volume, and it is also a measurement instrument with known, documented, reproducible faults. The faults are not subtle. They are large enough to reverse a comparison. The good news is that each one has a published detection method and a mitigation that costs little more than a second API call.
The symptom
The pattern that should trigger suspicion: a judge reports a clear preference between two candidates, the preference does not survive small changes to how the comparison was presented, and nobody looked because the aggregate number seemed plausible. An eval that quietly reverses when the candidates are swapped is not producing a measurement at all.
Four failure modes account for most of it.
Position bias
Presenting two candidate answers to a judge and asking which is better introduces a preference for one of the two slots that has nothing to do with the content. Wang and co-authors documented this directly in Large Language Models are not Fair Evaluators: presenting the same pair of responses in the opposite order can flip the verdict, and the effect is strong enough that a weaker candidate placed in the favoured position can be scored above a stronger one. Zheng and co-authors report the same effect in the MT-Bench work and treat it as one of the three headline limitations of the approach.
Detection. Run every pairwise comparison twice, once in each order. Count how often the two runs agree. That consistency rate is a direct measurement of how much position bias your judge and prompt are carrying.
For scale, the MT-Bench authors measured it with their default prompt: GPT-4 gave the same verdict in both orders 65.0% of the time, GPT-3.5 46.2% and Claude-v1 23.8%, with Claude-v1 favouring the first slot in 75.0% of comparisons. Adding few-shot examples to the judge prompt raised GPT-4 to 77.5%, at roughly four times the API cost.
Correction. Score both orders and count only the consistent verdicts, treating inconsistent pairs as ties. The MT-Bench work proposes exactly this, and the Fair Evaluators paper proposes balanced position calibration together with multiple-evidence calibration as a more systematic version. The cost is a doubling of judge calls, which is almost always worth paying because the alternative is a number that means nothing.
The related and cheaper option is to avoid pairwise comparison entirely. Grading each candidate independently against a fixed rubric removes the slot problem by construction, at the cost of a coarser signal.
Verbosity bias
Verbosity bias is a judge’s tendency to score a longer response above a shorter one that is no less correct or useful. Zheng and co-authors identify it as a distinct failure mode. This one is dangerous in a specific way: if the eval is being used to select a prompt or a model, and the judge rewards length, the optimisation loop will produce a system that pads.
How strong it is depends on the judge. In the MT-Bench “repetitive list” attack, GPT-4 rephrased the numbered list in each of 23 answers without adding information, and the rephrased copy was inserted ahead of the original, so a five-item list became ten items. Claude-v1 and GPT-3.5 preferred the padded answer in 91.3% of cases, GPT-4 in 8.7%. On AlpacaEval, Dubois and co-authors found that prompting gpt-4-1106-preview to be concise or verbose moved its win rate anywhere from 22.9% to 64.3%, with the model itself unchanged.
Detection. Log response length alongside every judge score and check the correlation. A strong positive correlation between token count and score, on a task where length should not matter, is the whole diagnosis.
Correction. State length neutrality explicitly in the rubric, and grade the specific properties you care about rather than overall quality. “Does the answer contain the account number from the source document?” cannot be won by padding. “Is this answer good?” can. Where length genuinely varies for legitimate reasons, controlling for it in the analysis is more honest than pretending the correlation is not there.
For pairwise win rates, length-controlled AlpacaEval is the published version of that analysis. A regression predicts the judge’s preference from a model term, the length difference between the two outputs and instruction difficulty, and the reported figure is the preference predicted at zero length difference. It narrowed the concise-to-verbose swing to 41.9% to 51.6% and raised Spearman correlation with Chatbot Arena rankings from 0.94 to 0.98. It controls for length and nothing else.
Self-preference bias
A judge can favour outputs produced by itself or by models in its own family, an effect Zheng and co-authors name self-enhancement bias. Their evidence is suggestive rather than settled: GPT-4 favoured its own answers with a 10% higher win rate and Claude-v1 with 25%, GPT-3.5 did not favour itself, and the authors judged the data too limited to conclude. The practical consequence holds either way: never use the same model as both the generator under test and the grader, and be careful comparing two models when the judge is one of them or shares a lineage with one of them.
Detection. Include a small set of items with known-correct human labels covering output from every model in the comparison, and check whether the judge’s error rate differs by which model produced the text.
Correction. Grade with a model outside the families being compared. Where that is impossible, run the same suite through two independent judges and report the disagreement rate as part of the result rather than picking whichever one is more convenient.
Score clustering
Asking a model for an integer score out of ten produces a distribution piled on a few values, typically 7 and 8, with almost nothing at the extremes. That compresses the range you are trying to measure across.
The G-Eval work of Liu and co-authors addresses this in two steps: the judge first generates chain-of-thought evaluation steps from the criteria, then fills in a form, and the final score is computed as a probability-weighted sum over the possible score tokens rather than the single sampled token. The weighted version produces a continuous score and recovers resolution that the discrete sample throws away.
Where token probabilities are not available, the practical substitutes are a shorter scale with concrete anchors, for example a three-point scale where each point is described in terms of an observable property, or decomposition of the judgement into several binary checks that are then summed. Binary checks are almost always more reliable than a holistic score, because the judge is being asked a question with a fact-shaped answer.
Calibrating against humans
None of the above tells you whether the judge is right, only whether it is stable. That requires human labels.
The procedure is a fixed cost paid once per rubric. Draw a stratified sample of items, ideally a couple of hundred spanning the range of quality the judge will see. Have humans label them against the same rubric the judge receives. Then compute agreement between judge and human, using a chance-corrected statistic such as Cohen’s kappa rather than raw percentage agreement, because raw agreement looks impressive on any skewed distribution.
The benchmark to compare against is human-to-human agreement on the same items, not perfection. The MT-Bench work reports GPT-4 agreeing with human preferences at above 80 percent, which is the level of agreement humans reach with each other, and that framing is the right one: a judge that agrees with people as often as people agree with each other is doing the job.
If judge and human agreement is well below the human-human ceiling, the rubric is usually the problem rather than the model. Rubrics fail when they ask for a global impression, when their criteria overlap, or when they use words such as “helpful” and “clear” without saying what would make an answer fail. Use the evaluation design guide when deciding how to revise the rubric and check it against human labels.
Repeatability and error bars
A judge run at nonzero temperature is a sampler, so the same item scored twice can produce two scores. Two things follow.
Fix decoding parameters for the judge and pin the judge model version. A provider silently updating the model behind an alias changes the instrument mid-experiment, and the resulting score movement will be attributed to whatever else changed that week.
Report uncertainty. Miller’s Adding Error Bars to Evals sets out the statistical treatment for language model evaluations: an eval score is an estimate from a finite sample, it carries a standard error, and comparisons between two systems need to account for both sampling variance and the fact that the same items are being reused across systems. A difference smaller than the confidence interval is not a result. This applies to model-graded scores exactly as it applies to the benchmark scores discussed in LLM benchmarks explained.
For a first pass on whether an eval set is even large enough to resolve the difference you care about, the Pass@k and evaluation cost calculator reports the worst-case margin of error for a given item count alongside the token cost of the run.
How to correctly report LLM-as-a-judge evaluations
Report a bias-corrected estimate with a confidence interval that covers both the test set and the human-labelled calibration set, never the raw share of outputs the judge passed. Alongside it, publish the judge’s measured sensitivity and specificity, the sample sizes behind each, and the judge model, prompt and tie rules that produced the number.
The case for this is made by Lee and co-authors in an ICML 2026 paper with that title. A judge errs in two directions, passing wrong answers and failing right ones. Its sensitivity, q1, is the probability it marks a correct answer correct; its specificity, q0, is the probability it marks an incorrect answer incorrect. Those errors pull the raw pass rate toward the middle, overstating low accuracy and understating high accuracy.
The correction inverts that relationship:
corrected accuracy = (p + q0 - 1) / (q0 + q1 - 1)Here p is the judged pass rate on the test set, and q0 and q1 are estimated on a calibration set where humans labelled items the judge also graded. It requires q0 + q1 > 1, a judge better than chance. The effect is not cosmetic. With the inputs from the authors’ README example (p = 0.4, q0 = 0.7, q1 = 0.9), the corrected accuracy is 0.1 / 0.6, about 0.17: with that judge, a 40% pass rate describes a system that is right about one time in six.
The interval has to carry two kinds of uncertainty: sampling noise in the n test items, and noise in the sensitivity and specificity estimates from m0 incorrect and m1 correct calibration items. Calibration size is usually what limits precision. In the paper’s worked case (p = 0.3, q0 = 0.7, q1 = 0.9), an interval shorter than 0.1 needs about 200 calibration examples, and the requirement climbs steeply as judge accuracy approaches chance. The paper also shows the correction stays unbiased when calibration and test accuracy differ, where existing approaches, including the difference estimator behind prediction-powered inference, can become biased.
The reference implementation, LLM-judge-reporting, wraps the estimate, the interval and an adaptive split of the calibration budget:
from llm_judge_reporting import point_estimator, confidence_interval
from llm_judge_reporting import allocate_calibration_sample
p = 0.4; n = 1000 # judged pass rate, test-set size
q0 = 0.7; q1 = 0.9; m0 = 200; m1 = 200 # specificity, sensitivity, calibration counts
th_hat = point_estimator(p, q0, q1)
ci = confidence_interval(p, q0, q1, n, m0, m1, alpha=0.05)
# Split a 200-label calibration budget after a 10-per-label pilot
m0, m1 = allocate_calibration_sample(200, p, q0_pilot=0.7, q1_pilot=0.9, m_pilot=10)Then publish, with every judge-graded number:
- The instrument. Judge model and pinned version, decoding parameters, and the full judge prompt and rubric.
- Raw and corrected scores. Both, each with its n, plus the estimated sensitivity and specificity with m0 and m1.
- Paired differences for comparisons. When two systems are scored on the same items, report the per-question paired difference with its standard error, as Miller recommends, and use clustered standard errors when questions come in related groups. The MMLU guide walks through a paired delta clustered by subject.
- Tie rules. For pairwise judging, state whether ties and order-inconsistent verdicts count. The choice moves the headline: in MT-Bench, GPT-4 agreed with humans on 85% of first-turn comparisons when ties were excluded and 66% when ties and inconsistent verdicts were included.
- Bias diagnostics. Position-swap consistency, the length-score correlation or a length-controlled win rate beside the raw one, and chance-corrected agreement with humans next to the human-human ceiling.
- Per-item verdicts. The judge’s rationale and verdict for each item, so readers can audit disagreements.
When not to use a judge at all
A model grader is the fallback for things no assertion can check. Where an assertion can check it, the assertion wins on every axis: cheaper, deterministic, reproducible, and free of all four biases above. Valid JSON, a schema match, code that compiles and passes tests, an extracted field equal to a reference string, a citation that resolves to a real document. Any of these should be a programmatic check.
The useful arrangement is layered. Programmatic checks handle everything they can, a calibrated model judge handles the open-ended remainder, and a small human-labelled set recalibrates the judge whenever the rubric or the judge model changes. That layering is part of the broader design covered in designing an LLM evaluation that actually tells you something, and the frameworks that implement model-graded metrics are compared in LLM eval frameworks compared.
Checklist
- Judge model and version pinned, decoding parameters fixed.
- Judge is not the model under test and not from its family.
- Pairwise comparisons run in both orders; inconsistent verdicts counted as ties.
- Response length logged and checked against score.
- Rubric decomposed into concrete, testable criteria rather than a global impression.
- Agreement with human labels measured with a chance-corrected statistic on a held-out sample.
- Per-item outputs logged, not just the aggregate, so regressions are diffable.
- Bias-corrected scores and confidence intervals, including calibration uncertainty, reported on every comparison.
A judge that survives all eight is an instrument. One that survives none is a random number generator with good prose style.
Related on this site
FAQ
what is verbosity bias in llm evaluation
Verbosity bias is an LLM judge rating a longer answer higher than a shorter one of equal quality. It matters most when the judge drives optimisation, because the system learns to pad. Check it by correlating response length with score, and correct it with length-neutral, criterion-level rubrics or a length-controlled win rate.
should i report the raw llm judge score or a corrected score
Report both, but lead with the corrected one. The raw pass rate mixes system quality with judge error, inflating low accuracy and deflating high accuracy. The corrected estimate removes the effect of the judge’s measured sensitivity and specificity, and its confidence interval should include uncertainty from the human-labelled calibration set as well as the test set.
how many human labels do i need to calibrate an llm judge
Plan on roughly 200 human-labelled calibration items as a starting point. Lee and co-authors show that with a judged pass rate of 0.3, specificity 0.7 and sensitivity 0.9, a confidence interval narrower than 0.1 needs about 200 calibration examples. Weaker judges need more, because the requirement grows quickly as judge accuracy approaches chance.
can an llm judge grade its own outputs
It should not. MT-Bench found GPT-4 favoured its own answers with a 10% higher win rate and Claude-v1 with 25%, though the authors called the evidence too limited to be conclusive. Use a judge from outside the model families being compared, or run two independent judges and report how often they disagree.
