LLM-as-Judge and Its Biases
In one line: a model can grade generated output at roughly human-level agreement and at a fraction of the cost, and it arrives with three specific biases you have to design around before the numbers mean anything.
Why it became the default
Human evaluation is the gold standard and it does not scale — it is slow, expensive, and cannot run on every commit. A model judge runs in seconds for cents, which makes evaluation something you do continuously rather than quarterly.
And the agreement is genuinely reasonable. Strong judge models reach above 80% agreement with human evaluators on benchmark comparisons — against a human-to-human agreement rate of about 81%.
That comparison is the one to quote, because it reframes the objection. The judge is not perfect; neither are your annotators. Two trained humans disagree about one time in five, and asking a judge to beat a ceiling that humans do not reach is the wrong bar.
The three biases
Position bias. In pairwise comparison, judges favour the first-presented response — measured as high as a 75% preference for the first position, and strong judges have been shown to flip their verdict when the two answers are simply swapped. If your evaluation harness always puts the new system second, it is systematically disadvantaged.
Verbosity bias. Longer responses are preferred regardless of whether the extra length adds anything. This one is quietly dangerous because it creates a gradient: optimise against a verbose-biased judge for a few iterations and you have trained a system to pad.
Self-preference. Judges favour text from their own model family, by roughly 10–25%. If you generate with one model and grade with the same model, you have built a system that grades itself, and it will report that it is doing well.
The mitigations
Each bias has a specific and cheap countermeasure, and naming the pairing is what makes the answer concrete.
| Bias | Mitigation | Cost |
|---|---|---|
| Position | Evaluate both orderings and average, or discard disagreements | Two judge calls instead of one |
| Verbosity | Penalise length in the rubric explicitly, or control for it | Prompt work, and a check that it took |
| Self-preference | Judge with a different model family than you generate with | A second provider dependency |
| Idiosyncrasy | Panel of judges across families, majority vote | Several calls per evaluation |
The position mitigation deserves emphasis because it doubles as a quality signal. Run both orderings: where the judge gives the same verdict either way, trust it. Where it flips, that pair is genuinely too close to call, and counting the flip rate tells you how much of your evaluation is noise. A high flip rate means the comparison is not measuring anything.
Designing the rubric
Most judge quality comes from the rubric rather than the model, and the moves are unglamorous.
Score one dimension at a time. A single "quality 1–5" collapses faithfulness, tone, completeness and format into one number nobody can act on. Separate scores are more reliable and tell you what to fix.
Prefer comparison to absolute scoring. "Which of these two is better" is far more stable than "rate this 1–5", because absolute scales drift and cluster. If you need absolute scores, anchor each point with an example.
Use few points, not many. A 1–10 scale invites false precision and judges cluster in the middle. Three or five well-defined points are more reliable.
Require a reason before the verdict. Making the judge state its reasoning first improves consistency and, more usefully, gives you something to read when you disagree with the score.
The failure that ends the argument
There is one use where a judge should not be trusted at all: grading its own generations in a loop that optimises against it.
That closes exactly the circuit the estimation-agent literature warns about — an evaluator inside the optimiser's reach. The system will find the judge's blind spots, because that is what optimisation does, and the score will improve while quality does not.
If a judge is used as a training or selection signal rather than only for reporting, it needs a periodic human check on the frontier of what the optimiser produced — not on a random sample, because random samples will not contain the exploits.
Key takeaway
Model judges reach roughly human-level agreement — above 80%, against 81% human-to-human — so accuracy is not the objection. The objection is three systematic biases that sampling cannot average away: position, worth up to 75% preference for whichever came first; verbosity; and self-preference of 10–25% for the judge's own family. Mitigate each specifically, use the order-flip rate as a noise measure, put the effort into the rubric rather than the model, and never let a judge grade generations from a loop that is optimising against it.
Next: evaluating retrieval-augmented systems, where the failure usually is not the model.