If you let an AI do the scoring, start by doubting the scores

When I had an AI score quality, just measuring the same thing twice moved the number by 6.1 points. And that wobble was exactly the size of the “+7.2-point improvement” I was about to adopt. Put the two head-to-head and the win rate was 53% — indistinguishable from a dead heat.

A judge is an instrument, and every instrument has its own noise and its own bias. Don’t make decisions off its output until you’ve measured that first.

In one line: measure the judge’s spread first, and don’t call any difference at or below that spread an improvement.

Turning quality into a number is hard. Whether a translation is good, whether a summary is on point, whether an answer is helpful — there’s no yardstick for that kind of “good.” So there’s a move everyone reaches for now: have an AI do the scoring (LLM-as-judge; from here on, judge = the AI doing the scoring). Faster and cheaper than a human, and it will hand back “this one passes, this one fails” for 100 items or 1,000.

I had handed the entire judgment of translation quality over to that judge. Try a change, have it scored, adopt it if the score goes up. Looks like the obvious way to run things.

Then at some point I realized the order was backwards. I was measuring improvements without ever having measured the judge itself. If the yardstick itself stretches and shrinks, what does an “improvement” measured with it even mean? When I actually turned around and doubted the judge, one of the improvements I was about to adopt turned out to be — an illusion.

Finding 1: measure the same thing twice, and it moved 6 points

The first stumble came from a trivial check. What happens if you give the same translation to the same judge and have it scored twice?

If the answer comes back the same, the yardstick is stable. It didn’t. Scoring translations from the exact same model with the exact same settings under absolute scoring (the approach where each translation gets a per-item pass score) gave me 89.6% the first time and 95.7% the second. Nothing had changed, and the two runs were 6.1 points apart.

Six points, in translation, is the width of what counts as a “big improvement.” Which means the judge’s noise was the size of one whole improvement.

And it did real damage. At the time I was holding a hypothesis: hand the model context (the surrounding lines of dialogue) and the translation gets better. Measured under absolute scoring: no context 89.6% → with context 96.8%, +7.2 points. It looked like a clean win. I came very close to concluding that context works.

Finding 2: head-to-head, the win rate was 53% — a dead heat

But +7.2 is almost exactly the size of the gap the judge produced just by measuring the same thing twice (6.1 points). A difference no bigger than the spread can’t be believed on its own. So I changed how I measured.

I dropped absolute scoring for pairwise (relative scoring: put two translations side by side and ask nothing but “which one is better?”). For the same source text, I put the no-context translation and the with-context translation left and right, and let the judge pick only the better one. The order is shuffled to cancel out positional bias.

The result: with-context won 53% (win rate with ties excluded — and there were a lot of ties). Essentially even. And that 53% is only a point estimate; I didn’t record n or a confidence interval at the time. So to be honest about it, this isn’t “a win at 53%” — all I can say is “indistinguishable from 50%, a dead heat.” Either way, the +7.2 points were gone.

Here’s what was actually going on. Under absolute scoring, the criteria shift from one scoring run to the next depending on whether the judge is in a strict mood or a generous one. The no-context run happened to be scored harshly and the with-context run happened to be scored kindly — that mood gap was what +7.2 really was. Put the two directly against each other on the same source text and the mood falls equally on both, cancels out, and only the true difference (roughly none) is left.

This isn’t just a story about a lever I thought worked turning out to be an illusion. It was a change in how I operate: anything I’d ever ruled a “win” under absolute scoring is worth re-measuring pairwise.

Finding 3: the AI judge writes off paraphrase as an error

There was one more thing: the AI I’d handed the scoring to (the judge) had a systematic bias. It is far too strict about rewording and paraphrase.

Ask it to score whether a translation matches, and the judge will happily drop “same meaning, different wording” into “mismatch (= error).” A translation any human would recognize as saying the same thing gets a mechanical X. And this held stubbornly no matter how I worked the prompt. Swap in a smarter model for the judge, hand it the surrounding context, make the judging procedure more careful — the bias doesn’t go away. Even with the smartest judge, nearly 80% of the “error” verdicts were wrongly flagged (the detailed numbers are in the appendix).

That skews the whole aggregate. I audited 90 pairs by hand: the true fail rate is 6.7%. The judge, meanwhile, shouts “fail” on 15–20%. A 2–3x over-report. Use that raw fail rate as a KPI and you’ll park people on rework nobody needed, driven by a false alarm that quality is bad.

The fix is calibration. For this judge: true fail rate 6.7% ÷ raw fail rate 15–20% = a calibration factor of 0.3–0.45. Multiply by that measured range before aggregating — never use the judge’s raw numbers as they come. The factor differs from judge to judge, so always re-measure it with an audit.

So this is how I changed the way I operate

The three findings look like three separate failures, but they share one root. A judge is an instrument, and every instrument has its own noise and its own bias. Don’t make decisions off its output until you’ve measured that first.

Lessons

1. Measure the judge’s spread first, and don’t call any difference at or below that spread an improvement. +7.2 was the same size as the 6.1pt spread produced by measuring the same thing twice; on its own it couldn’t be called an improvement.
2. Judge head-to-head (pairwise), not on absolute scores. Under absolute scoring, the criteria themselves move with the judge’s mood. Put the same two conditions back against each other and +7.2 came back as a 53% win rate — a dead heat.
3. If you’re going to use an absolute rate, measure the calibration factor with a human audit. The judge’s raw fail rate was 2–3x the true value. An uncalibrated absolute rate distorts business decisions directly.

In practice, it came down to four rules.

Only with that calibrated judge in place does the “42% → 84%” from raising a small model into a specialist, or the “+14.7pt” from comparing models through a gate — both improvement numbers covered in other articles — become worth believing. Fixing the yardstick is work you do before you start measuring.

Measure-first doesn’t mean “measure a lot.” It means measure the right thing, and measure it right. And the first thing you ought to measure right is, usually, the instrument itself.

---

Appendix: raw data

Terms: precision = of the items the judge called X (mismatch), the fraction that really were X. better-match = a procedure where the judge first picks the closest counterpart out of a pair of translations, and only then judges.