That "accuracy" is a definition you chose

When you say “improve accuracy,” that “accuracy” isn’t a law of physics. It’s nothing more than a definition someone chose.

Optimize against a judge that returns nothing but pass or fail, and only the axes that definition captures get polished; the ones it doesn’t are left behind. And because the number keeps going up, you can’t see that anything is being left behind at all — optimize all-out toward an impoverished definition and you converge neatly on “correct but flat” (Goodhart’s law).

So when improvement stops, there are two questions to ask. Everyone asks “has the model hit its limit?” The one that gets skipped is “does my definition of ‘accuracy’ actually capture the goodness I’m trying to measure?” In my case, the ceiling wasn’t in the model. It was in the definition.

“Improve the accuracy.” It’s one of the most common instructions thrown around in a meeting. Nobody objects. Higher accuracy is better — it looks self-evident.

But stop for a second. What, concretely, is that “accuracy”? For translation, is it the rate at which the meaning comes out right? For summarization, how much of the key content it covers? For an answer, the absence of factual errors? Every one of those can be called “accuracy.” And — which one you call “accuracy” is something you chose. It isn’t the single number the laws of physics handed down. It’s a definition that was picked.

I didn’t come to this because a number plateaued. It came from a single comment from my target readers.

What I assumed: accuracy is one number, and you just keep raising it

I ran translation quality off a single number. Hand the translations to a judge (an AI grader) that returns pass or fail, and call the pass rate “accuracy.” A judge will mass-produce pass/fail verdicts for 100 items or 1,000. This judge looked at three axes — is the meaning right, is the language right, has anything fallen apart — and folded them together into one “overall OK” (overall_ok). With a single number, up or down settles it on the spot. As an operational setup, it’s clean.

And all I was thinking about was how to push that number up. Re-pick the better translation (best-of-N), then fine-tune the model on those preferences (CPO — training that teaches the model your preferences from the gap between good and bad examples). Drive overall_ok upward. And if that number eventually plateaued, it would be the sign the model’s capacity had run out — that’s how I intended to read it. When the improvement curve flattens, you’re capacity-bound: technically, that looks like sound reasoning. As it happened, I stopped before I ever put that reasoning into practice.

The realization: it started with one reader’s comment — “correct, but not good”

What stopped me wasn’t a number. It was feedback from the target readers. “The meaning is right. The language is right. And still, it doesn’t land when you read it.” Correct, but not good — that was the complaint.

With that in hand, I went back and read the translations overall_ok had passed side by side with the source text. Now that it had been said out loud, it was exactly right. Every passing translation was indeed “not wrong.” The meaning was right. The language was right. Nothing had fallen apart. But they weren’t good. Correct, and flat. A translation that moves the person reading it and a translation that merely satisfies the conditions were sitting in the same “pass” bucket.

Here’s how it worked. My judge looked at three axes. But the axis that actually separates value — the core of what “good” means to the reader — wasn’t among those three. If the meaning is right, overall_ok returns yes. So from the judge’s point of view, a flat translation and a living one are indistinguishable. Both are yes.

Only then did I see how precarious my earlier reasoning was. What would have happened if I had gone on chasing the number, running best-of-N and CPO? overall_ok would climb, and eventually flatten. I would have read that as “the model’s limit” and gone off to make it bigger. But the ceiling wouldn’t have been in the model — it would have been in the definition. On the value the judge doesn’t capture, improvement doesn’t show up in the number even when it happens. Optimize toward an impoverished definition and you converge neatly on the single point of “correct but flat” — you don’t have to wait to observe that; it follows from the structure. One comment from a reader stopped me before I walked into that dead end.

When a metric becomes the target, it stops being a metric (Goodhart’s law)

This was the textbook trap that Goodhart’s law — the rule of thumb that the moment you make a metric a target, it stops being a good metric; the formulation is Strathern’s — lies in wait with.

overall_ok was never more than a proxy for quality (an easier-to-measure quantity that stands in, close by, for the thing you actually want to measure). What I actually want to measure is “does the reader feel this is good.” That’s hard to measure. So I proxied it with a pass/fail on three measurable axes. Up to here, this is all correct.

The problem starts the moment you install that proxy as the target of the optimization itself. best-of-N and CPO both, the smarter they work, pull toward “translations that raise overall_ok.” And what happens then is that only the axes the proxy captures get polished, and the axes it doesn’t are left behind. And because the number keeps going up, you can’t see that anything is being left behind at all. Optimize all-out toward an impoverished definition and you converge neatly on that single point, “correct but flat.”

So when it looks like “improvement has stopped,” there are two questions. Most people ask only one of them.

• “Has the model hit its limit?” — everyone asks this one. And this is often where they go wrong.
• “Does my definition of ‘accuracy’ actually capture the goodness I’m trying to measure?” — this one gets skipped, never asked.

In my case, the answer was in the second. The ceiling wasn’t in the model. It was in the definition.

Lessons

1. “Accuracy” is a definition that was chosen, not a fact you were handed. The moment you decide what to call accuracy, you have also decided which value you won’t capture.
2. The moment you install a metric as the target, it stops being a metric. The smarter the optimization, the more it polishes only the axes the proxy metric captures, leaving the rest behind, flat.
3. When improvement stops, suspect the definition before the model. Keep the real definition in a small human-calibrated set, and validate the judge by its agreement with that set. Humans are the definition; the judge is the volume.

So here’s how I changed the way I work

In one line: before “improving accuracy,” design what you’re going to call accuracy. A definition isn’t a tool sitting downstream of measurement. The definition is the furthest-upstream design decision there is. Concretely, it came down to four things.

1. Suspect the definition before you optimize. Before running best-of-N or CPO, check against real outputs whether there’s value this judge isn’t capturing. Running an optimization on top of an impoverished definition is sprinting at the wrong target. The faster you run, the further away you get.

2. Don’t fold it all into a single overall_ok. An overall_ok crushes axes of different value into one bucket. The moment you crush them, you can’t see which axis you’re losing on. The axis that actually matters should stay as an independent axis and be looked at on its own. What to unify and what to keep apart is the same skeleton — don’t fold the axes, split them out — that I worked through in When “which one is faster?” turns out to be the wrong question.

3. Keep the real definition with the humans; use the judge for volume. The ultimate definition of “good” is whether the target readers find it natural. Carve that into a small human-calibrated set and validate the judge against it as ground truth. The judge says yes, a human says “this isn’t good” — when that disagreement shows up, what you fix is the judge’s definition, not the translation. Humans are the definition, the judge is the volume: that’s the division of labor.

4. Don’t take the judge at face value. The judge is itself a proxy, and a proxy has its own bias and its own spread. Before you use pass/fail numbers to make decisions, calibrate the judge itself — I went through that in detail in If you’re going to let an AI grade, start by doubting the grades. This article’s problem — that overall_ok is only ever a proxy — is one form of the damage that does.

When improvement stops, that is often not a signal that you’ve hit a limit but a signal that your definition is impoverished. When the number stops moving, suspect the definition of “accuracy” you chose before you make the model bigger. Measure-first doesn’t mean “measure a lot.” It means measure the right thing, and measure it right — and what decides “the right thing” is, always, the definition you chose.

Appendix: raw data

The core of this article isn’t a quantitative value; it’s a design decision about a definition. Here is what I can leave behind as support.