Don't stop at measuring a single mode

I pushed the Live translation number from 90 to 96 and was about to declare “improved.” Then I re-measured across all modes: Page and Manga had each quietly dropped 4 points.

This wasn’t an accident. It’s a shape that shows up again and again (a regime shift). On the same day I checked it with three separate moves, and all three reproduced the same thing: the targeted mode goes up, the other side goes down.

The discipline is simple. If your product is used in more than one mode, you don’t get to declare victory on one mode’s number. Only when you lay the +6 and the −4 next to each other does a scale exist at all. As long as you’re looking at one side only, there’s nothing to decide.

LYR’s translation model runs not in one mode but in several modes (usage scenarios where what the user is doing is fundamentally different). Live, translating video subtitles in real time; Page, translating a still screen in one batch; Manga, laid out in panels. The same model does double duty across all three.

One day I set out to sharpen one of them — Live. And I got exactly the number I was aiming for. I was about to declare “improved.”

If I had, I would have been wrong.

I sharpened Live, and it went up exactly as intended

Live’s translation quality had clear weak spots: it translated idioms literally, picked the wrong subject, hallucinated words. So I had the teacher model produce N candidates, selected among them by quality (best-of-N), trained on that high-quality data, and finished with Live-specific preference training (CPO).

The result was exactly what I aimed for. The 8B model’s (8 billion parameters — the small one) Live overall (an aggregate quality score, as a % on the same evaluation set) went 90 → 96. The gap people used to describe as “the 8B’s Live is 6 points below the strongest model currently in service, the 27B (27 billion parameters)” had gone to zero. The high-quality data built with best-of-N had closed Live’s gap entirely. The story of that achievement itself is in A small specialist drew level with the strongest model in service.

At that point, this was the only number in front of me.

+6 points. Level with the 27B. It looked like an unambiguous win. If I had stopped there, I would have used that number to decide “adopt the Live-specialized model.”

Measured across all modes, something was quietly broken

But this product isn’t used only for Live. The same model translates Page and Manga too. So instead of Live alone, I re-measured across all modes.

Here’s what came out.

The price of taking Live up +6 was −4 each on Page and Manga. The modes I wasn’t targeting had quietly degraded. If I had only ever looked at the Live number, that −4 would have stayed invisible forever.

A few points either way is a number to handle carefully. But the fact that the same shape — targeted mode up, untargeted mode down — reproduced in three independent experiments, as we’ll see below, is hard to explain as coincidence.

This isn’t an accident. It’s a regime shift

Why does sharpening Live drag Page and Manga down? Once, and you can write it off as noise. But this was different: I hit the same phenomenon consistently with three separate moves.

The targeted mode goes up. But every time, another mode goes down. All three moves have the same shape. That’s not an accident; it’s structure.

The culprit is a regime shift (the phenomenon where the same model behaves differently in regions with different usage patterns). The 8B is a multi-task model doing double duty across three modes with a single set of weights. And each mode has a different output “shape.” Live returns a single line. Page and Manga return multiple blocks (3–7 entries). Add more data in one shape, and the distribution of entry counts and styles the model learns gets skewed. The mode with the opposite shape falls apart as a result.

The third row of the table above makes this especially clear. When I added single-line data (3661 items), Page broke the most in this experiment (−5.1, versus −1.1 for Manga) — of the two modes that return multiple blocks, its entry-count distribution got pulled toward “1” more deeply. The modes interfere strongly with each other through the data distribution. Which means optimizing for one of them is, necessarily, an attack on the other.

One mode’s number is not proof of victory

The discipline you can pull out of this is simple.

If your product is used in more than one mode, you don’t get to look at one mode’s number and declare “improved.” A gain in the mode you targeted hides degradation in the modes you didn’t. And that degradation goes on quietly existing until the day you measure that mode.

With the context training, I nearly did exactly this. I was about to look at Live alone and conclude “a pure improvement.” It was only because the discipline made me re-measure across all modes that the −5.1 on Page/Manga came to light. The number for the mode you targeted always arrives first, and always wearing its best face. Which is precisely why you can’t stop there.

This is the mode-level version of the manifesto’s first principle, Measure First. Narrowing what you measure down to one thing is the same as choosing an impoverished definition. Only by measuring richly are you actually looking at a trade-off — able to put +6 and −4 on the scale and decide “take it anyway / don’t.” As long as you’re looking only at Live’s +6, the scale itself doesn’t exist.

Only once the trade-off is visible does designing the −4 away (retraining on a mix, per-mode routing) get onto the agenda — the details of those moves I’ll leave to A small specialist drew level with the strongest model in service. Had I stopped at declaring victory on one mode, that agenda would never have existed.

The pitfall of an era where you can measure fast is always right here. You measure one targeted point fast, you see a good number, you become confident. And right next to that confidence, the thing you didn’t measure is quietly breaking.

Lessons

1. One mode’s number is not proof of victory. A gain in the mode you targeted hides degradation in the modes you didn’t. And that degradation goes on quietly existing until you measure it.
2. Only by measuring richly do you see the trade-off. Only when you can lay +6 and −4 side by side can you decide “take it anyway / don’t.” As long as you’re looking at one side, the scale itself doesn’t exist.
3. Interference happens through the data distribution. As long as a single model does double duty across modes, skewing it toward one side’s format breaks the other. Add data in a shape that doesn’t wreck the distribution of entry counts and styles.

---

Appendix: raw data