The "97% improvement" I threw away

The automated measurement came back with good news: junk English down 97%. I didn’t take the number. I threw it away.

Going through the raw output one page at a time, the 97% turned out to be two measurement bugs on top of each other. (1) Randomness in the compression step occasionally produces a broken instance, and (2) the aggregator’s parse-failure fallback was counting the entire raw text. Measured fairly, all three came out about the same (5 / 7 / 9 words).

But don’t bury the technique itself along with the number. “The metric is rejected; whether the technique works is still open.” Drawing the line between what you know and where your knowledge stops is what leaves you room to measure it properly next time.

Up to the previous parts, I had compressed (quantized) my own small model to make it fast and cheap. Quality barely drops — except that in the just-under-10% of “borderline” cases, it occasionally falls apart. One form that breakdown takes is raw English leaking into the translation (stray English words left sitting inside the Japanese output).

To kill this “leftover English,” I tried a repair technique. The automated measurement came back with dramatic good news. Junk English down 97%.

I didn’t take that number as it stood. This article is about why I threw away a spectacular 97%.

The good news: 97%

I applied the repair technique (a post-processing step that recovers the accuracy lost to compression by rearranging the model’s internal weights; I’ll call it rotation from here on) to a hard-compressed model and had it translate 30 pages of manga. Here’s what the aggregate looked like.

And lined up next to the un-compressed full-precision version (FP16), the story looked perfect.

• Full-precision version’s junk: 3 words (essentially clean)
• Hard-compressed version: 968 words (compression blew junk up by +965 words)
• Repaired version: 33 words (935 words recovered, back to roughly the full-precision level)

“Compression catastrophically multiplies the junk, and repair brings it back to full-precision levels” — a storyline as clean as a textbook. Take it as it stands, and you can conclude that the repair technique is worth shipping.

The number was too good

What stopped me was that −97% was far too clean. Real model breakdowns don’t line up that obediently in one direction.

There was something else: two separate measurements disagreed. Measure the same repair technique with a general-text metric (perplexity) and the recovery stops at 29%. A 97% recovery on the translation task only — when two metrics are orders of magnitude apart, one of the two measurements is usually broken.

So before trusting the aggregate, I decided to look at the raw output itself. I had all three models (full-precision, compressed, repaired) translate 30 pages of manga, lined the outputs up in a browser, and went through them page by page with my own eyes.

Re-measuring it fairly

Looking at them, it was obvious immediately. Two corrections were needed.

Correction 1: recount. The aggregator was counting even the model’s “talking to itself” as English. Before returning a translation, the model writes a reasoning scratchpad internally (a working area it closes with </think>). Pull out just the translation — excluding the scratchpad — and recount, and the junk in the full-precision and repaired versions dropped to single digits.

Correction 2: re-roll the instance. But the compressed version’s 968 didn’t disappear from recounting alone. That instance was emitting raw English into the translation itself, not into the scratchpad (hole 1, below). Re-run the same compression step, draw a healthy instance, and the translation-only count fell to 7 words.

Here’s the result after both corrections (the compressed column is the value from the re-rolled healthy instance).

* A re-rolled healthy instance. Not the unlucky instance that emitted 968.

All three, about the same. Compare healthy instances against each other and count only the translation, and neither compression nor repair moves the junk much at all. Both the 968 and the 33 were illusions that stood only until those two corrections. The −97% vanished without a trace.

Two holes

So where did that 968 come from? The two bugs compounded.

Hole 1: randomness in the compression. The process of compressing a model includes a “calibration” step that tunes the rounding using a small number of samples, and that’s where randomness enters. Same procedure, and yet every run produces a slightly different model. One unlucky time, a “broken” compressed version was born, and it emitted raw English instead of a translation. Re-run the same process and all 30 pages were normal, with 7 words of junk. The 968 was nothing more than one unlucky draw.

Hole 2: the aggregator’s fallback. The program that counts junk was designed to parse the model’s structured output (JSON) and pull out just the translation. But when parsing fails, it falls back to counting the English in the entire raw text. The reasoning scratchpad, echoes of the prompt, everything gets counted as English. The chain goes like this.

broken instance → JSON parse fails → the fallback counts English across the whole raw text → explodes to 968

An unlucky instance (hole 1) times an over-counting fallback (hole 2) multiplied into an enormous fake difference. Either one on its own wouldn’t have broken this spectacularly.

Ironically, the supposed “winner” — the repaired version — had a defect of its own. On 29 of 30 pages, the </think> tag from the reasoning scratchpad had leaked out ahead of the translation. The aggregate said not a word about that regression.

Keep only what’s real

Once the noisy numbers were thrown out, this is all the “real” I had left.

• Perplexity on general text: 10.65 → 10.42
• Agreement with the reference model: 81.6% → 85.8%

Small, but solid improvements. Nothing spectacular.

And one more thing: I was careful about how I threw it away. I did not conclude that “the repair technique doesn’t work.” Both the </think> leakage and the failure to reduce junk could well come from a confound in my own measurement procedure (I measured through a makeshift evaluation path rather than the proper server). So I closed it out this way — “the −97% automatic metric is rejected; whether this repair works in production is still open, and it needs to be measured properly again.”

Don’t confuse “the measurement was wrong” with “the technique was wrong.” Being fair means not killing the technique as collateral damage — and that matters as much as killing the flashy number.

Lessons

1. The flashier the number, the more you look at the raw output before killing it. A number as clean as −97%, or metrics disagreeing by orders of magnitude, isn’t a victory — it’s an alarm. Don’t accept good news until you’ve looked at the raw output upstream of the aggregate, page by page, with your own eyes.

2. Pin the seed on non-deterministic steps, and suspect the aggregator’s fallback. If randomness is in the mix, a single run tells you nothing but instance-to-instance variation (an improvement that doesn’t reproduce isn’t an improvement). And a fallback on parse failure will quietly manufacture an enormous fake signal — look at what it’s actually counting once it has swallowed the error.

3. Close out measurement errors and technique errors separately. When you throw away a false victory, don’t bury the technique with it. Drawing the line — “the metric is rejected, the technique is still open” — is what leaves you room to measure it properly next time.

And underneath all of this sits the manifesto’s first principle, Measure First — but measure it right. A wrong measurement is worse than no measurement. Because it lets a false victory ship as a real one.

---

Appendix: raw data

Scope: a 4B model compressed hard (INT4) + learned rotation repair, translating 30 pages of manga, measured on an A100.

Footnotes

1. “Agreement with the reference model” = how well the compressed version reproduces the next word that the un-compressed full-precision version (FP16) produces (top-1 agreement rate under teacher forcing). The +4.2pt means “it moved a little closer to FP16’s behavior,” not that the translation itself got better. ↩