I cut the bottom-rank rate from 64% to 24%, and held the top-rank loss to 12pt

I cut the bottom-rank rate hard, from 64% to 24% (−40pt), and held the drop in top-rank rate to a small 24% → 12% (−12pt). For the 3 firsts I gave up, I handed back 10 lasts (out of 25).

What breaks the user experience in a subtitle product isn’t the average, and isn’t the best — it’s a single worst case. So I moved my target from the ceiling (peak quality) to the floor (the one worst case). I sold a little ceiling and bought floor.

The average agreement rate (89–96%) doesn’t capture that improvement at all. If the problem is in the tail, the average dilutes it. The ceiling and the floor move independently — what I should have decided first was which of the two I wanted to move.

How it was measured: 25 hard cases translated by three configurations (compression only / +rotation / before compression), then ranked blind. Bottom-rank rate / top-rank rate = the share of cases where a configuration came last / first among the three (each sums to 100% across the three configurations). I use the bottom-rank rate as a proxy for catastrophic failure — I confirmed by eye that most of the translations that fell to last place were critical breakdowns.

I compressed my own small model to a quarter of its size (quantization). It got faster and cheaper. Quality, on average, barely moved. On paper there was nothing to complain about.

But when I actually lined the translations up and read them, every so often something made me flinch. A Japanese line comes back still in Japanese. A character’s gender swaps. A proper noun turns into something else entirely. The frequency is low. But in subtitles, one of these is enough to shatter immersion instantly.

Same average, completely different experience. Chasing that discrepancy ended up forcing me to rebuild how I use the word “quality” at all. What was mattering to the product was not the ceiling (peak quality) but the floor (the one worst case).

The average makes the difference disappear

First, what was actually happening.

The heavily compressed model (INT4 = rounding the weight values to 4 bits) was wildly hit or miss. I took 25 deliberately difficult lines (held-out hard cases), translated them with three configurations, and had a stronger model rank them blind. Every number below is a relative rank within those 25 cases — not an incidence rate across production as a whole. Here’s what came out.

• The compressed model’s top-rank rate (the share where it gave the best translation): 6 of 25 = 24%
• The compressed model’s bottom-rank rate (the share where it gave the worst translation): 16 of 25 = 64%

Sometimes it produced the best output; often it produced the worst. The spread is wide — the technical term is bimodal (it splits into a good peak and a bad peak). Superb when it hits, an accident when it misses. As subtitles, that’s unusable.

And yet none of this danger shows up in the average. Here is the token agreement rate before and after compression.

The difference is essentially zero. Had I looked only at these numbers and concluded “no degradation,” I would have missed the failures entirely.

Why is the average so blunt? The loss from compression concentrates in the just-over-10% of “borderline” calls (the story I wrote in Part 1). The remaining 90% return the same answer as before compression, so the average agreement rate gets diluted by that 90%. The average waters down the tail — the bad side. What mattered was the just-over-10% buried inside it.

Why a single echo is fatal

Why weight “the one worst case” that heavily?

The north star of subtitle translation is “subtitles with no presence.” The translation dissolves naturally into the screen and the user forgets they’re even reading a translation — that’s the best state. But the moment a Japanese line comes out still in Japanese (echo = the failure where the model returns the source untranslated), you snap out of it: “ah, this is machine translation.” Immersion isn’t a continuous quantity. Once it breaks, it goes back to zero.

This has the same structure as the “critical breakdown” I wrote about in the article on the bottleneck. Even if 99 out of 100 are perfect, one echo decides the impression of the whole experience. Users don’t experience the average. They experience the worst case.

So the metric to chase wasn’t average quality. It was the floor — quality at its worst. A catastrophic-failure rate is nothing more than that floor measured upside down (the lower the rate, the higher the floor).

So how do you measure it? The headline number of this article (64% → 24%) is the share of those 25 hard cases where a model produced the worst translation of the three — the bottom-rank rate. Because these are relative ranks, coming last doesn’t automatically mean the output was critical (even before compression, 12% end up last). But when I read the last-place translations one by one, most of them were critical breakdowns: gender flips, mangled proper nouns, subject and object swapped, echo. So the bottom-rank rate can be used as-is as a proxy for catastrophic failure — and that’s what “catastrophic-failure rate” means from here on.

Don’t raise the ceiling; raise only the floor

So how do you reduce catastrophic failures?

What worked here was a correction called rotation. If you mathematically “rotate” the model’s weights before compressing, the rounding error doesn’t pile up on a handful of values; it spreads thin across all of them. Borderline calls become less likely to be flipped by error — it’s a technique that suppresses breakdowns in the tail on purpose.

I added it and re-measured the same 25 cases.

Read this carefully. The top-rank rate went down (24% → 12%). Rotation does not make the model “smarter at its best.” But the bottom-rank rate fell from 64% to 24%, to roughly a third (mean rank moved 2.40 → 2.12 as well). −12pt given up against −40pt gained — what came back is far larger than what I handed over. Echo went from 4 to 0. Quality at its worst — the floor — moved much closer to before compression (12%).

Perplexity on general text (ppl = a measure of the model’s “lack of confidence,” lower is better) tells the same story: compression had degraded it to 10.65, and rotation brought it back down to 9.95, essentially level with the 9.94 from before compression. In exchange for a slight drop in the top-rank rate, the broken floor — quality on the worst side — recovered.

As a product, that was enough. For subtitles, removing ten worst translations does more than adding one best translation. I dropped the ceiling and chose the floor.

(The “ceiling or floor” choice itself came up in the Pareto article as “how to pick an operating point.” There I laid them out as method A = compression only and method B = rotation. This article digs into the measurement behind that single point. Where you sit is decided not by which one is “better,” but by which axis the product needs.)

The ceiling doesn’t come back. And automatic metrics hide the failures

First, this isn’t a clean win. What rotation restores is the floor; the ceiling doesn’t come back — it still falls short of before compression (top-rank rate 64%, mean rank 1.48). The peak quality lost to compression cannot be recovered by this correction.

And one more thing. The scariest part of this episode wasn’t the technical side. It was that the metric I trusted first had made the failures invisible.

An agreement rate of 89–96% — a respectable number. Had I used it as the acceptance criterion, the compressed model would have passed as “essentially no degradation” and gone to production spitting out echoes. It’s not that the metric was wrong. And it’s not that averages are bad either — ppl (perplexity) is an average too, and it captured the floor’s recovery just fine. The question is what the metric dilutes. Token agreement rate waters down the small number of cases where the failures happen with the vast majority of “nothing wrong here,” so the worst cases I most wanted to see became structurally invisible. When the problem lives in the tail, question what your average is diluting, and if you have to, count the worst cases directly.

In the end, what caught the failures was a plain manual job: reading the generated translations by hand and counting critical breakdowns one at a time. I still haven’t automated it. If the automatic metric is blunt, the only option is to count the tail directly. Shift what you measure one notch, from the average to the worst case. That was all it took — and I very nearly failed to see it.

Lessons

1. A product’s value is in its floor, not its ceiling. In a product where a single breakdown ruins the experience, the floor is the KPI. Write your acceptance criteria as “worst-case rate at or below X%,” not “average score of X or higher” — users don’t experience the average, they experience the worst case.

2. Ask what your average metric dilutes. Not every average is bad (ppl captured the floor’s recovery). But a token agreement rate of 89–96% was watering down the just-over-10% where the failures happen with the vast majority of “nothing wrong here.” If the problem is in the tail, switch to a metric that counts worst cases directly.

3. The ceiling and the floor move independently. I cut the bottom-rank rate hard, from 64% to 24% (−40pt), and held the drop in top-rank rate to 24% → 12% (−12pt). For the 3 firsts I gave up, I handed back 10 lasts. Decide “which part do I want to improve” first, rather than “make the whole thing better.” Aim at the wrong place and even the right technique comes up empty.

The manifesto’s third principle — Optimize the Right Proxy for UX, and within it “a good proxy looks at the one worst case (the tail), not the average” — was shoved in my face in its plainest form by this episode. In speed or in quality, what ends up mattering is always out at the edge of the distribution.

---

Appendix: raw data

Scope: our own 8B model; held-out drama, 25 hard cases; free generation identical to production, ranked blind by a stronger model.

General rule: always evaluate a compressed model at the same quantization and free generation as production, and make the worst-case rate, not the average, the primary KPI (same root as the evaluation principle in quantization eats into fine-tuning).