Some AI models are disqualified before you ever measure quality

In real-time subtitle translation, the smartest model is the first one out. Not because its quality is low. Because it “thinks” before it answers.

Model selection has a gate that comes before you measure quality — can you stop the reasoning, and can it always return in a fixed format? A model that doesn’t clear it is unusable no matter how smart it is.

And once I re-measured with the gate held constant, it was the received wisdom that fell apart. On short outputs, a small model is not fast. The differences in generation speed between models get swallowed by the network and the queue, and disappear.

When I was choosing the model to translate with, I naively assumed the job was “just pick the one with the highest quality.” Run the benchmarks, take the smartest model. It looks obvious.

But in real-time subtitle translation, that obvious thing didn’t hold. Before you ever measure quality, a lot of models are eliminated at the gate. And those gating criteria were completely invisible to me at the start.

The model was “thinking” before it translated

As I wrote in Part 1, I broke the latency down and found that one model was emitting more than 500 tokens of reasoning for a single line of translation (27x the tokens, 5x the cost, 4x the latency). A smart model “thinks” before it answers — it generates a huge chain-of-thought that no human ever sees. That helps on hard problems, but on a mechanical one-line translation, that reasoning is pure waste.

This is where the gating criteria came from

That one incident fixed the necessary condition for model selection. Before you measure quality, does it clear this?

Can you turn the reasoning off?

It looks like a trivial setting. It’s actually life or death.

• One model (the Qwen3 family) had a setting that turned reasoning off completely. A one-line translation comes back instantly, with zero reasoning.
• Another (the gpt-oss family) let you set reasoning to “low, medium, high” but couldn’t turn it off completely. As a result, the reasoning ate through the output budget, and generation itself failed on 26 of 100 cases. The quality — how good the content was — wasn’t bad at all, and yet it was structurally disqualified for real-time.

Smarts (quality) is a sufficient condition. Reasoning-off — and always being able to emit a fixed format — is a necessary condition. A model that doesn’t meet the necessary condition never gets into the ring, however smart it is. That only really landed for me once I measured three models in a production-equivalent setup.

* Real-time success rate = the share of requests that, sent under the same conditions as production, came back with a valid translation (non-empty, returned in the fixed format) inside the deadline. The "26% generation failures" are the cases where the reasoning ate through the output budget and the translation never came back in full.

”Fast means small” was an assumption too

Along the way, another obvious thing fell apart: “a lighter model has to be faster.”

Why was I on a small model in the first place? Speed and cost. But when I ran into a quality wall and tried twelve different variations of prompt and context, quality plateaued — the small model’s ceiling. So as the next move, for the first time, I measured the option of changing the model itself, under completely fair conditions (300 cases, identical prompt).

What surprised me was that the mid-size model was “faster” than the small one (253ms < 308ms).

The reason is in how the time a user waits gets built. Perceived wait time isn’t decided by generation alone.

E2E = Queue + Network + Inference

Of those, making the model smaller cuts only the last term. And on short outputs, that term is the smallest of the three.

So “small means fast” was a conclusion drawn from one term of E2E. On inference alone the gap between small and large is 1.27x, but in the actual wait — queue and distance included — it’s 1.02x. Making the model smaller barely moves what the user feels. This is the manifesto’s second principle — Hunt the Constraint: it can still be cheaper and faster, so re-measure the constraint against the whole production picture rather than an isolated benchmark — in its model-selection form.

The small model’s one and only advantage was being cheap (and that “cheapness” would later bare its teeth in a way I never saw coming — Part 4).

So which model did I choose?

It clears the gate, it beats the small model on quality by +14.7pt, and it wins on speed too. So should I go all the way to the large model (+18.0pt)? I chose the mid-size one. The large model is another +3.4pt on quality, but it costs more than twice as much as the mid-size (5.2x → 11.6x). For subtitle translation, those 3.4 points aren’t worth double the cost. The one point that satisfies the necessary condition (reasoning off) and strikes the best balance between quality and cost — that was the mid-size model. (This way of thinking — not picking a winner, but picking the point that fits you out of options that each give something up — comes back in Pareto.)

Lessons

1. Before quality, there’s a gate. “Can you turn the reasoning off?” and “can it always return in a fixed format?” — in a real-time application, a model that doesn’t clear these is disqualified however smart it is. Keep necessary conditions and sufficient conditions apart.
2. Smarts aren’t always a virtue. For a mechanical task, “excess smarts” is a pure liability in cost and latency.
3. Doubt “lighter means faster.” On short outputs the speed difference gets swallowed by the network and the queue and disappears, and the fastest model wasn’t the smallest one. And these gating criteria didn’t come from a spec sheet — they came from the logs. When the bottleneck moves, the received wisdom flips.

---

Appendix: raw data