One setting took output tokens to 1/27, latency to 1/4, and cost to 1/5

For a single line of translation, the model was spending hundreds of tokens. Not on the translation. On “thinking” before it answered.

The slowness wasn’t the model’s performance — it was the model thinking before it answered. The lever that worked wasn’t “switch to a faster model,” it was “make the same model stop thinking.” 27x the tokens and 4x the latency went back to 1x.

And once the premise changed, so did the whole landscape of options. With reasoning off and the conditions held constant, the mid-size model was faster than the small one — and better.

This isn’t a story about being good at measurement

To be honest, I wasn’t measuring cleverly from the start. For a long time I assumed the reason translation was slow was “the model.” When it felt slow, the only ideas that came to mind were “a faster model” and “a bigger model.”

The API I was using at the time returned only the translation, so I had no visibility into where the time was going. And because I couldn’t see it, all I could do was pin the blame on “the model.”

The turning point was switching translation providers. The responses from the new one carried an unfamiliar usage breakdown — wait time, input processing, and the time and token count for generation — right there in the payload. “Hold on, it’s telling me exactly where the time goes.” From then on I stopped looking at “slow” as a single lump and got into the habit of logging the breakdown and reading it.

I didn’t start out with a measurement mindset. A number in a log happened to catch my eye, and it became a habit from there. That order is what pays off later.

A single line of translation was generating hundreds of tokens

The breakdown showed most of the time going to “generation.” No surprise there. But the token count was off.

A translation’s output should be a dozen or two tokens at most. And yet the model was generating hundreds of tokens per translation. I looked inside, and there was the reason. Before translating, the model was “thinking” — pouring out a chain-of-thought (“the subject of this sentence is… the preceding context is… so the translation should be…”) on and on, ahead of the answer.

The smarter the model, the more it thinks before answering. On hard problems that helps. But translation — at least in this use case — is a mechanical task you can answer by reflex. And the model was sitting there deliberating over it. Here’s the breakdown of what the model generated in one translation (generation time is roughly proportional to token count).

96% of the generation was “thinking,” not translating. The result, compared with reasoning off:

• Token volume about 27x
• Cost about 5x
• Latency about 4x

(An LLM API bills by tokens processed, and generation time is roughly proportional to token count too. So these “thinking” tokens, not one of which ends up in the translation, turn straight into cost and waiting. And the reason 27x the output tokens only lands at 4x total latency and 5x cost is that each has a “fixed cost” riding on it — the network round trip for latency, the input prompt for cost. Only the generation part grows, so 27x doesn’t come through as 27x.)

Generation was heavy, but not because translation is heavy. The model was spending its time thinking, not translating.

The lever wasn’t “change the model,” it was “make it stop thinking”

What this told me was that the lever that works isn’t “switch to a faster model.” The move was — make the model stop thinking.

Luckily, that model had a setting to turn reasoning off completely (reasoning=none). On: 500 tokens of reasoning. Off: a dozen-odd tokens of translation, immediately. Measured with it off, one translation came in at a median of 253ms and 18 output tokens (N=300) — with reasoning on it had been taking about 4x that. 27x the tokens went back to 1x, and latency and cost came down with it. I never changed the model. I just made the same model stop thinking.

A mechanical task like translation doesn’t need the model’s “smarts” (= its ability to reason). If anything they’re a pure liability. Out of this came a principle: for real-time use, use a model that answers by reflex (reasoning=none).

(That said, some models can’t turn reasoning off completely. Those models, however smart, are unusable in real time — more on that in Part 3.)

”Smart means slow” fell apart

Once “reasoning=none” was the premise, the model-selection landscape changed. Re-measuring several models fairly, with reasoning held off across the board, the surprise was that the mid-size model with reasoning off was faster than the small one (median generation 253ms < 308ms). And its quality was far higher (pass rate +14.7pt).

With short outputs, the difference in generation speed between a small model and a big one gets swallowed by the network round trip and disappears. So as long as you cut the reasoning, “smart means slow” falls apart. You can use a smarter model without paying for it in speed — reasoning=none is what opened that door. (The full picture of this comparison, and the model-selection story that starts from it, is in Part 3.)

Postscript — the bottleneck moved again

Once reasoning was off and generation had gotten light, I ran the measurement again — by then a habit. The breakdown had shifted further. Generation, which used to dominate the time, was now just 10% of the total. The dominant term had moved to the network round trip and the queue.

Knock out a bottleneck and it moves. So don’t measure once and be satisfied — keep measuring. (Continued in Part 2.)

Lessons

1. Look at what’s inside the generation. When generation is heavy, whether that’s “the translation” or “the model’s thinking” is an entirely different problem. Suspect the contents of the tokens: is the model deliberating over a mechanical task?
2. “Change the model” isn’t the only lever. “Make it stop thinking” took 27x back to 1x without changing the model — and once the premise changed, “smart means slow” fell apart too, and a better model became usable at no cost in speed.
3. You can’t question what you can’t see. I only noticed because the log showed me the token breakdown. Hundreds of tokens for a single line of translation — that anomaly happened to catch my eye.

---

Appendix: raw data