Last year's right answer becomes this year's wrong one

A few months later I re-ran the same measurement, and the breakdown had flipped. Generation was 10% of the total. The remaining 90% was queueing and distance.

The lever that had worked best last time was now beside the point. The bottleneck moves — right answers have a shelf life.

So I tried running inference closer to the user (edge inference) to eliminate the round trip, and this time the whole thing got 3x slower. Local optimization loses on the whole.

Last time I wrote that the cause of the slowness was the model’s generation — more precisely, that the model was “thinking” before it answered. I turned the reasoning off (reasoning=none), and generation got dramatically lighter all at once. At the time, that was enough.

The problem is that right answers have a shelf life.

A habit makes you doubt your own old conclusions

Once you learn to “break it down and measure,” it becomes a reflex. While working on something else, I ran the same breakdown measurement again. There was no deep intent behind it — I just wanted to see where things stood now.

The numbers that came back stopped me in my tracks.

Generation, which used to dominate the time, had shrunk to 10%. And since two devices, one new and one old (38 samples each), agreed almost exactly, this isn’t measurement noise — it’s structure.

The reason was clear right away. Last time I stopped the model from “thinking,” and after that I kept narrowing the output, so generation itself had gotten lighter. Exactly to the extent that generation got lighter, “waiting” and “distance” rose to the surface in relative terms. The dominant term had changed hands.

A queue is not something you talk about in averages

That 167ms is only the median of the queue. And a queue turned out to be the textbook case of something you must not describe with an average. What drove that home wasn’t the current model but a different model — the one that had been the production candidate before it.

That model was a huge model running on shared infrastructure. Measuring the same queue under congestion — even the median was an order of magnitude away.

This model was already at about a second at the median — roughly 6x the current model’s 167ms. And one time in a hundred, you wait about five seconds. The awkward part is that you can’t predict when it happens: 150ms in a quiet window, this distribution when it’s busy. Not “always fast,” not “always slow” — unpredictable.

If you only look at the average, this tail is completely invisible. But what users remember isn’t the average — it’s the one time it was worst. That is exactly the manifesto’s third principle: a good proxy metric looks at the worst single case (the tail), not at the average. And as long as I’m on someone else’s shared infrastructure, this tail can’t be eliminated — I don’t get to choose who I’m queued behind. (Take dedicated capacity and the whole tail goes with it. That’s Part 5.)

Break the 275ms of network down further and: the connection was being reused out of a connection pool, so establishment cost was near zero, and the response was small enough that the transfer was instant. Which means the 275ms was almost pure geographic round trip — the physical round-trip time out to an inference server far from the device (in the US at the time). (For the details, measured across AWS regions worldwide, see the deep dive “Geography is a latency floor you can’t move”.)

So the lever to pull changes too

Last time’s right answer, “don’t let the model think,” has nothing left to give. Whatever I do to a generation step that’s only 10%, perceived speed barely moves. What dominates now is:

• Distance (275ms round trip) → put inference near the user and it goes away. That 275ms is a physical “floor” you can’t move
• queue (167ms) → the queue on shared infrastructure. Take dedicated capacity and it goes away

In other words the lever had moved away from “make the model faster” and away from “trim the output,” to holding the inference server physically near, and dedicated.

Testing the hypothesis (and losing again)

If “kill the round trip and it gets faster,” then run inference at an edge near the user. Conveniently, one edge inference service hosted exactly the same model. I was already routing through that edge platform, so I could compare by swapping nothing but the endpoint. On paper it was perfect.

Here’s what happened.

A clear-cut loss. I eliminated a full 250ms of round trip and the whole thing got slower.

The cause was obvious immediately, thanks to the breakdown table. Near or not, generation on the edge’s commodity GPU couldn’t keep up. Even pure post-warm-up generation was slower than the current setup (about 1.5x, as far as I could measure it), and in real operation cold starts and waiting piled on top, so the translation was cut off at the time limit (1500ms) before it could finish — hence no subtitles. The “1500ms” in the table isn’t generation speed; it’s the client-side time limit itself. However much round trip you eliminate, if generation goes past the deadline you lose overall. Generation, which was supposed to have shrunk to 10%, decided the outcome the moment I put it on slow hardware.

The only way out is near and fast, together

This is where the other lesson from last time starts to pay off. Local optimization is a trap. Eliminating the round trip alone is meaningless if generation is slow. And fast generation alone is meaningless if it’s far away.

So the requirement becomes proximity and fast inference, both at once. My estimate said that if I could put a purpose-built fast chip in a region near the user (in-country, say), the round trip would be a dozen-odd milliseconds and the total would fall to around 100ms. That is the real starting point for me looking at running inference myself (self-hosting). I didn’t want a faster model. I wanted a place that was near and fast.

(So why didn’t I self-host from the start? That’s a separate story, about the supply side, continued in Part 4.)

Lessons

1. The bottleneck moves, so keep measuring. A right answer you got from one breakdown expires when the situation changes. I noticed the mismatch only because measuring had become a habit; if I’d shelved last time’s conclusion as “something I already know,” I’d still be pulling a lever that does nothing.
2. Think in relative terms. Generation didn’t get “lighter” — everything else got “heavier,” relatively. The dominant term changes hands by share, not by absolute value.
3. Local optimization is a trap. Eliminating the round trip backfires if generation is slow. You only win end-to-end, by satisfying several segments at the same time.

---

Appendix: raw data