Geography is the latency floor you can't move

Make the model faster, quantize it — distance is the one thing that never shrinks. When I measured round-trip times from Tokyo to points around the world, they lined up from 5ms in Tokyo to 373ms in Cape Town: a spread of more than 70x.

This is a floor set by physics rather than performance, and no amount of code moves it. Worse, the 166ms round trip to the US East Coast eats 55–92% of the budget translation is allowed (~180–300ms) — a remote server misses the deadline even with generation at 0ms.

So the biggest lever wasn’t performance, it was placement. Just moving the inference server from the US to the Tokyo area erased ~160ms — more than any model optimization I had done.

In Part 2 I wrote that the dominant term in the slowness had moved to “distance” (a 275ms round trip). But distance is a different kind of thing from queueing or generation. No amount of polishing the code shrinks it. Packets have to physically travel.

So how big is that floor, exactly? I measured it.

Round-trip times to the rest of the world

The cleanest way to measure this is to take fixed points spread evenly around the globe as your reference. Cloud provider regions (AWS here) are exactly that — scattered across the world’s major cities at known locations. Here is the raw round-trip time (RTT) from Tokyo to each region.

Inside Tokyo, 5ms. But cross the Pacific to the US East Coast and it’s 166ms; Europe is over 220ms; and South Africa (Cape Town), on the far side of the planet, is 373ms — more than 70x local Tokyo. None of this has anything to do with server performance. It’s the time a packet takes to physically make the round trip through optical fiber.

(The “275ms of distance” I quoted in Part 2 was the effective value on the production path: this raw 166ms of distance, plus the extra round trip through a proxy layer sitting in between. Pull out pure geographic distance on its own and you get 166ms.)

This is a floor you can’t get under

Here’s the heart of it. That 166ms is a floor no optimization can get under.

• Make the model faster: you can’t get under it.
• Quantize it, put it on a faster GPU: you can’t get under it.
• Take generation to 0ms: the 166ms is still there.

As long as the server is far away, you’ve already spent 166ms before translation even starts. For the network portion, where you put it matters more than how much performance you add.

And this floor is stable. Keep measuring the Tokyo↔US round trip and it stays around 166ms (in the 170s under continuous measurement) — the transpacific hop itself barely wobbles. The jitter I used to file under “the network is unstable” turned out to be the proxy layer in between, not geography. Distance is predictable, and it doesn’t move. Which is exactly why optimization can’t make it go away.

A distant server can’t get inside the cliff

Think back to the deep dive on “the cliff” — the deadline by which a subtitle still lands in time. Once OCR is done, the budget available for translation was roughly 180–300ms.

Now drop the 166ms US East Coast round trip into that. Against a translation budget of 180–300ms, the round trip alone burns 55–92% of it before a single character has been translated. Which means a distant server can’t get inside the cliff, no matter how fast the model is. It’s physically impossible.

Put the other way round, there is exactly one way to lower the floor: put the server closer. When I actually moved the inference server from the US to the Tokyo area, I cut about 160ms. That was bigger than any model optimization I had done up to that point.

Lessons

1. Geographic latency is a floor set by physics. Tokyo 5ms versus Cape Town 373ms — a spread of more than 70x that has nothing to do with performance, and doesn’t wobble. It’s predictable, and in exchange, no optimization makes it go away.
2. When the floor eats the budget, stop optimizing the model. A server whose round trip alone already goes over the cliff won’t make it even with generation at 0ms. The only move left is where you put it.
3. Placement beats performance. A decent server placed close is faster than a fast server placed far away. That’s what makes geography the biggest lever on latency.

This is the manifesto’s second principle — “it can still be cheaper and faster” — in its most physical form. The biggest win wasn’t inside the model; it was on the map.

Appendix: raw data