I moved one stateful component and perceived latency went from 469ms to 252ms

Perceived latency 469ms → 252ms (−217ms). Not one line of the actual work got faster. I just moved a stateful component to the right place on the map.

I’d brought inference close to the user, and yet what people perceived barely shrank. Break it down and the round trip really had disappeared — while a different segment I hadn’t touched at all had jumped from 20ms to 175ms. The culprit wasn’t inference (135ms); it was the physical location of a single health switch wedged in between — I had moved the processing to Tokyo and left the state it touches behind in the US.

A stateful component has an address. In a distributed system, local optimization gets cancelled out somewhere else.

In the deep dive “Geography is latency’s floor, and you can’t move it” I wrote that the biggest lever on latency isn’t performance, it’s where you put things. So the obvious next move was to bring the inference server, and the routing layer in front of it, into a region close to the user.

I did. One of the segments got faster, exactly as intended. And yet the total time the user perceives didn’t shrink the way I expected. On the segment ledger I should have banked over 200ms, but most of it had vanished somewhere.

I’d put it close. Something was still going halfway around the world.

Let me lay out the setup first

To run my own inference (self-hosting), I keep one thin routing layer between the device and the GPU: a small function running at the edge (Cloudflare Workers). It has two jobs — relay the request to the inference server (the pod), and watch whether inference is down and, if it is, immediately divert to a third-party API (a circuit breaker — in plain terms, an “is it still up?” health switch).

That “is it up?” state has to be remembered somewhere. The edge is spread across points of presence all over the world (data centers everywhere) and has no particular location of its own, so the state alone goes into a special component that is pinned to a single place. Cloudflare calls it a Durable Object (DO from here on). Just as the name says, it’s one durable entity, and it stays in the region where it was first created, forever after. That’s the setup for everything that follows.

At first the whole thing lived in the US. The edge function, the DO, the pod (at that point still a rented GPU in the US) — all of it in the US. They were neighbors, so nothing looked wrong.

I executed the “move it closer” plan

I moved the pod to Asia (a 4090 in Taiwan) and configured the edge function to run in Asia-Pacific (APAC) as well. I measured the round trip from the edge function to the pod (upstream), and it was exactly what I’d aimed for.

Round trip 290ms down to 65ms. The fallback rate (how often inference misses the deadline and the request escapes to the third-party API) went from 25% to 0%. Looking at the segment alone, an unambiguous win.

(An aside: there’s a lesson from “measure your premises and kill them” buried in here too. The edge dashboard displayed the entry point of presence as “Tokyo (NRT — Tokyo’s airport code),” but that was where the request was received, not where the function ran. The 290ms round trip was 2.5x the 115ms I got hitting it directly from my own machine — meaning that behind the Tokyo signage, the function was in fact running in the US. “Where it was received” and “where it ran” are two different things.)

And yet the whole thing doesn’t shrink

The segment got faster. But the time until a translation comes back, measured on the device (the wall-clock time from when the app asks for a translation to when it receives the result, timed on the device side), didn’t drop as much as I’d expected. I’ll call the median of that p50 (the value half the requests beat). So, my usual reflex — I broke the whole thing down into segments.

The breakdown surfaced a segment I hadn’t expected. Before dispatching the request to the pod, the edge function was sitting through a wait. Measured inside the edge function, this “time before forwarding even starts” — time spent without having sent a single character to the pod — was consistently about 175ms. Back when the function ran in the US, it had been a mere ~20ms. Bringing the pod closer and moving the function to Tokyo — that one same move had made this wait, and only this wait, jump from ~20ms → ~175ms.

(That 175ms is a wait measured strictly inside the edge function. The time perceived on the device carries a device↔edge round trip on top of it. So it isn’t a simple subtraction where “device total − function→pod” = 175. I’ll close the segment ledger at the end.)

Pinning down the culprit took just one more level of breakdown. Before dispatching the request to the pod, the edge function checks that “is it up?” switch once. And what it checks is the DO, pinned to a single place — the DO that had been created in the US and had been sitting in the US ever since.

In other words, exactly what Figure 1 up top shows. I’d moved the pod close. But the move right before that — the “is it up?” check — was quietly making a Pacific round trip every single time. When the function was in the US, the function and the DO were neighbors in the US (the check took ~20ms). The moment I moved the function alone to Tokyo, that one move turned into a Tokyo↔US round trip. The round trip itself is ~165ms, plus the remainder of the check itself (~10ms), which is how the pre-dispatch wait landed at ~175ms (~20ms → ~175ms).

Most of what I’d gained by optimizing geo was being cancelled out by this invisible Pacific round trip. A component of a system I thought I’d placed nearby was going halfway around the world on every request.

The fix wasn’t to eliminate the culprit

What’s interesting is that this was never an “inference is slow” problem at all. Inference (int8 quantized, 135ms) was innocent from beginning to end. The culprit was the physical whereabouts of a single health switch that had nothing to do with inference.

The first idea that came to mind was “fire the check in parallel with forwarding the request (don’t wait on it).” I rejected it. Parallelizing doesn’t remove the DO round trip I actually wanted gone — it just runs speculatively and ends up being the rate limiter anyway. And letting the fallback decision run ahead of itself risks hammering a dead pod with requests while inference is down. Local optimization opens a different failure mode.

What I did instead was the plainer move — put the DO back down in the same Asia-Pacific region as the edge function. A stateful component is pinned to where it was created, so I recreated it under a “new generation” name, moving it, creation site and all, to APAC.

Here’s the result.

Pre-dispatch wait 175ms → 12ms. The p50 perceived on the device: 469ms → 252ms (−217ms). And the region around the median is extremely steady — the spread from p10 to p50 is a mere 21ms, exactly the kind of stability you want for subtitles, where everything depends on them continuing to arrive at a constant rhythm (the right-hand tail does stretch — p90 460ms, p99 977ms — but what counts is that the middle, the core of what gets perceived, doesn’t move).

−217ms. That was a bigger cut than any model optimization I’d pulled off up to that point. Not a single line inside got faster. I just moved one stateful component to the right place on the map.

Local optimization bares its teeth in a different segment

At bottom this failure is the nastiest form of the local-optimization trap I wrote about in Part 2, “Last year’s right answer becomes this year’s mistake”. That time it was “I eliminated the round trip and then generation couldn’t make the deadline” — a trade-off along the same path. This one is worse: the single move of bringing the pod closer had pushed a completely different segment (the state check) further away. A place I hadn’t even touched had gotten worse.

In a distributed system, when you move where something executes, the physical whereabouts of every state dependency that function touches move with it — or, by not moving, get left behind. Look at one segment alone, cheer that “it got faster,” and you’ll never notice it being quietly cancelled out in another.

And the pattern doesn’t end here. There are other stateful components — the cache, the rate-limit counters, they all have addresses in the same way. This time one health switch was all it was, but the next time I move a workload the same audit is required. The only reason I caught it is this: I was breaking the whole thing (end-to-end) into segments and measuring. Had I been satisfied with “function→pod got faster,” I’d still be carrying a switch that round-trips the Pacific, scratching my head at “why doesn’t the perceived latency come down?”

Lessons

1. Local optimization gets cancelled out in another segment. A move that brings one component closer can push a different component you never touched further away. Judge wins and losses end-to-end (E2E), not segment by segment.

2. A stateful component has an address. When you move a workload, move the state it touches along with it. Otherwise the state alone gets stranded on the old continent. And while you’re at it: where a request was received is not where it runs — verify with a measured round-trip time, not the signage.

3. The culprit is usually not the component that looks most suspicious. The slowness wasn’t inference (135ms); it was the physical location of one health switch that looked entirely unrelated. Before you glare at the “heavy-looking” work, break it down first. The numbers will point at the culprit.

This is the manifesto’s second principle — “It can still be cheaper and faster”: re-measure the constraint on the full production picture, not on an isolated benchmark — in its most distributed-systems form. The biggest −217ms wasn’t inside the model, and it wasn’t inside the segment I’d sped up. It was at a single point on the map that nobody was looking at.

---

Appendix: raw data

All measured p50, n=25.

Related: the motivation for proximity is in Part 5; the measured floor that distance imposes is in “Geography is a floor you can’t move”.