Don't let a benchmark decide what "fast enough" means

“Under 500ms is fast.” Where did that number come from? Usually, from nothing in particular. And a target that came from nothing in particular always misses.

The deadline for subtitle translation wasn’t set by a benchmark. It was set by how long the subtitle stays on screen. Work it backwards and it moves with the content — 400 to 750 ms — so a single number misses on both sides.

On top of that, speed isn’t a continuous value. It’s a cliff. Inside the deadline a difference in speed buys you nothing, and the moment you cross it the value goes to zero. So the first thing to settle wasn’t “how many ms should we aim for” but “where is the cliff.”

How do you decide whether your AI is fast enough?

Most teams put down a number that feels about right. “Under 500ms is fast.” “Under a second is good enough.” But where did that 500ms, that one second, come from? Usually, from nothing in particular. And a target that came from nothing in particular always misses.

Building subtitle translation forced me to change how I set that target, from the ground up.

The target wasn’t mine to set

In real-time subtitle translation, it doesn’t matter how good the translation is: if it arrives late, it’s worth zero. The subtitle flips to the next line right away. If the translation doesn’t make it in time, the user is already watching the next scene.

So the bar for “fast enough” was never mine to set. How long the subtitle stays on screen — that’s what sets the deadline. Land inside that window and the translation has value; miss by even 1ms and it has none.

So I decided to work the target backwards. First I measured, per content type, how many milliseconds a subtitle actually stays on screen (dwell time).

(The rightmost column, deadline compliance, strips out failure causes other than the translation itself and looks only at timing. The tighter the deadline, the lower it goes — variety shows worst of all. I look at p10 rather than the median because the subtitles that vanish fastest are exactly the ones I need to hit.)

Plot it and you see it at a glance. The dashed line (the current pipeline, ~500ms) sits inside the p10 for drama and games, but it overshoots variety’s p10 (400ms) — variety’s fast subtitles aren’t making it in time.

Now the target existed as a number. Subtitles flip fastest in variety shows, where the fastest 10% are gone in 400ms. Detection, OCR, translation and display all have to fit inside that 400ms, or you don’t make it. So the cliff is exactly that: a deadline worked backwards from subtitle dwell time. About 400ms for variety, the tightest case (it stretches to around 550ms when the content is gentler). And once you subtract detection, OCR and display from the inside of that deadline, the budget left for translation itself is about 180–300ms. The deadline (the cliff) is 400ms; the translation budget inside it is 180–300ms — these are two different numbers. (How the physical distance to the server eats that translation budget alive is the next deep dive.)

Speed is a gentle slope with a vertical cliff at the end

This was the most important thing I found. Latency tends to be treated as a continuous value that acts smoothly — the faster you go, the slightly better things get. That’s half right. The other half is missing.

Plot the UX value of subtitle translation as a function of time-to-display and you get this shape.

• Up to the deadline it’s a gentle slope. The faster you deliver, the slightly better the UX gets (the translation reaches the eye sooner). But the payoff is mild, and it diminishes.
• And the moment you go over the deadline — the cliff — by even 1ms, the translation washes into the next scene and the value drops vertically to zero.

Most people picture only the gentle slope. So they invest uniformly, anywhere along it, on the theory that a bit more speed buys a bit more quality. But there’s a cliff at the end of that slope. Which is why the biggest win isn’t shaving time somewhere along the slope — it’s pulling the subtitles that are over the cliff back inside it. That’s where the value jumps up from zero all at once.

And where the cliff sits changes with the content. Variety’s cliff is tight (~400ms), drama’s is loose (~750ms). The same “500ms” target is too lax for variety and too strict for drama. A single benchmark number misses both.

What a benchmark target would have gotten wrong

If I’d set the benchmark target “under 500ms is fast,” here’s what would have happened.

• Drama (cliff at 750ms): already comfortably in time, yet I’d keep pouring inference optimization into “getting under 500ms” — speeding things up pointlessly, inside the cliff.
• Variety (cliff at 400ms): 500ms is already over the cliff. I’d believe I’d hit the target and keep dropping the subtitles that never made it — the illusion of having hit it.

A cliff worked backwards from UX prevents both. It tells you, correctly and per content type, how much faster is still worth it, and when you’re already done.

(This cliff is the same thing I called the perception wall in Part 5 and elsewhere. When self-hosting eliminated the round trip and the queue, E2E landed inside this cliff — and only then could I say it was making it in time.)

Lessons

1. Don’t let a benchmark set the bar for “fast enough.” Ask where the number came from. Usually, from nothing in particular. Work the target backwards from the user’s situation.
2. Treat latency as a cliff, not a continuous value. Inside the cliff a difference in speed is meaningless; over it, the value is zero. Work out first where “faster” actually pays off.
3. The cliff moves with the content. A single benchmark number misses on the fast side and on the slow side alike. Only a cliff you worked backwards points correctly at where to invest.

This cliff is the manifesto’s third principle — Optimize the Right Proxy for UX — reduced to a single number. Before you chase a benchmark figure, measure the deadline your users are imposing on you.

---

Appendix: raw data