When "which one is faster?" is the wrong question to begin with

Speed and accuracy, quality and cost — when objectives conflict, “which one is better” has no answer. There is only a frontier.

There are three things to do. Knock out the dominated options (points that lose to something else on every objective), then pick, from the frontier that’s left, the operating point your product needs. Not the aggregate score on a benchmark — decide where to sit on the axis that actually reaches users.

And this way of looking at things is also good for throwing improvements out. If a cheap parameter tweak beats an expensive improvement on every axis, that improvement added nothing.

“Which model is better?” “Which setting wins?” They look like natural questions. But when the objectives are in conflict, the question misses the point. Speed and accuracy, quality and cost — when taking one means losing the other, “which one is better” has no answer. There is only a frontier.

Once a Sweep over the range has given you a cloud of points, the next step is to read that cloud against several objectives at once. That’s where you need the Pareto view.

When there is no winner

Take the case of picking a model. A bigger model has higher quality, but the cost jumps to match. And speed? Let me line up the three models from Part 3 again, this time with cost alongside.

(Incidentally, Mid is faster than Small, the smallest of the three — 253ms < 308ms. “Bigger means slower” doesn’t hold for short outputs, exactly as we saw in Part 1.)

Plot quality against cost and the relationship between the three becomes clear.

Ask me which one is best and I can’t answer. Large wins on quality (+3.4pt), but Mid is faster than Large and half the cost. Mid is far higher quality than Small, but 5x the cost. None of them loses to any of the others across the board. All three are “on the frontier.”

A frontier is the set of points that don’t lose to any other point across the board. Conversely, options inside the frontier — something that’s slower than Mid, lower quality than Mid, and more expensive too — can be knocked out, because nobody has a reason to pick them.

The decision was never “pick the winner.” It was picking my own operating point — where to sit — on the frontier. I picked Mid (the best balance of quality, speed and cost). If quality comes first, Large; if cost comes first, Small. What the product needs is what decides where you sit.

Sometimes you pick the operating point by the “floor,” not the “ceiling”

Another example of conflicting objectives. There are two ways to compress (quantize) a model’s weights. Across 25 hard cases, I counted, for each method, how many times it produced the best translation and how many times it produced the worst (the rest landed in the middle).

• Method A: top rank 6 times / bottom rank 16 times (when it lands it’s the best, when it misses it’s the worst — it swings)
• Method B: top rank 3 times / bottom rank 6 times (never wins big, but never falls apart either)

“Which one is better” — here too, the answer depends on the product. A has the higher ceiling; B has the higher floor. In subtitle translation, a single critical breakdown destroys immersion. So I gave up the ceiling and picked the floor (Method B). For this product, avoiding the worst case is worth more than reaching the best translation.

Ceiling or floor — that is a choice of operating point on the frontier, and the question “which one is superior” has no meaning to begin with.

(The sweet spot from last time — resolution 736, “accuracy −1.3pt, half the latency” — is also a single point on the accuracy × speed frontier. The Sweep produces the cloud of points; Pareto reads its frontier. Sweeping and reading the frontier are two sides of the same coin.)

The frontier exposes fake improvements

Pareto has one more use: killing a claim.

At one point, a retrained model looked like it had “improved.” But when I lined it up against a control — the old model with a single threshold changed — that control beat the retrained version on every axis (it Pareto-dominated it).

Here’s what that means. If an option you can produce with nothing but a parameter tweak beats your expensive change across the board, that change added nothing. What looked like an improvement was the effect of some other factor entirely — a confound — that had nothing to do with the change. This is how a frontier rejects false progress.

Lessons

1. When objectives conflict, “which one is better” is the wrong question. There is no winner. There is only a frontier. Start by knocking out the points that lose to something else on every objective (the dominated options).
2. Pick the operating point by what the product needs. Ceiling or floor, quality or cost. Decide where to sit on the axis that actually reaches users, not on a benchmark’s aggregate score.
3. If a cheap control Pareto-dominates it, the improvement is nothing. If a mere parameter tweak beats your expensive change across the board, the change added nothing.

The only reason I can do this — sweep a surface and read its frontier — is that the harness made multi-axis measurement cheap. Back when measuring a single point was all I could manage, drawing a frontier never even occurred to me.

---

Appendix: raw data