A duplicated benchmark lies to you quietly

A benchmark duplicated by imitating only the main logic kept measuring a setting production had dropped long ago. Rewriting it to call the production functions directly recovered the numbers by +26 to +31pp.

And when, before any A/B, I checked whether “the same code run twice agrees,” I finally saw what the notorious “±20pp of noise” actually was. Not random noise: nondeterminism that had leaked into the A/B delta — a confound.

That’s what’s frightening about measure-first: the faster you can measure, the faster you can become confident about the wrong thing.

When you develop measure-first, at some point you stop questioning the instrument itself — the harness, the scaffolding you measure on. Numbers come out. On the strength of those numbers you adopt one option and drop another. But — what if the numbers were lying from the start?

For the past six months I’ve made the harness my default mode of development. Measuring got fast and cheap, and the excuses for not measuring disappeared. But once you can measure fast, a different pit opens up: measuring the wrong thing, fast, and with confidence.

This article is the story of falling into that pit twice. The first time: a duplicated bench was measuring something other than production. The second time: running the same code twice gave me numbers that didn’t match. Both times, the numbers that came out looked perfectly plausible.

The first time: a duplicate drifts, quietly

The harness in question measured OCR region tracking for Live translation (the feature that translates video subtitles in real time). How much of the subtitles it picks up without misses — completeness.

This harness shared only the main computation from production’s tracking logic and hand-duplicated everything around it (when to discard a tracking box, full-width recovery, the horizontal envelope, box lifetime). “The core is shared, so it must be measuring the same thing production does” — that’s what I thought.

That was an assumption.

On 2026-07-04, production had judged an optimization called “horizontal grid narrowing” to be net negative and dropped it. But the hand-written part of the duplicated harness kept measuring that supposedly discarded narrowing. Something production no longer did, the bench had been scoring all along. From the moment I duplicated it, the two had been branching apart quietly — drifting.

I dropped the duplicate and rewrote it to pass production’s own tracking-state object straight through and call production’s functions. Completeness (the share of subtitles picked up) recovered like this (pp = percentage points, throughout).

Nearly thirty points. Production actually delivers 82.6% / 81.2%; the duplicate scored it as 56.5% / 50.0%. Had I trusted this duplicated harness’s numbers to judge whether a change to the tracking logic was good or bad, I would have been optimizing a world that doesn’t exist in production.

The lesson is simple. A harness should instantiate production’s components and call them directly. Don’t duplicate the logic. The moment you duplicate, it becomes a different thing that measures “the duplicate’s performance,” not “production’s performance.” If production code is structured so that it’s hard to call, don’t route around it with a duplicate — refactor production so that it can be called.

The parts that are hard to call — work you can’t lift out because it depends on the screen — get extracted from production as pure functions, and then you stop drift with the type system. Share production’s state object whole, and make the computation function demand, as an argument, “the token returned by the setup step at the top of the cycle”; then a duplicate that skips the setup won’t even compile. “Being careful” is less reliable than “making it not compile.” The extraction procedure itself (pin the current behavior with a golden characterization test, then do a move-only extraction) is in the appendix.

The second time: same code, numbers that don’t match

With it rewritten to call production, the bench was trustworthy now — or so I thought, and I ran an A/B (a comparison of two options) on the tracking logic. A difference showed up. I got the urge to explain the mechanism behind it.

But there was one thing to check first. Run the same code, on the same data, twice — do you get the same numbers?

I didn’t.

The cause was the harness’s clock. Into the calculation that decides when the next piece of work can start, I was feeding the measured OCR duration. But with the device’s CPU governor and thermals, that duration wobbles by tens of ms. That changes how many cycles fit into one video, and everything downstream — display success rate, coverage, when a subtitle appears — shifts with it. Here is what two runs of the same code on the same data actually measured.

First, just running the same code twice made the intermediate quantities themselves wobble.

This run-to-run wobble on its own is still only “in-house error from measuring the same option twice.” The problem came when that wobble got mixed into the A/B delta. Comparing the same two options (old vs new) on different recordings, even the sign of the A/B delta in display success rate flipped.

The same two options, and yet on one recording “the new option is 6.7pt worse,” on the other “the new option is 12.5pt better.” That was the state the bench was in while I ran A/Bs and said “there’s a difference, and here’s the mechanism.”

Then it clicked. The thing I had long treated with half-resignation as “this harness has ±20pp of noise” — the source of that spread was exactly what I had just seen: the A/B delta scattering from −6.7pp to +12.5pp (about 19pp wide) across recordings. And it isn’t random noise. It’s the nondeterminism created by feeding measured values into the clock — run-to-run wobble — leaking into the A/B delta. The nondeterminism by itself is just wobble within each run. But the moment it mixes into the A/B delta, it decides the sign of the difference. Only then does it earn the name confounding (when the effect you want to compare and the wobble you don’t can’t be separated).

Kill the source rather than measure the noise

The fix was paradoxical. Rather than measuring the noise over and over and comparing it against the delta, eliminating the source of the noise was faster and surer.

I stopped letting wall-clock time into the simulated timeline and used a fixed value instead (the default is roughly the measured median). Measured latency is recorded for measurement purposes only; it never enters the timeline. Runs where I actually want to measure latency are kept separate.

That gave me a deterministic mode. Same input, same cycle sequence, reproduced bit for bit. Re-running that A/B in deterministic mode, 282 measurements matched and the difference I had been calling a “mechanism” vanished. A hypothesis I had talked myself into by reading the code was refuted by a deterministic A/B without ceremony.

Here is the counterintuitive conclusion. “One deterministic run” is a stronger claim than “three runs and statistics.” If two runs agree, the third will agree too. Rather than averaging noise away with statistics, build scaffolding on which the noise doesn’t exist in the first place — that’s what lets you say a difference is a difference.

And the trick to verifying determinism is not that the aggregates match but that the cycle sequence itself matches. Aggregates can line up by accident while the path through them differs. Conversely — and this is the beautiful part — the measured latency is allowed to change. It changes and the result doesn’t move: that is precisely the evidence that the confound has been separated out.

Lessons

1. A bench should call production directly. A duplicate will always drift. A duplicate that shares only the main logic keeps measuring settings production dropped long ago. If it’s hard to call, refactor production and stop drift with the type system (skip the setup and it shouldn’t compile). Just rewriting it to call production recovered completeness by +26 to +31pp.

2. Before any A/B, run the same code twice and check that it agrees. If it doesn’t, every delta you ever measured with that harness is suspect. The notorious “±20pp of noise” is usually nondeterminism (run-to-run wobble) confounded into the A/B delta, and then misnamed as random noise. Nondeterminism and confounding are different things — wobble only becomes a confound once it leaks into the A/B difference.

3. Killing the source is faster than measuring the noise. And once you fix the instrument, every past accept and reject goes back onto the re-audit pile. Don’t dilute it with statistics; remove the nondeterminism itself (“one deterministic run” beats “statistics over three”). But the moment you fix it, every option you rejected and every option you accepted with that harness goes back up for re-evaluation — estimate that rework too when you decide the order in which to fix things.

That’s what’s frightening about measure-first: the faster you can measure, the faster you can become confident about the wrong thing. So before you measure the subject, doubt the instrument once. Does the same input return the same number — that obvious little check is, in a world where last year’s right answer becomes this year’s wrong one, the only ground you have for believing the numbers at all.

---

Appendix: raw data