You are the bottleneck

In four months I built 122 measurement harnesses. Not for rigor. For speed.

What measure-first is really about isn’t accuracy but cycle time, and the first bottleneck to knock out was — me.

Determinism (the same input always producing the same number) isn’t the goal; it’s the precondition for taking the human out of the loop. Once iteration is cheap, instead of picking one point on instinct you can run a dozen-plus and see the shape. And the more you work with AI, the more the human left in the verification loop becomes the rate limiter.

Measure-first was never really about “accuracy”

Say “let’s measure properly” and people usually assume you mean rigor. Remove the bias, take a control, look for a statistically significant difference — those matter too. But the real reason I built 122 measurement harnesses in four months wasn’t rigor. It was speed. And the bottleneck I wanted to speed up was — me.

Let me take it in order. I didn’t see this from the start.

Origin 1: break it down and measure

Half of what set this off was something I wrote earlier: how turning reasoning off cut output tokens to 1/27. Looking at the generation-time field in the provider’s logs, I stopped talking about “slow” as one lump and started breaking it into segments and measuring them. That was the seed of the “measure the right thing” mindset.

But that was still just reading the logs with a sharp eye. It wasn’t a system.

Origin 2: the first “harness” came out of determinism

The first time I used the word “harness” was about six weeks later. The motive was different — a much more concrete pain.

I was optimizing the OCR for Live translation (the feature that translates video subtitles in real time), and when I measured on a real device the numbers thrashed around depending on the scene. I ran one candidate improvement twice and got −3% and −10.7%. To compare v3 against v4 I ran five minutes of Live three times (baseline / v3 / v4), but I couldn’t capture the same scene twice — the number of regions OCR ran on differed every time, so the denominator wobbled. Was “skip rate up +8.6pt” a real improvement, or had the scene just changed? Once the frames change, you can’t separate the two.

The one line I wrote at the time was, in hindsight, the birth of the harness idea.

I wasn’t doing the obvious hypothesis test: “if the same frame sequence run twice gives matching numbers, then the difference is an implementation difference.” That’s the motive for a harness.

The fix was simple. Abstract the frame source, and split it into the side that records the real screen and the side that replays it deterministically. Now I could push any number of configurations through the same frame sequence and run a deterministic A/B.

— But determinism was still the surface. Underneath it was a more fundamental motive.

The real motive: I was the one shooting the recording every time

To be honest, wanting determinism was only half of it. The real reason was that shooting the recording myself every time was tedious, and it had become the bottleneck. Every time I wanted to try one candidate improvement, I had to pick up the device and record five minutes of video. Development speed was rate-limited by my own manual work.

A line from an Elon Musk interview had been sitting in my head the whole time.

“Any given thing can be sped up.”

So without quite thinking about it, I was designing for a state where the bottleneck isn’t me. Once you’ve baked the recording to disk, my hands are never needed again. A harness was an instrument, yes — but it was also a device for taking myself out of the work loop.

The biggest payoff, once I ran it: not accuracy — cycle time

Once I was running everything through the harness, I did get the effect I’d expected: my hypothesis tests got more accurate (the same input gives the same number, so I can say flatly that a difference is an implementation difference).

But that wasn’t where it paid off most.

Once it’s baked to disk, the need to re-shoot five minutes of video every time disappears. The manual loop I’d been spending my time on vanished entirely. And I could push many different configurations through the same frame sequence — in parallel, deterministically. No picking up the device to re-record, no worrying that the scene had come out different and made the comparison impossible. The try-measure-fix cycle started spinning much faster. And the instrument count grew like this —

122 in about four months (of which 119 came after May, when I named the thing and it took off properly as a “harness”; the other three are benchmarks from the prehistory). Measuring had stopped being something I did occasionally and become the default mode of development. Not because the philosophy was noble. Just because measuring got fast and cheap, so the reasons not to measure disappeared.

When iteration gets cheap, you measure a surface, not a point — Sweep

When the cycle gets fast and automatic, the questions you ask change. Instead of “is A better than B?” (comparing points), you start asking “if I sweep this parameter end to end, what shape is the curve?” (scanning a surface). That’s the idea behind Sweep.

The first thing I did was vary the image resolution for OCR across the range and take a look at how accuracy and speed moved. Before long, doing the same thing for every parameter — thresholds, model settings — was just what you did. Instead of settling on one value by instinct, you run a dozen-plus in one go and find the sweet spot in the curve. A number of runs that would make you question your sanity by hand takes minutes with a harness.

This is the Hyperbolic Time Chamber effect

These days I lean on the harness as a “Hyperbolic Time Chamber” (Dragon Ball’s room where a year inside passes as a day outside) — a device that pushes the number of iterations per unit of real time into another dimension. A day in the outside world becomes dozens of experiments inside the room. What comes out stronger isn’t the model, and it isn’t me: it’s the speed and confidence of my decisions.

And when you work with AI, this compounds quadratically

This is the part I most want to get across.

Since I started developing with AI agents, I’ve noticed something. If you’re sitting inside every verification loop, then you become the new bottleneck. The AI can write the code in ten minutes, but if you’re the one checking by hand whether it’s any good, the rate limiter comes right back to you.

A harness dissolves this. Given deterministic, measurable scaffolding plus automation, the AI can run the loop itself — change something, run it, read the numbers, decide what’s next. You get to step outside the loop.

So here’s the claim. “The harness is a shared language between humans and AI” (both looking at the same numbers, keeping each other’s hallucinations in check) and “the harness is a device for taking yourself out of the bottleneck” are two sides of the same coin. And — this is exactly where it pays off in an age of working with AI. The faster humans get, the more the next rate limiter becomes “a human is doing the verification.”

Lessons

1. Measure-first is really about cycle time, not accuracy. The first bottleneck to knock out is usually yourself.
2. Determinism isn’t the goal; it’s the precondition for automation. The same input gives the same number, so you can hand the loop to a machine. And once iteration gets cheap, measure a surface, not a point (Sweep) — it’s faster and surer than picking one point on instinct.
3. The more you work with AI, the more a human left in the verification loop becomes the rate limiter. The harness is what gets the human out of the loop — compound interest for the AI era.

When people talk about optimization, they reach for the object — the model, the code, the latency — and try to make it faster. But the any given thing in “Any given thing can be sped up” includes you. Take yourself out of the loop first. That’s when time inside the room starts to flow.

---

Appendix: fact check