Are you still picking that setting by gut feel?

Thresholds, image resolutions, batch sizes — most of them get picked by gut feel. Try one or two, take whichever looks better.

The moment verification got cheap, I stopped deciding that way. Instead of picking a single point, I sweep the range end to end and look at the shape of the curve. The sweet spot, the knee, the cliff — none of them show up until you measure the shape. A single point tells you nothing about the shape.

And whether you can sweep at all comes down to how cheap verification is. When one run is expensive, people fall back on gut feel. Make verification cheap first — that comes before anything else.

Engineering is full of magic numbers (concrete values sitting in the code with no clear justification behind them). Thresholds, image resolution, batch size, timeouts. How do you decide those values? To be honest — mostly, by gut. You try one or two at best and take whichever looks better.

When the harness made verification cheaper by an order of magnitude, I stopped deciding this way. Instead of picking one point, I started to sweep the range end to end and look at the shape of the curve. That is a Sweep.

A curve shows you what a single point can’t

Take the image resolution used when OCR detects text. “Higher resolution means higher accuracy” — that’s what intuition says. So if you’re going by gut, you pick the higher one without hesitating.

But when I varied the resolution across a range and measured accuracy (CER, lower is better) and speed together, this is the curve that came out.

The middle one, 736, was the obvious “sweet spot.” Against 960, the error rate goes up by a mere 1.3 points (9.4% → 10.7%), and the speed halves. Drop to 480, on the other hand, and accuracy collapses (the error rate goes from 10.7% to 35% — more than tripling). The bend in the curve — the knee — sits exactly at 736.

Had I picked 960 by gut, I’d have gone on paying double the time for a sliver of accuracy. Had I picked 480, the text wouldn’t be readable at all. This sweet spot only became visible once I measured both ends and the middle. Measure a single point and you will never find it.

Curves have a “knee”

There’s one more thing only a curve will show you: the knee — the bend past which doing more barely improves anything.

When I swept how quality rises as you add more training data, reading-comprehension accuracy on the difficult cases went flat past a certain amount of data (a very small amount). However much data you add, the hard cases don’t improve. Past the knee, the investment is wasted — a signal that a different lever (the model’s capacity) was what it needed.

This too is invisible from one or two points. Only when you draw the curve do you see the boundary where “nothing beyond here works.”

Why sweeping is possible now

Sweep is an old idea. But maybe not that many people do it thoroughly in practice. The reason is simple: it’s expensive. Trying eleven thresholds by hand means eleven verification runs. That’s a number of runs that makes you question your sanity.

What changed is that the harness took the human out of verification. Once you’ve baked the frame sequence to disk, you can run eleven configurations in parallel, with zero manual work. Where a person used to hold the device and record once per configuration, everything now runs in one batch. The moment verification got cheap, “sweep a surface and look at the shape” became more reliable and better value than “pick one point by gut.” Sweep was the first use I found for the slack the harness created.

Lessons

1. Don’t set magic numbers by gut. Thresholds, resolution, batch size — most of the time there’s no reason why the value is that particular number.
2. Sweep the range and look at the curve, not one point. The sweet spot and the cliff only become visible once you measure the whole shape. The knee especially — the bend where “nothing beyond here works” — is the signal to move to a different lever.
3. Whether you can sweep comes down to how cheap verification is. When a single verification run is expensive, people fall back on gut feel. Make verification cheap first. Then sweeping becomes the obvious thing to do.

Next time: what you see when you read that cloud of swept points against several objectives at once (speed × accuracy × cost) — on to Pareto. Sweeping and reading the frontier are two sides of the same coin.

Appendix: raw data