Don't explain the "why" until you've taken a control

When a number moves, the thing you most want to explain is why it got better. And that is exactly where it’s easiest to be wrong.

Before you explain a mechanism from a single observation, take one control: a comparison that differs only in the factor you want to test and is matched on everything else. I nearly concluded, from an n=1 observation on one feature, that “this step is the culprit” — and the control (n=14) proved it innocent. I was on the verge of swapping the design for a different plan.

A control also quietly knocks out false improvements. If a cheap parameter tweak beats an expensive retraining run on every axis, that retraining added nothing — what looked like an improvement was confounding.

The number moved. It got better. And then — you want to explain why it got better. “This lever worked.” “It improved through this mechanism.” The impulse is natural. But that impulse is exactly what produces the most expensive mistake in measure-first practice.

Before you explain a mechanism from a single observation, there’s something you have to do first: take one control arm. A control is a comparison that differs from the treatment in only the factor you want to test and is matched on everything else — say, “the same configuration, with one threshold put back.” Without a control, you can’t tell whether the improvement you saw is really thanks to your change, or thanks to something else.

This article is about the time I skipped that distinction and nearly dove into a misdiagnosis, and about the time a control quietly knocked out a false improvement.

n=1 can’t tell you the mechanism

At one point, on a build with a certain feature turned ON, I found one obvious anomaly. On one of the two measurement paths the contents came back empty, while on the other path there were contents. “This step I turned ON must be dropping the data.” One sample and I could see the mechanism — or so I thought. My hand was already on the next move: swapping the whole design for a different plan.

What stopped me was a single question. “Does the same thing show up with it OFF?”

I took a control — the same check, run with the feature OFF. With it OFF (n=14), one case of the same kind of disagreement showed up as well. And it pointed the other way. The real cause wasn’t the step I suspected; it was a slight offset between the moments the two measurement paths were sampled (time skew). One was capturing “the next instant,” the other “this instant.” ON was innocent.

n=1 can’t tell you the mechanism. Intermittent phenomena in particular can turn into something else entirely on a single boundary timing. When you find an anomaly on the treatment side, before you say a word about mechanism, run the same check on the control at the same n. If the disagreement rate is comparable, it isn’t your lever’s effect — it’s just measurement noise.

”It improved” — until you put a control next to it

The second case is quieter and more dangerous. It’s when something looks like an improvement, not an anomaly, that you’re most tempted to skip the control.

At one point a retrained model looked like it had “gotten better.” The numbers were up on the previous version. Naturally, I assumed the retraining had worked. So I put a control next to it — the old model, with one threshold changed. Against an expensive change like retraining, that comparison costs essentially nothing.

Here’s how it came out.

The cheap control beat the expensive change on every axis (Pareto dominance = no worse on any axis, better on at least one). The meaning is brutally clear. If an option you can produce with nothing but a parameter tweak beats your change across the board, your change added nothing. What looked like an improvement was the effect of some other factor with nothing to do with the retraining — confounding (the effect you want to measure and some other factor can’t be separated, so you can’t say which one you have to thank).

This “the control kills the improvement” pattern is two sides of the same coin as knocking out false improvements with the Pareto frontier. The frontier mechanically erases dominated points. If a cheap control covers up an expensive change on the frontier, the picture is telling you that the change contributed zero.

Before you say “I controlled for that,” did you actually vary it?

There’s a deeper trap around controls: settling it by argument — “I controlled for that factor, so it can’t be a confound.”

In the first case, this is what I thought at first: “I’m toggling ON/OFF within the same session, so environmental factors can’t confound it.” Plausible enough. But what I had controlled was the change — the toggle — not the environmental factors themselves. The factor I should have suspected, I had never actually varied. Saying “it can’t be a confound” about a factor you never varied isn’t measurement; it’s a wish.

So the question to ask yourself about a control is this: before you say “I controlled for X,” do you have a measurement in hand where X was actually varied? If not, that isn’t control. It’s just you saying “I don’t think X matters.”

So what is a control, in the end?

All three cases are saying the same thing.

• A control is a comparison that differs from the treatment only in the factor you want to test and is matched on everything else.
• A control is the baseline that lets you subtract out everything other than your lever’s effect — measurement noise, timing, unrelated side changes.
• An n=1 with no control has no standing to explain a mechanism, whether what it shows is an improvement or an anomaly.

And taking a control has a prerequisite of its own: the instrument itself has to be held constant. If the treatment and the control are measuring different things, the subtraction doesn’t hold. So before the controlled experiment, check first whether running the same code twice gives you the same numbers — the determinism of the harness. If the ground is wobbling, the control just captures the wobble instead of the difference.

Lessons

1. Take one control before you explain a mechanism. “It worked,” “here’s the mechanism” — those are words you’ve earned only once you’ve put a control next to it. n=1 in particular can’t tell you a mechanism: intermittent phenomena turn into something else on a boundary timing.
2. If a cheap control Pareto-dominates, your change is nothing. If a mere parameter tweak beats an expensive change on every axis, what looked like an improvement was confounding. It added nothing.
3. Before you say “I controlled for it,” ask yourself whether you actually varied it. And before the control, hold the instrument constant. Arguing that a factor you never varied “can’t be a confound” is a wish. If the treatment and the control measure different things, the subtraction doesn’t hold.

What’s frightening about measure-first is that the faster the numbers come, the more you want to attach a story to them. “Why did it get better” is the thing you most want to explain and the thing you’re most likely to get wrong. Measure the right thing, and measure it right — and the last piece of that “right” is the control. Until you’ve taken a control, don’t explain the why.

---

Appendix: raw data