Five rebuilds and it still never beat chance — the cause was the labels, not the model

I wanted to catch only the moment a subtitle switched, so I could cut how often I called the expensive downstream work (text recognition and translation). I rebuilt the small classifier that does that five different ways. I changed the input, changed the architecture, changed how it aggregated. Not one of them was more accurate than chance.

The cause wasn’t the model. The ground-truth labels were off by about a second. When I audited them, the frames the labels called “changed” had less pixel motion than the ones called “unchanged” — the labels pointed the opposite way from the truth (AUC 0.056).

And broken labels cost you twice. They don’t just produce a bad model — they had also been undervaluing a good implementation I already had as “25% misses.” Re-measured against correct labels, that implementation’s true misses came to 1 out of 401.

Five rebuilds, and still no better than chance

What I was building was a small model that decides whether the subtitle on screen has switched. Get that right and you can cut how often you call the expensive downstream work — text recognition and translation.

I rebuilt it five times.

1. Feed it the raw screen (grayscale)
2. Feed it a text-likeness map
3. Feed it both, as two channels
4. Drop the global-average layer so positional information doesn’t get flattened
5. Tweak the architecture further

Every one of them came out no better than chance. There are two conclusions you’d normally reach at this point — “this task can’t be learned” or “the model is too small.” I started writing exactly that.

Suspect the labels, not the model

Then I stopped mid-sentence. Getting the same result five times running is, if anything, suspicious. If I’d changed both the input and the architecture and everything was still stuck at the same plateau, the sensible thing to suspect is whatever they have in common. What they have in common is the ground-truth labels.

So I decided to score the labels themselves — with an independent signal that had never gone into training. What I used was the pixel difference between adjacent frames: a quantity that has nothing to do with the model and looks only at how much the screen moved.

On a frame where the subtitle switched, the pixels should move a lot. So if the labels were right, positive frames should show more pixel motion.

The result: AUC 0.056. Below the positive rate — that is, inverse correlation. The frames the labels called “changed” were the ones where the screen moved less.

The labels pointed the opposite way from the truth.

Tracing it back, the cause was simple. The labels were built from the official subtitle file (the data that says what to display at what timestamp). But on the actual screen, subtitles fade in, and the display durations are filled in with assumed values. The result: the labels fired about 5 frames (one second) later than the change on screen.

There was a second pitfall. I thought, “let me check this against a different set of candidate labels” — but that candidate was also derived from the same subtitle-track timings. Compare two label sets from the same source and of course they agree; you’ve checked nothing. It’s a tautology.

Rebuilding the labels

The fix was to throw the subtitle track away and build the labels from the screen itself.

Run the same detection and recognition as production over every frame. Mark the points where the recognized text differs from the previous frame as positives. One episode is about 8,000 frames, and it ran at several dozen times real time.

Score the rebuilt labels the same way.

The clearest piece of evidence is that the best shift correction came out at 0. The old labels only lined up with the screen once you pulled them back five frames.

Broken labels undervalue good implementations too

Here’s the part that actually cost me.

I had been evaluating the implementation I already had against those same broken labels. It’s a naive change detector built on hashes, no learning involved. Its score under the old labels was “25–28% misses.” That’s why I decided there was value in replacing it with a learned model, and started on the five rebuilds above.

Re-measured against correct labels, that implementation came in at 98.8% precision / 99.5% recall. It was already at ceiling.

I also found out what the “misses” really were. Going through the frames flagged as misses one by one, 400 of 401 had no text on screen at all. Frames the subtitle track merely claimed were “displaying” — actually empty. The true misses came to one.

In other words, there was never any room to replace it. The learned-model project closed there. It’s less that the five rebuilds were wasted work — it’s that the broken labels had manufactured a false investment decision: that this was worth doing.

Open questions

The rebuilt labels aren’t a cure-all either.

The pixel difference can’t tell apart things that moved a lot but aren’t text (a scene cut, the whole screen going dark). Push precision up to 0.7 and recall drops to 0% — it works for auditing labels, but it can’t be the change detector itself.

And this check was run under the condition where the video is black and only the subtitles are composited on top. On video with a moving background, the pixel difference itself means something else. The same procedure won’t necessarily carry over as is.

Lessons

1. If five rebuilds all land on chance, suspect the labels, not the model. When you’ve changed the input and the architecture and everything is still stuck at the same plateau, score the thing they have in common — the gold data — first.
2. Audit with an independent signal that never went into training. Check against data from the same source as the labels and of course they agree; you learn nothing. Here what worked was the pixel difference, a quantity unrelated to both the model and the subtitle track.
3. Broken labels cost you twice. They don’t just produce a bad model — they undervalue the good implementation you already have and make an investment that isn’t worth making look like one that is. That wrong decision cost more than the five failures did.

This sits one step in front of the manifesto’s first principle — Measure First. Before you measure, measure the yardstick you’re measuring with. The same discipline as questioning the judge applies to the gold data too.

Appendix: raw data

Measured on a black background (the video is black and only the subtitles are composited on top). Higher AUC is better; 0.5 is chance.