The AI that was smart on the bench suddenly started making mistakes in production

A fine-tuned 8B model was a star pupil on the benchmark, and then lost to the smaller 4B in a production-equivalent setup (21 to 11). It looked like the limit of the model’s capacity.

The culprit was neither the model nor the training. It was that the precision I trained at and the precision production ran at were different. Putting the same adapter on both bases and comparing them, the verdict came back 23 to 2: it was the environment mismatch.

Once I re-trained with production’s compression priced in, the 8B came out ahead of the 4B. Hold the conditions you train the skill under and the conditions you serve it under to the same thing. That alone was enough for the skill that had gone missing to show up intact.

I was holding a winning hand.

I wanted better translation quality. Which means the bigger model — an 8B (8 billion parameters) remembers more than a smaller 4B and can read subtler context. And in fact the 8B I’d trained with the fine-tuning from last time was posting unimpeachable numbers on my own benchmark. All that was left was to put it into production. That’s what I thought.

But when I loaded it into a production-equivalent setup and re-measured quality, that 8B lost to the 4B that was supposed to be the smaller one. On one evaluation, 21 to 11. The order is backwards. The bigger model is supposed to be the smarter one — and yet in production the smaller one is winning.

For a moment I was about to conclude: put Live translation on the 4B, plain and simple. The bigger model just isn’t suited to this use case.

That was the mistake. The 8B hadn’t gotten weaker. The 8B I was measuring and the 8B I put into production were two different things.

The model’s “precision” was different on the bench and in production

Before the reveal, one piece of vocabulary. Quantization is the compression technique that takes the model — that “huge mountain of numbers” — and re-stores it with fewer bits so it gets lighter. I covered it in detail in Part 1, but the gist is this: to run it fast and cheap on a production GPU, you compress the model before you load it.

That’s where the pitfall was. Without ever deciding to, my workflow had ended up like this.

• Fine-tuning (training) happens at full precision. In BF16, a format that holds the numbers finely, I train the translation skill into the model.
• Production (deployment) runs compressed. I load it in INT4, a format shrunk to a quarter of the size, and take the win on speed and cost.

In other words, the form the skill was trained into and the form the customer got were different. The benchmark measures the full-precision form; production runs the compressed form. It’s as if two different people had written the exam paper and the real one.

Same adapter, different base — and the result flipped

Confirming the suspicion was anticlimactically easy. Take the exact same fine-tuning (the adapter), put it on two bases, and have them fight over the same 40 items.

• Base A = full precision (BF16). The same form it was trained in.
• Base B = compressed (INT4). The form deployed to production.

(An adapter is a “skill-correction part” you bolt onto a base model after the fact. The base changed; the correction bolted on top stayed identical down to the last bit.)

The result was blunt. Judging I handed to opus (put the two translations side by side in front of a third-party high-capability AI and have it decide which one is better).

The same skill was supposedly trained into both, and merely compressing the base dropped 23 wins to 2. And when I read through the defects one by one, subject mix-ups (getting whose line it is wrong) and polarity flips (turning “doesn’t” into “does”) — the kind of error that breaks the backbone of a sentence — came entirely from the compressed side. The 3 defects left on the full-precision side were all minor ones of another kind.

The good scores on the bench didn’t reproduce in production. Not because of capability, and not because the training had failed. The form I measured and the form I ran were different. That was all.

Why compression makes the skill go dull

You might find that odd. In Part 1 I wrote that “compressing it barely costs you any smarts (−1 point of quality).” Surely that’s a different story.

There’s no contradiction. The two act in different places.

The skill you train in through fine-tuning is a correction that sits thin and wide across the whole model. “In this context the subject is this one,” “this tone gets translated like that” — a delicate overlay whose job is to tip close calls onto the right side. And compression shaves off exactly that close-call margin first. Round the numbers coarsely and the majority doesn’t move, but the borderline decisions flip easily. The “subject” and the “negation” that fine-tuning had carefully tipped the right way get tipped back the other way by compression’s rounding error.

Here’s the analogy. When I trained the skill in, I drew the lines with a high-precision ruler — one with fine gradations. In production I swapped it for a ruler with coarse gradations and tried to copy those lines across. The finer the line, the less of it a coarse ruler can capture.

The fix: train it in the production form

If that’s the cause, the fix is obvious. Re-train the fine-tuning on the same compressed base you deploy to production. Draw the lines with the coarse ruler from the start. Build the skill with the rounding error already priced in, and that skill shows up in production intact. This way of “holding training and production to the same precision” is what’s called quant-aware training (training with production precision priced in).

I re-measured on the same 40 items.

The clearest effect was in the head-to-head against the old, mismatched 8B. It came out ahead 20 to 8, and counting only the items where the verdict was clear-cut, it was 10 to 0 — every subject and polarity error stayed on the old side, and the re-trained side was clean. Swap the opponent for “the 4B in its production form” and the 8B still showed no regression (16 wins / 15 ties / 9 losses, n=40. With 15 ties that isn’t enough to declare a winning record, but it is at least not losing). The assumption that “it loses to the 4B” fell apart right here. On Manga translation, it also held echo (copying the source verbatim) at zero across 434 items.

I withdrew the “Live goes on the 4B” decision. The 8B had never been the weak one.

Before you suspect the model, suspect how you’re measuring

What stung most is that I came within an inch of blaming the model’s capacity. “The 8B isn’t suited to this use case.” It’s a plausible hypothesis, and concluding it would have settled the matter quickly. But the real culprit wasn’t inside the model. It was in the procedure I used to measure the model. The environment for training and the environment for production didn’t match — not a model problem, a system problem.

“An AI that’s supposed to be smart suddenly turns stupid in production.” When you see that, the first thing to suspect isn’t the model’s ability. It’s whether the bench and production are really running the same conditions.

Lessons

1. Suspect the measurement before you suspect the model. When an AI that’s supposed to be smart starts making mistakes in production, what to suspect is not the model’s capacity but the measurement procedure. Put the same adapter on both forms — the training one and the production one — and you can separate whether the culprit is the model or the environment (it was the latter, 23 to 2).

2. Hold the conditions you train the skill under and the conditions you serve it under to the same thing. Fine-tuning and compression are separate steps, but get the interaction between them wrong and the skill you trained in disappears in production. Only once you hold everything from training to serving constant, as one continuous system, does the quality actually arrive.

3. The bigger the design decision, the harder you should check that the playing field is level. A switch from 8B to 4B nearly got decided by a single measurement mismatch. Before you conclude “it isn’t suited to this,” make sure the conditions you’re comparing are genuinely fair.

---

Appendix: raw data

Conditions: all Live, 40 items, opus pairwise verdict (win / tie / loss).