Taking a small AI's accuracy from 42% to 84% — and on some tasks, level with the giants

Accuracy 42% → 84%. On some tasks, level with the giant model currently in service (96). And the model stays small.

What worked wasn’t “making it smarter” — it was narrowing it to one job. Most of a giant model’s smarts are things LYR’s task never needs. Training comes in two stages: model answers for the form, preferences for the flavor.

But there’s a line around what you can train in. The form of how things are said can be trained in cheaply, but deep comprehension and emotional register need capacity (size). The register the 4B struggled with showed up the moment I moved up to 8B.

Last time I made my own small model fast and cheap. But that’s only half the job. Left as it is, a small model loses on quality to the giant models.

By how much? Hand an un-fine-tuned small model (4B = 4 billion parameters; from here on, 8B = 8 billion and 27B = 27 billion for the large ones) a Japanese→English translation and its accuracy was only 42%. Unusable in practice.

So how do you close that gap? The answer is Figure 1 above — I pulled that 42% up to 84%, the same as the retired original model, and on some tasks all the way to level with the giant model currently in service. And the model stayed small.

This is about the recipe — and about the wall the recipe can’t get over.

A giant AI is a jack-of-all-trades

First, why does a small model lose? And why is there room for it to win at all?

GPT, Claude, the large Qwens — today’s giant models do everything: poetry, legal summaries, code, medical questions, and translation too. They’re giant precisely because of that range.

But what LYR needs is one thing: subtitle translation. It never once uses the ability to write poetry, or to write code. Most of that general-purpose intelligence is dead weight on LYR’s task (overspec).

Which means — if you narrow it to a single job, even a small model ought to reach the giants on that one point. Give up generality, get specialization back. That was the bet behind “specialization.”

Training comes in two stages — model answers, then preferences

So how do you narrow it? You apply a treatment called fine-tuning, in two stages.

This is actually the same path ChatGPT took to get smart. A base language model → show it model answers so it follows instructions (SFT) → polish it with human preferences (RLHF). LYR followed the same order.

1. SFT: show it model answers, in bulk

Stage one is SFT (supervised fine-tuning). In short: show it a huge number of model answers and hammer the form of the task into it.

I had the larger giant model (27B) produce the model answers. Have the giant translate a few thousand items, then have the small model imitate those translations. This is called distillation. Picture the student copying out the teacher’s answer sheet.

The effect was dramatic. Japanese→English accuracy went 42% → 84% (+42 points). That is the same level as the retired model I had been running in production until recently. Stage one filled in most of the gap.

That said, SFT has a ceiling. The student imitates the teacher too faithfully — it copies the teacher’s phrasing tics wholesale (always rendering “look” as the same literal 「見て」, say). And above all, what the model fundamentally cannot understand doesn’t get fixed by imitation.

2. CPO: put a good translation and a bad one side by side

Stage two is CPO (preference optimization — fine-tuning that teaches the model “preferences”). After model answers comes preference.

The method: show it a good translation and a bad one as a pair, and have it learn “this one is better.” It’s the stage that corresponds to human feedback (RLHF) in ChatGPT’s case. The sense of “this is what natural sounds like” and “this isn’t,” which model answers alone can’t convey, gets trained in by contrast.

When I applied this, specialized to Live subtitles, subtitle translation quality went from 90 → 96 (that 90–96 is a score for overall subtitle quality; the 42→84 earlier was accuracy on Japanese→English translation — note that these are different yardsticks). That’s level with the giant model currently in service (96), and within touching distance of the retired original (97). Literal renderings of idioms (“it isn’t like you” now coming out as the idiomatic 「あなたらしくない」) and picking the wrong subject got corrected, deliberately, one after another.

Doubting the “gold data”

This was the biggest discovery of the whole thing.

In stage-one SFT, you use the teacher’s (27B’s) answers as “model answers = the right answer.” Naturally, I assumed that one answer was the gold data, and that training meant getting closer to it.

At some point I doubted that. What if I had the teacher produce several translations instead of one, and re-picked the best of them on quality? The result — the best candidate disagreed with the “gold data” 86% of the time.

In other words, the teacher’s first-shot answer — the thing I had been calling “the right answer” — was not the best one in 86% of cases. It was verbose, or too literal, and a better candidate sat elsewhere in the set.

The lesson: stop preparing one right answer and training toward it. Generate a lot of candidates and select on quality itself (QE = quality estimation). There is no single “correct translation” for a subtitle. (I’ve stepped in this trap — where chasing the gold data actually makes things worse — in another setting too.)

And the method has a second payoff: it produces the SFT model answers and the CPO material (good/bad pairs) at the same time. That 90→96 above was in fact a combination: stage-two CPO applied with this “material re-picked from candidates.” What was doing the work wasn’t CPO as a technique so much as the quality of the material you feed it.

How this meshes with quantization

One last warning. The smarts you went to the trouble of training in can be shaved away by last time’s quantization. Fine-tuning’s effect sits “thin and wide” across the whole model, so aggressive compression tends to take it out first.

So hold fine-tuning and quantization to the same conditions. And always evaluate at the quantization you deploy in production. Celebrating “it got smarter” in the uncompressed state means nothing if the gain is gone after compression.

What you can train in, and what you can’t

Fine-tuning is not a cure-all, though. As I stacked up rounds of training, the walls you can get over and the walls you can’t separated out clearly.

• Transferable by fine-tuning: the “form” of how things are said. Phrasing, idioms, brevity, formatting — the form of how to say it can be pulled up toward the larger models by fine-tuning, even on a small model.
• Needs capacity (size): deep comprehension, and emotionally loaded register. Who the subject is, which gender, whether a negation applies, what the proper nouns are (= reading comprehension). And a register that matches the temperature of the scene. These two can’t be filled in by teaching good/bad alone; they need the model’s capacity. In fact, emotionally loaded register only reached giant-model level (98–100%) — a struggle at 4B — once I moved up to 8B.

To back that up, I ran a controlled experiment. Push the preference training hard on its own and Japanese→English accuracy (comprehension) goes 84 → 85% — inside the judge’s noise. The form of how things are said moves; the ceiling on comprehension does not.

To sum up — the “form” of how things are said is cheap to reach for, even on a small model. But “deep comprehension” and “emotionally loaded register” are the model’s capacity itself. The recipe and the capacity (the model’s size) have to be designed as a set. Specialization isn’t magic; it was the work of telling apart which parts you can reach by training and which parts need capacity.

Lessons

1. Most of a giant model’s smarts are things your task doesn’t need. Narrow to one job and even a small model gets there on that one point. Training comes in two stages — model answers (SFT = distillation) for the form, preferences (CPO) for the flavor.
2. Doubt the “gold data.” Generating candidates and selecting on quality (best-of-N) gets you further than fitting to a single right answer. The best translation is usually not the “right answer” you prepared.
3. What you can train in is the form of how things are said; what needs capacity is deep comprehension and emotional register. The form is cheap to train in, but comprehension and feeling need capacity (size). Design the recipe and the size together.

So now my own small model is fast (last time), cheap, and — on one job — smart. It draws level with a general-purpose giant AI on LYR’s task and nothing else. It’s the last piece needed to let go of borrowed AI.

---

Appendix: raw data