A general-purpose model is too wasteful a vessel for subtitles

My own model with no compression applied at all — raw, 8 billion parameters (8B) — still left 15 critical mistranslations out of 97. On the same hard cases, a general-purpose model more than three times bigger (27 billion = 27B) had 4. No matter how I varied the strength of the compression, those 15 didn’t move.

So the thing setting this floor — where the worst case bottoms out — was the model’s size (capacity), not the compression. Every time quality fell apart, I had suspected compression first, and acted on it — the clearest case being smoothing out compression error with a repair called rotation, which took catastrophic failures from 64% to 24%. That line of attack died here.

But before falling back on “make it bigger,” there’s one more suspect: the vessel (the architecture) itself. Subtitling is a job of short sentences, tight constraints and bounded context — and yet we’re using, as-is, a general-purpose LLM designed for free generation of arbitrary length. The hypothesis: cut the waste that generality costs, and a smaller model than today’s should produce better subtitles than today’s.

And what I’m after isn’t only speed. What LYR is aiming at is the experience of subtitles at the level a professional human translator would produce, running over the video you’re watching right now. Professional quality itself is available — if you wait. But that wait stacks up: booking a translator, a rough pass, proofreading, timing, rights clearance — weeks after release, and for some titles months. And — by far the bigger share is work that never gets translated into your language at all. Professional subtitles only get made for the languages and titles where the economics work, so for a lot of people “wait and you’ll get it” isn’t even on the table. Which leaves the viewer choosing between not watching when they want to and settling for crude automatic subtitles. Real time, on the other hand, is available right now — if you give up on quality. It only becomes magic when both hold at once.

So what I’m watching are two things: the note a native speaker gave me — “the meaning comes through, but the emotion is stiff” — and the speaker’s gender, obvious from one look at the screen and impossible to recover from the source text. My read is that these two only come within reach once you change the vessel.

Let me be honest about this. This piece isn’t a record of measurements; it’s a design hypothesis. I have a reasonable read on the architecture, but I have no data points on my own language pairs yet. So at the end I’ll also write down what result would make me throw the hypothesis away.

Where things stand — out of borrowed AI, and small and fast as far as that goes

The setup, briefly. LYR is a product that translates text on your phone in real time. It gets used three ways — Live, translating the subtitles running across a video one line at a time; Page, translating the text on the screen in one batch; Manga, translating the dialogue in comic panels. One and the same translation model does double duty across all three.

With that, let me fold the road so far into one paragraph — with the caveat that this article is only about the model. In practice, most of the speed and cost came out of things outside the model — queues, distance, on-device processing, the number of calls themselves — and in plenty of cases those mattered more (I’ll leave them to the infrastructure and OCR categories). Here I trace just one layer, the model, straight down.

Restricted to the model, the first step was lining up third-party translation APIs side by side and picking the fast one (some AI gets disqualified before you ever measure quality). Then I hit the wall that borrowed AI disappears through retirement and price increases, and moved to my own GPU. From there I confirmed that compressing to a quarter barely costs any smarts, and pinned down that what decides generation speed isn’t the parameter count but the byte count of the weights. On the quality side, I took an un-fine-tuned small model from 42% to 84%, and got to where narrowing to the single point of Live subtitles lets an 8B draw level with a 27B.

So “you don’t need a general-purpose giant; for one trick, a small model is enough” is already partly demonstrated. This article is about what comes after that. Can I push that narrowing to one trick past the training data, all the way into the vessel (the architecture)?

The hypothesis — foundation models are too multi-task for subtitles

Today’s LLMs write poetry, write code, do math. To be that general-purpose, they’re built to emit text of any length in any format.

The subtitle job is the exact opposite.

I call this mismatch the generality tax. You pay it in two ways.

First, you pay it as wasted data reads. Every time the model emits one character, it re-reads its own insides (the weights = a vast pile of numbers) in full. That’s why generation speed is decided not by how heavy the computation is but by how much data gets read (generation speed is decided by the byte count of the weights). The numbers that exist for poetry and code and math get read out too, on every single character of subtitle. You’re paying freight, on every character, for abilities you never use.

Second, you pay it as a forced choice between speed and quality. As things stand, pick a fast model and the translation gets rough; pick an accurate one and it gets slow. That exchange rate looks immovable. But it isn’t a law that comes with the subtitle job. It’s just that the general-purpose vessel we happen to be using sits on that particular rate. Change the vessel and there’s room for the rate itself to change (the Pareto view).

I don’t think this is LYR’s problem alone. Now that the “drop a foundation model into the product as-is” phase has gone around once, recutting the vessel per use case ought to be ground a lot of teams are stepping onto at the same time. I’m looking at it from the single vantage point of subtitles.

But “smaller at the same quality” isn’t unconditional — two counterexamples

Let me put the measurements that are inconvenient for the hypothesis first.

Counterexample 1: specialization is a reallocation, not a free lunch. The same training that took Live (video subtitles) from a 90 to a 96 success rate knocked the other two — Page (whole-screen batch translation) and Manga (dialogue in comic panels) — down by 4 each (A small specialist drew level with the strongest model in service; the mechanism is mode mixing). As long as one vessel does double duty across several jobs, whatever you gain has been carved out of somewhere else.

Counterexample 2: there’s a layer specialization can’t fill. Handing the model perfect context didn’t fill in deep comprehension (“add context and it gets smarter” was half a lie), and comprehending dropped subjects and idioms in Japanese demanded capacity itself (spotting the rate limiter in the language you’re worst at). “Fixable by polishing” problems and “needs capacity” problems are different animals.

So the question becomes: of the gap that remains, how much is capacity, and where does the generality tax start? Saying “change the vessel and everything gets better” with the two still mixed together is giving up on diagnosis. So I separated them first.

The verdict — the floor was capacity, not quantization

The first move in separating them was ruling quantization out as a suspect.

First, quantization in one line. The insides of an AI model are a vast collection of numbers. Quantization holds those numbers on a coarser scale and repacks the model into something lighter (compressing to a quarter barely costs any smarts). It’s like re-saving a photo as a JPEG — the size drops dramatically, and a little detail is lost. The production model runs in that compressed state, because it’s faster and cheaper.

And every time quality fell apart, compression is what I suspected first. I’ve observed both cases for real: the compression conditions were mismatched between training and production, and compression was inflating catastrophic failures (the floor). In the latter, a repair called rotation took catastrophic failures from 64% to 24% — but it never reached the pre-compression level (12%). Was that remaining half leftover compression damage, or something else entirely? Which raises the question: if I take compression all the way off, does the floor come back? Put another way — with a raw 8B that hasn’t lost a single bit of detail, do those critical errors go away?

I had it translate 100 Korean→Japanese hard cases (97 effective, after removing items that overlapped the training data) and judged them blind to which configuration produced which translation. The comparison was the same self-hosted 8B run at three levels — heavy compression / medium compression / no compression (raw) — against the general-purpose 27B. What I counted wasn’t an average score but the number of critical errors — translations that mix up the subject or the meaning and don’t work as a subtitle at all. Averages dilute exactly this kind of rare catastrophic failure until you can’t see it (raise the floor, not the ceiling).

The result is Figure 1 at the top. Here are the numbers again, on their own.

Here’s how to read it. Taking compression off entirely didn’t reduce the critical errors. So quantization is not what set this floor. Since the 27B keeps it to 4 on the same hard cases, what’s left is the difference in model size. A spread of 15 against 4 is hard to explain as chance wobble.

That separated the levers available to me into three.

The conclusion of the experiment is that I stopped betting on quantization. It’s still live as a speed lever, but it wasn’t a tool for raising the quality floor. The investment moves from compression to the vessel.

What I expect from specialization isn’t only speed

Here’s the point I least want misread. Talking about the value of a specialist model purely in terms of latency is too narrow. What I expect from recutting the vessel is three independent improvements.

① Emotional register. When I had a native speaker read translations off the actual device, the verdict that came back was: “the meaning comes through, but the emotional parts are stiff.” This isn’t a mistranslation. The meaning is right. And still, as a subtitle, it catches your attention — and in that instant the viewer steps outside the story. If the ideal translation is a subtitle you never notice is there, this isn’t decoration; it’s the main keep.

And register doesn’t move however you tune the instructions (the prompt). I know by now that the only way is to retrain the model’s insides (the register the 4B struggled with showed up at 8B is the same story). A general-purpose model’s register is pulled toward the general-purpose average. Getting off that average is the first motive for changing the vessel.

② Accuracy from translating while looking at the video. How much better does the translation get if you hand it perfect information about the scene? I’ve measured that elsewhere. What worked condensed down to three things: the speaker’s gender, how proper nouns are read, and dropped subjects (13 improvements against 0 regressions).

The biggest of the three was the speaker’s gender. Japanese exposes who is speaking constantly, through pronoun choice (ore vs watashi) and sentence-final particles (~daze vs ~dawa), while the English source carries none of that information.

So when the model has to infer gender from the text alone, it drops to 34% right and 21% wrong. Wrong is worse than “don’t know” — it invents a fact that doesn’t exist and contaminates the translation.

But this isn’t a hard problem in principle. Look at the screen and it’s obvious who’s talking. There’s the saying that a picture is worth a thousand words. In translation it bites harder than that — some of those thousand words were never in the source text to begin with. A model that only follows text can re-read the missing information as many times as it likes and never recover it. No amount of stacked context reaches what one frame would settle. In fact, simply training gender information into the model raised the rate at which it correctly switches first-person pronouns and sentence endings to match a male or female context from 19% to 74% (a first demonstration on a separate evaluation set). As long as the vessel has no entrance except text, there’s no way to put this information in. Building that entrance is the second motive for changing the vessel.

③ Latency. The third is the freight a general-purpose model pays on generation. That’s known ground (covered elsewhere), so I won’t repeat it.

What matters is that the three are independent of each other. Which means that if it works, you don’t get an addition — the whole curve lifts.

The next vessel — a “top-heavy” subtitle-specialist model

The insides of a model split broadly in two: the Encoder (the reading side), which reads the input and grasps the meaning, and the Decoder (the writing side), which writes that out as a translation. Today’s models give the two roughly the same thickness. And the subtitle job gets pushed through that same allocation.

What I’m considering is changing that allocation. A big Encoder, a thin Decoder. The subtitle job isn’t “talk at length and freely,” it’s “render it short and correct” — so the natural allocation puts the weights thick on understanding and thin on writing out.

The reason I expect this shape to work is that the number of times each side runs is asymmetric. The Encoder runs once per input; the Decoder runs on every character it emits. So thinning the Decoder drops latency alone, while the thickness of the understanding stays.

The three shifts each push a different axis. The aim isn’t to pick a winner, it’s to move the trade-off curve itself.

Let me separate what’s backed by evidence from what isn’t.

What’s backed by evidence. “Thick Encoder, thin Decoder” is an architecture already demonstrated in machine translation (there’s prior work where even a single-layer Decoder beats strong methods in the same speed band). On the video side, I’ve confirmed for myself that feeding in gender information switches the register correctly (19% → 74%). None of the parts are new. The only new thing is binding them into one model around the single point of subtitles.

What isn’t. That prior work was demonstrated mainly on English↔German, and Japanese↔English/Korean is the language combination this kind of speedup struggles with most (word order differs so much that the writing side carries more of the load). So there’s no guarantee that “thinner without losing quality” carries over to my own languages as-is. My estimate is that the realistic answer isn’t extreme thinning but something like keeping it at 1–2 layers + having it imitate a larger model’s translations as material (distillation) + routing only the hard cases out to a larger model.

And the thing I should be most honest about: the upper bound on the lift is limited to what you recover by dropping generality. As Figure 1 showed, part of the Korean→Japanese floor is set by model size, and changing the vessel leaves it there. What the vessel buys back is speed and register; the comprehension floor can only be filled by raising size, or by a design that routes the hard cases away.

Open questions — I have a read on the architecture; I have no data points yet

Everything above is what I’ve thought through to some degree. What follows is what still needs thinking.

Let me put the falsification conditions up front too. If thinning the Decoder (the writing side) costs more quality than what dropping generality recovers — concretely, if the thinned vessel loses on quality to a general-purpose model at the same speed — I throw the hypothesis away. If adding video doesn’t move the register metric, I cut the video shift loose. Because the three shifts are independent of each other, one of them failing leaves the other two standing. Rather than asking “is this whole plan right?” as a single question, I can verify them one at a time and drop only the ones that don’t hold.

That’s as far as I can go for now. The intuition that “foundation models are too wasteful for subtitles” has an explanation behind it — the generality tax — and a measurement that ruled quantization out as a suspect. But I don’t yet hold a single piece of evidence that changing the vessel wins. Keeping that line sharp is, I think, the condition for measuring the next thing right.

Lessons

1. “Faster through compression” and “smarter through size” are different levers, and you must not talk about them as one. With compression off entirely, the critical errors stayed at 15; the model more than three times bigger had 4. The moment the quality floor turns out not to move, compression drops out of the quality toolbox — even though it stays alive as a speed move. Hold each lever together with where it acts.
2. Don’t talk about a specialist model’s value in terms of speed alone. A general-purpose model’s register is pulled toward the general-purpose average, and a vessel that only accepts text has no entrance for video. Speed, register and video are three axes that act independently — and because they’re independent, there’s a chance of moving the whole trade-off curve rather than adding up gains.
3. Write “I don’t have the data yet” plainly. Having a read on the design and having it work under your own conditions are different things. The prior work was demonstrated on a different language combination, and there’s no guarantee it carries over as-is. Putting a hypothesis down as a hypothesis and fixing the falsification conditions first finishes faster than starting to build on momentum.

---

Appendix: raw data

A. Separating quantization from capacity (the data behind Figure 1)

How to read it: the floor doesn’t move with or without compression, so compression isn’t its cause. The gap to the 27B remains, so the cause is model size (capacity). → Quantization is done as a quality move; the investment moves to capacity / video / the shape of the vessel.

B. Why change the vessel (the three shifts, and the evidence so far)

C. Caveats

• Figure 2 is a schematic of the plan; there is no implementation and no measurement. The dashed line in Figure 3 is likewise an unmeasured hypothesis, and the points on the solid line are a conceptual diagram showing relative position — the coordinates are not measured values.
• The n=97 in A is small. The p-values come from paired comparisons on the same set; read them as a hint about direction.
• The upper bound on the lift is what you recover by dropping generality. Part of the Korean→Japanese floor is set by model size and remains after changing the vessel (→ the assumption is that a design routing hard cases to a larger model fills the rest).