Swahili from 0% to 30%. But the value wasn't the score — it was that the failures changed

Swahili is one of the languages AI is worst at. Translation accuracy went from 0% to 30%. But the value wasn’t in the score — it was that the failures themselves changed.

From “strings that aren’t even words” (= the model doesn’t have the language) to “fluent, but the meaning slips” (= the teacher is bad). The score is just as low either way, but it is a strong clue that the rate limiter has moved off capacity and onto the teacher.

And in low-resource languages, a huge teacher doesn’t necessarily help. Tripling the teacher’s size produced no difference, and a smaller teacher designed for low-resource languages won instead (9 wins / 5 losses). About $15 per language jumps a model from “doesn’t have it” to “usable but rough.”

Translation AI has languages it is good at and languages it is hopeless at. English and Spanish are easy — there are mountains of parallel text online. And then there are languages with over 100 million speakers that AI simply cannot translate. Swahili is the poster child.

When I had LYR’s own model translate into Swahili, accuracy was 0%. Not one of the 10 items passed. This is where most teams conclude: “too much to ask of a small model — either scale it up or give up.”

But that isn’t what this article is about. It’s not about the score being 0% — it’s about how that 0% was wrong. I looked inside the failures, and the rate-limiting step (the bottleneck) showed itself.

There are two kinds of “0%”

Start by looking at how the base model got Swahili wrong. What came out when it tried to translate was this: “Kwa kari”, “H,h…”.

That isn’t even Swahili. It isn’t even words — it’s garbled characters. The model simply doesn’t have the Swahili language. It isn’t mistranslating; it has no vocabulary to speak with. It’s like asking a baby to read a contract aloud. Seven of the ten items were this kind of “string that isn’t even a word,” and three barely took the shape of a language. Passes: 0.

What matters here is that the same 0% can be completely different in kind. Is it 0% because the meaning was misread, or 0% because the language isn’t there at all? The first can be polished out. The second is a problem that comes before polishing. The number on the score tells you nothing about which one you have.

Pouring in the fundamentals

If the substance of the failure is “the language isn’t there,” there’s only one move: pour the fundamentals of that language into the model.

The method: before teaching it to translate, have it read a huge amount of Swahili prose (continued pretraining = adding a language to the base after the fact). A hundred million words of clean Swahili collected from the web, read through and nothing else. And to keep it from outputting only Swahili (forgetting the languages it already had), I mixed in a quarter as much Japanese and English. Half a day of GPU time; this reading step (continued pretraining) alone cost about $13. Add requantization and the step that trains translation in (imitation and CPO), and it comes to about $15 per language in total — one taxi ride.

Then I had it translate again. Accuracy: 0% → 30%.

Let me be honest about this. 30% is still a long way from the 60% ship line. On the number alone, it’s “still not good enough.” Anyone who stops the story here would call this investment a failure.

But the 30% wasn’t what I was actually looking at.

The “shape” of the failures changed

As Figure 1 at the top shows, the number (the pass count) moved only from 0 to 3. But the “shape” of the failures was replaced wholesale.

• Before: the main culprit was “strings that aren’t even words” — the language isn’t there.
• After: non-word output all but disappeared. Failures like handing back the Japanese as-is, or emitting some other language, were zero. What comes out is always fluent, correct Swahili. It’s just that the meaning sometimes slips — flipping “worthless” into “valuable,” or letting a few English words leak in.

That is the real return on this investment. The model acquired the Swahili language. From “broken” to “usable but rough.” The failures moved out of the language layer and into the meaning layer.

And that move is itself the diagnosis. When the main culprit shifts from “strings that aren’t even words” to “fluent but the meaning slips,” it means the rate limiter has moved off the model’s capacity and onto something else. Onto what?

The rate limiter was on the “teacher” side

“Fluent, but the meaning slips.” I had a suspicion about that particular way of being wrong.

The small model is trained by imitating model answers — translations produced by a larger model above it (the teacher). That’s the distillation from last time. If the student translates fluently but misses the meaning, then — maybe the teacher’s model answers are missing the meaning too.

The way to check is a controlled experiment. Leave the student (base) completely untouched and change only the teacher side. If the scores go up, that’s a signal that the rate limiter is on the teacher, not the student.

What I changed first wasn’t the teacher’s “model” but its “recipe.” Translating Japanese→Swahili directly falls apart, so I routed it through English (Japanese→English→Swahili). Sending it down the “wide road” that is English restores reading comprehension.

The result — that recipe change alone gave 9 wins / 1 loss out of 13 (3 ties). Not a byte of the student changed. That said, only 10 of the 13 were decisive, so I can’t call it settled. Even so, the signal that the rate limiter sits on the teacher’s “recipe” and not on the student’s capacity is quite strong.

If I had looked only at that first score of “30%” and concluded “the model is too small,” I would have invested in the direction of adding capacity (= a bigger model, expensive). But at least at this entry point (Japanese→Swahili), what worked was changing the teacher’s recipe (= cheap). Get the diagnosis wrong and I’d have sprinted full speed at the expensive option.

The counterintuitive part — the “teacher” is bad at this language too

There’s one more pitfall here.

“If the teacher is the rate limiter, just bring in a bigger, smarter teacher” — that’s the natural thought. What the model is imitating is a huge general-purpose model. Those can do anything. Surely Swahili is nothing.

But the huge general-purpose models are startlingly weak on low-resource languages. This isn’t my discovery; it has been demonstrated over and over in the literature. There is a report that even the very best general-purpose model beat a dedicated translation model only 5.5% of the time on English→other-language translation. Another survey found that general-purpose models lost to older translation-only models in 84% of 204 languages.

The giant models that look omnipotent are omnipotent only on the “wide paved road” of English and the major languages. Turn onto an unpaved road like Swahili and a dedicated translation model passes them without effort. The reflex of “have it imitate a big model and you’ll be fine” doesn’t work here.

So I actually made the teacher bigger. I took the translation-only model from 3B to 10B — more than 3x. The result: no significant difference (3 wins / 3 losses). More parameters left both the English leakage and the meaning flips exactly where they were. What helped wasn’t size but a different line of model designed for low-resource languages (English→Swahili passes went 0/7 → 5/7). Though that 5/7 rests on only 7 items too — “it helped” isn’t a verdict; read on the same yardstick as the 3-3 split, it’s a strong signal.

“Make the teacher bigger” isn’t something you can buy in a low-resource language. What you can buy is “pick a teacher that fits the language.”

The rate limiter differs by direction

The diagnosis taught me one more thing. The easy way to trip here is that “touching the teacher” actually covers two kinds of intervention. Changing the teacher’s “recipe” (the English pivot) and making the teacher “model” itself bigger or stronger (strengthening the teacher) are completely different moves. Mix them together and the attribution of the rate limiter looks inverted. Here are the four things I tried around the teacher, re-sorted by kind of intervention (all small n = read “it helped / it didn’t” as a signal, not a verdict).

Read the table by kind of intervention and the identity of the rate limiter splits cleanly by direction.

• English→Swahili: improve either the teacher’s “recipe” or its “model” and it goes up (teacher-bound = ③ 0/7→5/7). This is a problem of spelling and phrasing on the output side.
• Japanese→Swahili: the English pivot, which changes the “recipe,” works (① 9 wins / 1 loss), but no amount of making the “model” bigger or stronger gets past the plateau (②③④). The deep understanding it takes to “read through” dropped subjects and idioms in Japanese demands the model’s capacity itself (capacity-bound).

So the previous section’s “the teacher’s ‘recipe’ is the rate limiter” (① ja→sw) and “strengthening the teacher model doesn’t help” (④ the same ja→sw) don’t contradict each other. What works is the “recipe”; what plateaus is “strengthening the model” — same direction, different intervention, different rate limiter. Had I rolled these two into a single “swap the teacher,” I’d have gotten the attribution wrong. Had I filed the whole thing under “low-resource languages are just hard,” I would never have seen the split at all.

Low-resource isn’t a tail you give up on

Let me put that in business terms.

For most teams, low-resource languages are the “long tail you cut off.” Few users, AI is bad at them, the cost doesn’t add up. The edge of the distribution.

For LYR it’s different. “There’s content you can’t see because language is in the way” — breaking that is why LYR exists. The languages the world is worst at are exactly where LYR should be strongest.

And what this round showed is that this isn’t willpower — it pencils out economically. About $15 per language jumps a model from “doesn’t have this language” to “usable but rough.” That stacks up straightforwardly as “$15 × number of languages” — line up 50 hard languages and it’s $750; 100 and it’s $1,500. Less than one engineer-day of salary. After that, you diagnose the rate limiter and invest on the correct side. As long as the diagnosis is right, low-resource languages can be promoted to first-class citizens cheaply, one at a time.

The starting point was always the same — look at what the failures are made of, not at the number on the score.

Lessons

1. Look at the “distribution” of failures, not the absolute score. The same “0%” is one thing when the model doesn’t have the language and another when it misread the meaning. Same number, opposite moves. Only by looking inside the failures do you find out what the rate limiter really is. (The manifesto’s first principle, “Measure First” — except that you measure how it got it wrong, not what score it got.)

2. The change in the “shape” of the failures is what gives you the diagnosis. After pouring in the fundamentals, the failures moved from “strings that aren’t even words” to “fluent but the meaning slips.” That’s more important information than the score going up — because it’s a strong clue that the rate limiter moved from capacity to the teacher side (recipe, model choice). Read the shift, not the size of the gain.

3. In low-resource languages a “big teacher” doesn’t necessarily help — and they aren’t a tail to give up on. Tripling the teacher produced no difference; the design built for low-resource won. What you should buy isn’t “size” but “a design that fits the language.” About $15 per language jumps it from “doesn’t have it” to “usable but rough.” After that, separate the rate limiter out direction by direction, and knock out the cheap side (teacher choice) first.

---

Appendix: raw data (per experiment)

Shared conditions: our own 8B model, Swahili (0% at INT4 quantization / 10% at BF16 = base-bound, already established). Evaluated at the deploy quantization as shipped, comparing before and after within the same judge round as a controlled comparison. Every judgment is small n = read the point estimates as a signal, not a settled result.

Related: the basics of distillation and SFT/CPO are in Raising a small AI into a specialist at one job. (Note: the Swahili CPT base forgets other languages, so shipping keeps them separate by switching the base per language. That serving design is a separate article.)