"Add context and the model gets smarter" was half a lie

“Add context and the model gets smarter” was only half true.

The gaps it fills are the ones you could look up. Speaker, proper nouns, dropped subjects — context filled all of those (13-0). But deep reading comprehension and register don’t close even with the answer sitting right in front of the model — that is a question of capacity, not of raw material.

In a production-equivalent full factorial, the loss rate against the 27B stayed at 76% / 75% / 76% no matter how I supplied the context. And the most expensive move (baking it into the weights) couldn’t beat the cheapest one (pasting it into the prompt on the spot) — 94% identical outputs. Which half of the problem you have is something you can tell apart before you invest.

“If the model isn’t smart enough, give it more context” — that’s the most popular prescription in AI development right now. RAG (retrieving documents and handing them to the model along with the prompt) is an extension of the same idea. Give it the surrounding information and the model will “read the context” and answer correctly.

I assumed the same thing. In subtitle translation, my own small model (8B = 8 billion parameters) loses to a large model (27B = 27 billion) on reading comprehension. So — if I hand it the lines before and after — surely the small model catches up? Before investing, I measured.

The result split cleanly down the middle. Half of it was true, and half of it was a lie. And if I hadn’t caught the lie, I was about to pour weeks into a training investment that would have bought nothing. And more than that — which half your problem falls on is something you can tell apart before you invest. If it’s a gap you could look up, context fills it cheaply; if it’s a shortfall in comprehension capacity, only size moves it. That distinction is what decides whether you bet on RAG or on the model.

First — context really did work

The first experiment was to measure the upper bound: if the context were perfect, how much better does the small model get? Speaker gender, the correct reading of proper nouns, the dropped subject — I prepared the state of the scene omnisciently, then pitted the version handed perfect context against the current version handed none, on the same 8B model.

The judging was pairwise (put two translations side by side and pick the better one, item by item; the sign wobbles less than it does with an overall score). Measured across four episodes of one drama, 86 items —

With perfect context: 40 wins / without: 22 wins / 24 ties. 65% on a decisive-only basis (ties excluded from the denominator), statistically significant (p≈0.03). Context really did work. So far, the prescription holds.

— But break down what worked

The trouble started here. I split out what had worked, dimension by dimension, and recounted.

(The dimensions aren’t exhaustive, and one item can fall under several of them. So the dimensions don’t add up to the overall 40 wins / 22 losses.)

As you can see, almost all of what worked was concentrated in the top three dimensions — speaker gender, proper nouns, dropped subjects. And those three have something in common: every one of them is something you could look up. Who is speaking, how the name is read, who the dropped subject is. It’s all there in the script if you go and look — these are gaps in knowledge.

On the other side, register for the scene was a wash and semantic correctness didn’t move at all. Put the raw material — the context — right in front of it, and the 8B still can’t translate the difference. This isn’t a gap in knowledge; it’s a matter of capacity for comprehension and expression.

Incidentally, until not long ago I had concluded that “context doesn’t help — it’s a draw.” That was because I was measuring impoverished context with a wobbling score. Mix a component that works (gender) and one that doesn’t (register) into a single overall score and they cancel each other out, and it looks like a draw. Only by splitting it out by dimension did the 13-0 sign become visible (the manifesto’s first principle, Measure First — how you measure included).

The main question — does context let the small model catch the large one?

The upper-bound experiment told me that with perfect context there are dimensions where it works. Now the practical question: if I hand it the surrounding lines, does the 8B catch the 27B? This is the fork in the investment decision.

I ran my own 8B against the large 27B on 434 manga items, Japanese→English. Exactly the same prompt form as production, generation randomness pinned to zero (temp0 = the same answer comes out every time; this keeps the experiment’s variables clean). On top of that I compared three ways of supplying the context, with everything else held constant:

• A: no context (current)
• B: context injected into the prompt on the spot
• C: context baked into the weights by training (context-aware CPO = a version trained on preferences with the context attached)

Then I ran each against the 27B and looked at the loss rate against the 27B (decisive-only basis, ties excluded).

No context 76% → injected into the prompt 75% → baked into the weights 76% (this full factorial is all one and the same temp0 run). However I supplied the context, the gap to the 27B didn’t budge. On this temp0, decisive-only basis, the number of items where the 8B “won strongly” against the 27B was, across all three conditions, zero.

Even with the answer right in front of it, it can’t use it

Why doesn’t it move? It clicked once I lined up the outputs from the version regenerated with the context actually handed over (B, injected into the prompt). Even with the correct context right in front of it, the 8B can’t put it to use. All of the following are outputs produced with that context.

• 「太陽に溶かされて」 (“melted by the sun” — Japanese leaves the subject unsaid; the preceding line, a poem about wings, had been handed over): the 8B picked the wrong subject and gave “I’d be melted”; the 27B correctly gave “they melt”.
• 「天文部を頼む」 (“take care of the astronomy club”) (the context says the request is being made to you): the 8B translated the direction backwards; the 27B correctly gave “I want you to help”.
• The character name 「相川仁」 (the given-name kanji 仁 has several valid readings): the 8B missed the reading with “Masaru”; the 27B gave “Jin”.

The raw material — the context — was handed over identically in both cases. The gap shows up anyway. Which means the ability to use context is itself rate-limited by the model’s capacity. Put the knowledge right in front of it, and if the capacity to read it isn’t there, it can’t be used.

There was a clinching symptom, too. Given long context, the 8B broke the output format on 19 items (the 27B, on 4). The limit of capacity shows up not only in accuracy but as a drop in robustness like this.

”Baking it into the weights” didn’t beat “handing it over on the spot” by an inch

Last, I tested the most expensive option — baking the context into the weights (C). Instead of putting it into the prompt on the spot (B), train preference on data that carries the context, and maybe the model learns to use context better. That was the hope.

The result was blunt. The trained version (C) and the hand-it-over-on-the-spot version (B) produced 94% identical outputs at temp0 (the 6% that differed was trivial). Going to the trouble of training on 256 pairs of material got me effectively the same thing as pasting the context into the prompt on the spot. → This training was not adopted.

Context engineering is necessary. But “hand it to the prompt on the spot” is enough — there was no value in baking it into the weights. And above all: however you engineer the context, an 8B on its own doesn’t reach a 27B. Closing that gap was the job of capacity — a bigger base — not of context (the manifesto’s second principle, it can still be cheaper and faster — here: don’t optimize the context alone. The rate limiter was on the capacity side, not the context side).

Lessons

1. Context solves missing knowledge; the wall in front of understanding is capacity. Gaps you could look up — speaker, proper nouns, dropped subjects — get filled by context or RAG (13-0, significant). But deep reading comprehension and register don’t get filled by handing over material — they need model capacity. RAG is a specific cure for the former and does nothing for the latter. Separate the two before you invest.

2. Don’t measure what works and what’s capacity-bound mixed together. Mix a dimension that works (gender 8-0) with one that doesn’t (register 4-4) into a single overall score and they cancel out, it looks like a draw, and you read the sign wrong. Only by splitting it out by dimension can you see what works and what doesn’t (the manifesto’s first principle). I did in fact measure them mixed once, and wrongly concluded that context doesn’t help.

3. Sometimes the most expensive option doesn’t beat the cheapest one. Baking context into the weights couldn’t beat pasting it into the prompt — 94% identical outputs. And on top of that, handing over the knowledge is useless if the model can’t read it — with the correct context right in front of it, the 8B still missed subjects and proper nouns. Measure the cheap move and the expensive move on the same footing, before you invest.

Add context and the model gets smarter — that’s only half true. What gets filled is the gaps you could look up. The wall in front of understanding is, in the end, model capacity, and just as chasing smarts alone leaves you with something unusable in real time, chasing context alone won’t get you over the capacity wall.

Appendix: raw data

Only the measurement conditions, judging methods, and cross-checks against a separate run that aren’t in the body.

The output examples in the body (「太陽に溶かされて」, 「天文部を頼む」, 「相川仁」) are all outputs from the version regenerated with the context injected into the prompt.