AI apps get easier once you think in CPU and RAM

How many glossary entries to carry. How far back to go through the history. Whether to pass a genre tag. These decisions looked like separate problems; they were all the same question — how do you hold memory?

So I settled on one vocabulary. The LLM is a borrowed CPU; context (the surrounding text and data) is RAM I design myself. That alone folded the dithering into a single question: which level of cache is this?

And it settles where to invest. Anyone else can rent the same CPU you do, and it gets retired. What survives a swap is the design on the RAM side. But more context is not automatically better — I stacked +120% more history, paid +42% in cost, watched quality plateau, and reverted.

For a while, I kept working out the same kind of decision from scratch every single time. How much of the glossary to carry. How many subtitle lines back to go through the history. Whether to add a genre tag for the title to the prompt. Whether to pass scene-boundary information. Every one of them is a question about what to hand the LLM, and yet I was groping at each one separately, as if they were different problems.

One day I laid them out side by side, looked at them, and stopped. These are all the same question: how do you hold memory? Which information, how close, and for how long — it has exactly the same shape as the problem computers solved decades ago.

So I settled on one vocabulary. The LLM is a borrowed CPU. Context (the surrounding text and data) is RAM I design myself. That alone made most of the dithering go away. Where I landed was designing context as a four-level cache hierarchy. This article is about how that one metaphor organized the entire design.

The LLM is a CPU you’re renting

An LLM is a compute unit. It takes context (the input), computes a translation (the output), and hands it back. That’s the same role a CPU plays when it takes instructions and data and returns a result.

And what settles it is this — it’s borrowed. I’m not the one designing the foundation model. I rent it from outside and swap it out at every generational turn. In fact, as I wrote in Part 4, the model I was relying on got retired, got a price increase, and disappeared with no regard for my circumstances. A borrowed CPU has to go back eventually.

Once you think of it as a CPU, its properties carry over intact. Faster compute is always welcome (latency is decided by the byte count of the weights), but whichever CPU you swap in, the substance of the work you hand it doesn’t change. Qwen or Llama, the job of translating subtitles is the same. So the CPU itself is, in the end, nothing more than an interchangeable part. (Though “the substance of the work” here means the abstract role — “translate subtitles.” The concrete prompt format you hand that CPU differs from model to model, and that part gets rewritten with every swap — I’ll come back to that boundary at the end.)

So what’s the part you can’t swap?

So whose RAM is it?

To get a CPU to do work, you have to put data within its reach. For translation: what’s on the screen right now, how we decided to translate this title’s proper nouns, what the previous line of dialogue was — you structure all of that and lay it out where the CPU can reach it. That’s RAM’s job.

And this RAM is yours to design. Which information you pick up, how you lay it out, how much you hold, when you throw it away. None of that is borrowed; it’s 100% an in-house design decision.

(The finer details map one-to-one in the same vocabulary too: prefetch, cache eviction, the memory controller, a purpose-built ASIC. The full mapping is in the appendix.)

Making the CPU faster — renting a better model — does help, no question. But anyone else can play that same move. Where the difference comes from is outside the CPU: what data you can get within its reach, and how well targeted that data is. This was the manifesto’s second principle, Hunt the Constraint — the rate limiter is usually outside the model — exactly. The model is a small slice of the whole. The way to win is usually lying outside it.

Thinking in a cache hierarchy

Once you know it’s RAM, the next question sets itself: how do you carve up the memory hierarchy? Closer to the CPU is faster but smaller and more expensive; further out is slower but bigger and cheaper — that classic trade-off carries straight over into context design.

• C1 (straight into the prompt) = the L1 cache. The fastest place, wired directly to the CPU, alive only for the duration of one request. The text blocks on the screen, the matching glossary entries, the instruction about speech style — every one of those tokens goes into the LLM’s input as-is. This is the fastest level, and the narrowest.
• C2 (in-session memory) = main memory. Alive only while the app is running. The glossary cache (LRU = throw out the least recently used first; capped at 200 entries), and the recent history in subtitle mode (3 lines). A hit gets promoted into C1.
• C3 (persistent result cache) = the disk cache. It survives a reboot of the device. Keyed on the MD5 of the input (a fingerprint of the content = a short fixed-length hash), it pulls back a past translation verbatim. On a hit, the answer comes back without touching the CPU (the LLM). To be honest, the RAM metaphor creaks a little here — it isn’t about keeping data close by, it’s a different kind of saving: not doing the same computation twice (memoization). Still, the skeleton of the hierarchy — stash it in a far tier, pull it up to the fastest tier when you need it — is the same, so I treat it as C3. I deliberately don’t call it a “KV cache,” because this has nothing whatsoever to do with the KV cache inside a transformer (the intermediate state of the attention mechanism): it’s a cache of the results themselves.
• Cold (eval data, user logs) = the archive. Gigabytes of asset, but it never reaches the LLM directly. It’s the raw material for regression tests, for training a future in-house model, and for search that pulls things up into C2/C3.

Once those four levels are fixed, the first question I ask about any new context idea is always the same one: is this C1, C2, C3, or Cold? A decision I used to work out from scratch every time became a matter of choosing where to put it. One vocabulary took the variability out of the decisions.

”More context” backfired

Where the RAM metaphor really earned its keep was when it let me explain a failure correctly.

In Page translation, I once increased the amount of history I passed as context. The naive thought was: give it more context and the translation should get better. Input tokens swelled from 521 to 1212 — +120% — and cost went up +42% on average (+21–62% depending on the article). For a heavy user (200 pages/day), that was on the scale of eating 27% of the subscription gross margin. Prefill (reading the input in) was about +50ms.

The return didn’t come close to justifying that. What I wanted out of stacking history was consistency in how proper nouns get translated — and that benefit, as I found out later, could be had orders of magnitude more cheaply with a small glossary cache that pulls out just the proper nouns (at most 10 entries injected into the prompt). Stacking the whole history into every request only made cost grow linearly, while the quality gain had long since plateaued. So I reverted.

At first I was tempted to liken this to the memory wall (the hardware phenomenon where memory bandwidth fails to keep up with the growth in CPU compute speed, so the performance gap between the two keeps widening). But that isn’t accurate — write it that way and any CTO who knows hardware sees straight through it. The memory wall is not about things getting slower when you add capacity. Adding RAM doesn’t increase latency. What was happening here was a different dynamic: prefill cost grows linearly with the input token count, while the quality return diminishes. Not a wall — a question of where two lines with different slopes cross: cost that keeps rising, and quality that plateaus.

“More context” is not an unconditional good. There’s a limit to how much you should load into one request — in computing terms, the working set, the live range of data a process needs at once — and you have to design that limit deliberately. In LYR the limits are explicit: at most 10 glossary entries, at most 3 history items. That cap isn’t me “being stingy”; it’s drawing a line just short of where cost and quality cross. Without the vocabulary of the memory hierarchy, I’d have filed this failure away as “for some reason adding context made things worse,” cause unknown.

(The same vocabulary also makes sense of why that crossing point sits in a different place for Page translation than for subtitle translation. Page is input-dominant, subtitles are output-dominant, so cost and the ceiling on how much context you can carry are different animals. Even within the same “RAM design,” a different way of using the CPU calls for a different optimal hierarchy.)

Buy the CPU, design the RAM in-house

At this point the investment principle comes down to one line. Buy the CPU. Design the RAM in-house.

The LLM (the CPU) can be outsourced. Generations turn over fast, and there’s little value in chasing them yourself (and if you do chase, that comes only once the economics flip). The cache hierarchy, prefetch, eviction and working-set management, on the other hand, all accumulate as internal assets.

That asymmetry is what pays off. A borrowed CPU disappears when it’s retired — and sometimes it takes the optimizations you piled on top of it with it, un-transferable to the next model. But the asset on the RAM side — the design of which context you take from the environment, how you structure it, and which tier you put it in — survives a CPU swap intact. The glossary, the translation cache, the evaluation dataset: they keep their value no matter what the model is.

That said, I want to draw the boundary of that asset honestly. What survives is the structuring design — which context you select, how you prioritize it, how many tiers you put it in — while the concrete prompt format that actually hands it to the model is tightly bound to the CPU (the model). In fact, when the model disappeared in Part 4, the layer I ended up rebuilding was exactly that format layer. What you lose is the translation (the model-dependent form); what stays is the design (the meaning — which memories you hold, and how). That’s where the boundary sits. More than that: on the day I do build my own CPU (a purpose-built model), the data stacked up in that Cold tier becomes the fuel for training it, as-is.

So the investment leans toward the RAM side. The flashy part is the CPU — the new model — but the part you never have to give back is the RAM: your own context and data hierarchy.

Lessons

1. One good metaphor organizes the entire design. Decisions I’d been agonizing over separately — how many glossary entries, how deep the history, tag or no tag — folded into a single question under one vocabulary, “LLM = CPU, context = RAM”: is this C1, C2, C3, or Cold? A good metaphor isn’t decoration, it’s a decision-making tool.

2. The CPU (a borrowed LLM) is interchangeable; the RAM (your own context and data hierarchy) is the competitive advantage. Anyone else can rent the same model. The difference comes from outside the model — which context you can get within the CPU’s reach, and how well targeted it is. It’s the design-side version of the manifesto’s second principle, Hunt the Constraint.

3. More context isn’t automatically better. Borrowed things go back; assets stay. Cost grows linearly with input tokens while the quality return diminishes (+120% history bought +42% cost and +50ms prefill; quality plateaued, and I reverted it). The working set isn’t stinginess, it’s design. And the CPU gets retired, but the design and the data you’ve stacked up on the RAM side survive the swap — which is why the investment leans to the RAM side.

---

Appendix: measured values / raw data

Metaphor mapping: CPU = the LLM (borrowed, turns over each generation) / RAM capacity = the context window (8k–128k tokens) / RAM bandwidth = input read speed (about 750 tok/s on the 8B) / prefetch = the speculative OCR cache / eviction = glossary LRU capped at 200, sliding window over history / purpose-built ASIC = in-house OCR / a future custom CPU = an in-house subtitle foundation model. Related: the economics of owning your AI / the quantization that makes the CPU faster.