AI performance is decided by the order of your decisions
Four principles that decide an AI product's performance, arrived at by building LYR alone
When an AI product is “slow,” “expensive,” or “not good enough,” most teams suspect the model first. A bigger model, a newer model, a higher-priced plan.
But what decided things wasn’t which model I picked — it was the order in which I made decisions. Sometimes the answer sits outside the model (waiting, placement), sometimes it sits on the model side — but even on the model side, it’s usually not “make it bigger,” it’s “make it narrower” (specialization).
These four principles came out of building LYR, an AI translation app, on my own — cycling through failure and measurement over and over. None of this is about new technology. It’s about the order of your decisions. If your team is about to invest in AI performance, it’s worth running one diagnosis in this order first.
- Measure First — don’t guess where the time and money go
- Hunt the Constraint — it can still be cheaper and faster
- Optimize the Right Proxy for UX — pick the proxy that stands in for the experience
- Eliminate, Don’t Accelerate — cut the work instead of speeding it up
1. Measure First
“The AI is slow / the AI is expensive” almost always starts from an assumption. I kept getting my read on the slowness wrong.
For a long time I assumed the reason translation was slow was the “model.” Then I broke the pipeline down into segments and measured it for the first time, and the real culprit turned out to be somewhere else entirely. If I had switched to a faster model without measuring, the perceived speed would barely have moved.
And it isn’t only results you measure. Measure the premises. Kill the “probably” in “this is probably the slow part” before you spend anything on it. And measurement isn’t a one-shot thing: start by running the same code twice and checking that the two runs agree — because a duplicated benchmark lies to you quietly.
→ Before you commit budget, build something that shows you where the time and the money actually go. You can’t fix what you can’t see.
Examples: a single line of translation emitting hundreds of tokens of “reasoning” — that was what the slowness actually was (Part 1). / A duplicated bench that couldn’t even reproduce itself across two runs of the same code (Harness determinism).
2. Hunt the Constraint
Every system has a rate-limiting step (the bottleneck) somewhere, in exactly one place. Which means there is always room left to make it cheaper and faster.
The main culprit behind latency and cost is often not the model’s computation at all. Ironically, the lighter you make the compute, the more the waiting and the placement in front of it and behind it come to dominate the whole. And the rate-limiting step moves — last year’s right answer becomes this year’s wrong one. So numbers from an isolated bench can’t be trusted; you have to re-measure “where is the constraint right now” against the full production picture. I’ve had edge inference that should have been faster in isolation come out slower in production.
Specialization is a tool in this same context. Strip off the general-purpose model’s “tax” and narrow it to one task, and a far smaller model reaches the quality of the strongest models currently in service — more cheaply and more quickly. You throw away “bigger is better” in order to get cheap and fast.
→ “This is as good as it gets” only ever means “I haven’t found the rate-limiting step yet.” Look at the whole, and knock out the current constraint, one at a time. Knock it out and the next one appears.
Examples: about 90% of E2E latency was queueing and distance, not generation (Part 2). / Moving the inference server from the US to Tokyo cut about 160ms (Part 8). / A small model specialized for Live subtitles drew level with a 27B-class model in service (Specialization in practice).
3. Optimize the Right Proxy for UX
The goal isn’t a fast model, it’s a product that feels fast. But you can’t measure the experience itself directly.
So you pick one metric that proxies well for the experience, and you judge every other initiative by how much it contributes to that proxy metric. Get this wrong and everything goes wrong with it — if the proxy is off target, you can optimize as hard as you like and the experience won’t move a millimeter (I once optimized against pass-fail-only scoring and left everything that definition doesn’t capture sitting there, flat — Goodhart). And “the metric moved” is not the same as “it worked.” Contribution has to be measured against a control. If a cheap control beats an expensive change on every axis, that change added nothing.
A good proxy metric doesn’t look at the average. What users remember isn’t the average — it’s the one time it was worst. The rate of that worst case — the tail — is what belongs in your proxy for the experience. Even at the same average, a design that occasionally breaks catastrophically was disqualified as a product.
→ Aim your optimization at a proxy metric for what the user feels, not at a bench number. Pick that proxy correctly, and argue in terms of contribution.
Examples: the deadline for “fast enough” moved between 400 and 750ms depending on the content — a single bench value misses on both sides (Part 7). / That “97% improvement” had a circular proxy metric, and I threw it away (The 97% I threw away). / Reducing the worst case worked better than chasing the best translation (Raising the floor).
4. Eliminate, Don’t Accelerate
The fastest work is the work you never run. So is the cheapest.
When you think about optimization, the first thing you look for is a way to do the same job faster. A faster GPU, a compressed model, a cache that actually hits. But acceleration usually gives you back some tens of percent. Eliminate that job entirely and the reduction is 100%. For the same engineering effort, looking first for what you can delete pays better.
The work to suspect most is the work you added with good intentions. A step you added for quality can turn out to add nothing. Work that isn’t earning its keep is pure liability — slow and expensive and nothing else. This is where the earlier three principles pay off: break it down and measure, and you can see the work you can delete without losing anything; take a control, and you can see the work you added that isn’t doing anything. Deletion is the natural consequence of measurement.
Of course you can’t delete everything. What matters is the order. Put “do we need this work at all?” first and “how do we make it faster?” second. Reverse them and you’ll spend your best effort optimizing work you could have deleted.
→ Ask “is this needed?” first. If it isn’t, delete it — deleted work improves both speed and cost by 100%. Only once you know it’s needed do you start thinking about how to make it faster.
Examples: making on-device OCR faster got -51%; cutting how often it’s called dropped the power budget by an order of magnitude (Awake Layer). / Hundreds of tokens of “reasoning” for a single line of translation. Instead of hunting for a faster model I made it stop thinking, and 27x the tokens and 4x the latency went back to 1x (Part 1). / No matter how I added context, quality didn’t move — meaning that injection was work I could delete (Context vs. capacity). / If a cheap control beats an expensive change on every axis, that change added nothing (Take a control).
Not new technology — the order of your decisions
These four aren’t flashy new technology. They’re an order of decisions: measure, break it down, delete, then choose where to aim.
What I find interesting is that none of it was theory assembled at a desk. I picked these up one at a time, by feel, while building LYR alone and cycling through failure and measurement. Looking back, though, they line up uncannily well with what The Goal (Goldratt) told manufacturing forty years ago — speeding up every machine doesn’t speed up the factory; the constraint on the whole (the bottleneck) is what determines the system’s performance (the Theory of Constraints, TOC). I didn’t set out to apply TOC. The failures I stepped on one by one in the field just happened to converge on the same place. Swap the “things” flowing through a factory for the “requests” flowing through a system and the skeleton doesn’t change: look at the whole flow and knock out the constraint. I only realized that afterwards — that was the order it happened in.
AI performance isn’t decided by the model alone. How you measure, how you break things down, the order you invest in — that accumulation of decisions is what decides the user experience in the end.
On this site I’ll be publishing the experiments and failures behind those decisions, and what came out of them. How I tracked down the real culprit behind the slowness. How my benchmarks lied to me. How I turned out to be the bottleneck myself — the actual process, numbers and mistakes included.
This record is published as LYR Performance. Thoughts and corrections are welcome via the contact page (Japanese). Diagnosis and hands-on support for AI system optimization will start once the company is incorporated.