LYR Performance

Your AI can becheaper and faster

Slow, expensive, not good enough — most of it moves without switching models. What I learned building LYR, an AI translation product, on my own: the decisions behind latency, cost and quality. The method first, then the cases it came from.

#000 Manifesto — start here

AI performance is decided by the order of your decisions

Four principles that decide an AI product's performance, arrived at by building LYR alone

Read the four principles →

What follows is those four principles applied to real cases — with the numbers, and with the failures left in.

The road measurement took me down

#001 8 min One setting took output tokens to 1/27, latency to 1/4, and cost to 1/5 Hundreds of tokens of "reasoning" for one line of translation. 96% of the generation wasn't the translation — it was the model thinking before it answered. One setting stopped the reasoning: output tokens to 1/27, latency to 1/4, cost to 1/5. I never changed the model.
#002 7 min Last year's right answer becomes this year's wrong one I'd made translation faster by trimming its output. A few months later I measured again and the breakdown had flipped — generation was 10%, waiting and distance were 90%. Then I tried to eliminate the round trip with edge inference and made everything slower.
#003 7 min Some AI models are disqualified before you ever measure quality Model selection has a "gate" that comes before you measure quality. The smarter the model, the more it "thinks" before answering — which disqualifies it for real-time use. And once I re-measured with the gate held constant, it was the received wisdom, "smaller means faster," that fell apart.
#004 6 min Borrowed AI disappears overnight The model you picked and optimized around gets retired by the provider one morning. The whole cheap tier disappears, and the only survivor is the most expensive one. I could control latency and cost — the foundation was the one thing I didn't control.
#005 12 min I brought AI in-house: E2E ~500ms → 120ms, cost down to 1/3–1/10 Self-hosting isn't only about sovereignty. Dedicated capacity kills the queue and proximity kills the distance, both at once. But run it stock on a cheap GPU and compute becomes the new bottleneck. Cost projections, and latency versus before.
#006 8 min You are the bottleneck Four months, 122 instruments, one realization — measure-first is really about speed, not accuracy, and the first bottleneck was me.

Infrastructure — deadlines for speed, and floors you cannot move

#007 7 min Don't let a benchmark decide what "fast enough" means Decide "how many ms counts as fast" from a benchmark and you will always miss. For subtitle translation the target was a deadline worked backwards from how long the subtitle stays on screen. Speed isn't a continuous value — it's a cliff.
#008 5 min Geography is the latency floor you can't move Make the model faster, quantize it — distance is the one thing that never shrinks. Measuring round-trip times from Tokyo to data centers around the world exposed a floor you can't move: 5ms in Tokyo, 373ms to Cape Town. The biggest lever on latency wasn't performance, it was placement.
#009 11 min I moved one stateful component and perceived latency went from 469ms to 252ms I thought I was cutting latency by moving inference closer to the user, but the state-management component wedged in between (a Cloudflare Durable Object) was making a round trip across the Pacific. Break it down into segments and the round trip really was gone — yet a different segment had jumped to 175ms. In a distributed system, local optimization gets cancelled out somewhere else. Nobody could have seen it until I measured the whole thing.
#010 13 min AI apps get easier once you think in CPU and RAM Treat the foundation LLM as a borrowed CPU and your context and data as a RAM hierarchy you design yourself, and cache design, context design and the decision to self-host all fall under one vocabulary. The CPU gets retired; the RAM accumulates as an asset.

Models — smaller, faster, sharper

#011 7 min Compress a model to a quarter of its size and its smarts barely drop Repack an AI model into a lighter form and generation gets 1.8x faster, the volume it can handle and the cost both improve 4x — and accuracy barely moves. The real bottleneck is memory bandwidth; here, in business terms, is how quantization solves it.
#012 9 min Taking a small AI's accuracy from 42% to 84% — and on some tasks, level with the giants An un-fine-tuned small model scored only 42% accuracy on translation. I pulled it up to 84% — the same as the retired original model — and on some tasks all the way to level with the giant model currently in service. You don't need general-purpose smarts. Here, in business terms, is how to train in a single job, and where that stops working.
#013 8 min The AI that was smart on the bench suddenly started making mistakes in production A fine-tuned AI model was a star pupil on the benchmark, then started mixing up subjects and flipping negations the moment I put it into production. The cause was neither capability nor training — it was a precision mismatch between "the environment I tested in" and "the environment I deployed to." A record of the discovery, through re-measuring under the same conditions and training under the same conditions.
#014 9 min The "97% improvement" I threw away An automated measurement came back with good news — a 97% improvement. When a number is too good, how should the business treat it? The autopsy of a false victory, performed by going through the raw output one page at a time before accepting anything.
#015 10 min "Add context and the model gets smarter" was half a lie Hand a small model the surrounding lines and it will read the context and get smarter — that's what I assumed, so I measured it. Half of it was true and half of it was a lie. Context fills only the gaps you could look up: speaker, proper nouns, dropped subjects. Deep reading comprehension doesn't close — the capacity to use the answer isn't there even with the answer in front of it. The story of a production-equivalent, temp0, full-factorial experiment in which no way of supplying context moved the quality gap at all.
#016 11 min I cut the bottom-rank rate from 64% to 24%, and held the top-rank loss to 12pt A heavily compressed model produces the best translations, and also catastrophic ones. On average the difference is almost nothing. But what breaks the user experience in a subtitle product isn't the average or the best — it's one single worst case. On choosing to raise the floor instead of the ceiling.
#017 14 min Swahili from 0% to 30%. But the value wasn't the score — it was that the failures changed Swahili is one of the languages AI is worst at. The base model's accuracy was 0%. Even after instilling the fundamentals, 30%. Far from a pass. But what was worth more than the 30% gain was that the failures themselves changed. Look at the distribution of failures, not the absolute score — how I worked out what was really rate-limiting in a low-resource language.
#018 12 min Let the AI pick a translation better than the "right answer" "Prepare one correct set of model answers and have the model imitate them" — that's the conventional wisdom in AI training. But have the teacher produce N candidates and let a quality estimator (QE) pick the best, and you get model answers that beat the teacher's first shot — the "next move" I expect the most from in LYR. And what I'm sure of is the flip side: if the yardstick you select with is off target, you lose outright to one clean set of model answers (measured: 82% → 73–77%). Breaking the faith in the one right answer, and honestly measuring both what it buys you and where it stops.
#019 10 min A small specialist drew level with the strongest model in service You don't have to fight a general-purpose giant head-on. Narrow to a single task — Live subtitles — sharpen there, and a model about a third the size can draw level with the strongest class currently in service. This is the story of taking Live's success rate from 90 to 96 and tying the 27B in service. But there was a price: specialize on one mode and another mode (Page/Manga) breaks by −4. Specialization is a choice about allocation; it's not a free lunch.
#020 15 min Let another AI write your teacher data Training a small specialist AI takes "gold data," and everyone assumes people have to write it. At LYR I had a different, larger AI write the right answers instead — and accuracy on Japanese→English, the direction it was worst at, jumped from 42% to 84% on straight imitation alone. The ceiling beyond that is hallucination coming out of the student's (4B) comprehension capacity. The next move for pushing past it is best-of-N — have the teacher write N candidates and pick the good one. This is the methodology for producing teacher data automatically.
#030 8 min Generation speed is decided by the byte count of the weights, not the parameter count Measured on the same GPU, 4B-INT8 and 8B-INT4 — a 2x difference in parameters — ran at almost the same speed (5.98ms against 6.50ms per token). Every token generated reads out all of the weights, so what decides speed isn't the amount of compute or the parameter count but the byte count. Step outside the premises, though, and the law breaks; get the measurement method wrong and a 2.35x gap disappears, or inverts.
#032 22 min A general-purpose model is too wasteful a vessel for subtitles Even a raw 8B with no compression at all left 15 critical mistranslations out of 97. The 27B, more than three times bigger, had 4 (p=0.015). Varying the strength of the compression doesn't move that gap. So the next suspect is the vessel itself — subtitles are short, constrained, bounded-context, and yet we use, whole, a vessel designed for free generation of arbitrary length. What I expect from specialization isn't only speed: emotional register, and accuracy from translating while looking at the video. Here is the design hypothesis, and honestly, what would falsify it.

Measure & tune — measure it right, then fit the settings

#021 5 min Are you still picking that setting by gut feel? Thresholds, resolutions, batch sizes — most of them are picked by gut feel. Once verification gets cheap, you stop picking a single point and start sweeping the range to see the shape of the curve. A sweet spot only becomes visible when you measure the whole thing.
#022 6 min When "which one is faster?" is the wrong question to begin with Speed vs accuracy, quality vs cost. When objectives conflict, "which one is better" has no answer. There is only a frontier. Stop hunting for a winner and pick an operating point. If a cheap parameter change "dominates" an expensive improvement, that improvement is nothing.
#023 8 min If you let an AI do the scoring, start by doubting the scores When I had an AI score quality, just measuring the same thing twice moved the number by 6 points. The "+7-point improvement" that wobble produced turned out to be a dead heat head-to-head. Calibrate the judge itself first, or your improvements are chasing illusions.
#024 9 min A duplicated benchmark lies to you quietly A bench duplicated by imitating only the main logic kept measuring a setting production had dropped long ago. Rewriting it to call the production functions directly recovered the numbers by +26 to +31pp. And checking, before any A/B, that "the same code run twice agrees" revealed what the notorious "±20pp of noise" really was.
#025 9 min That "accuracy" is a definition you chose When you say "improve accuracy," that "accuracy" is nothing more than a definition someone chose. Optimize against a judge that only returns pass or fail, and the value that definition doesn't capture is the one thing left behind, flat. The ceiling on improvement isn't the model — it's the impoverished definition.
#026 7 min Don't explain the "why" until you've taken a control The number moved, it got better, here's the mechanism — you'll want to say all of it. Before you do, take one control (a comparison that differs from the treatment in the one factor you're testing and is matched on everything else). If a cheap control beats an expensive change on every axis, that change added nothing. What looked like an improvement was confounding.
#027 8 min Don't stop at measuring a single mode I pushed the Live translation number from 90 to 96 and nearly declared it an improvement. But when I re-measured across all modes, Page and Manga had each quietly dropped 4 points. If your product is used in more than one mode, you don't get to declare victory on one mode's number. The trade-off of a regime shift only becomes visible once you measure every mode.
#031 8 min Five rebuilds and it still never beat chance — the cause was the labels, not the model Change the inputs, change the architecture, rebuild it five times, and accuracy still doesn't beat chance. The cause wasn't the model — it was ground-truth labels that were off by about a second. And broken labels cost you twice — they don't just produce a bad model, they had also been undervaluing a good implementation I already had as '25% misses'.

OCR — do less work, on the device

#028 10 min I cut OCR's power budget by an order of magnitude — not by making it faster, but by calling it less OCR reading subtitles on the device eats battery. My first investment went into making OCR faster. It did get 51% faster. The order of magnitude didn't move. What moved it was letting a 0.5ms always-on layer decide whether to call OCR at all. But a cheaper gate on its own doesn't move power one inch — the paradox of 6.9x over-firing.
#029 10 min False fires on UI elements: 44% → 22%, without adding a single outer if A gate that's supposed to be watching for subtitles fires on a video app's UI controls and a browser's URL bar. Add a filter on the outside and it goes away. It did go away, and power dropped too. But that was a contract to keep buying insurance, every single frame, against the model being weak. How I fixed the label definition and made the model side solve it — and the one condition under which outer logic is still the right answer.