Compress a model to a quarter of its size and its smarts barely drop

AI generation isn’t slow because the math is heavy. It’s slow because the memory bandwidth for reading out the model’s weights is the rate limiter.

So when you repack the weights into a quarter of the space (quantize them), latency and cost come down at the same time. Measured: generation 168→95ms, 4x the volume it can handle, quality −1pt. What decides speed isn’t the parameter count — it’s the byte count.

There’s a catch, though. Speed only shows up if you measure through the same path you deploy on, and quantization eats into what fine-tuning bought you. If you don’t hold the measurement method constant, this win turns out to be an illusion.

In Part 5 I wrote that applying “quantization” to my own GPU sped generation up from 168ms to 95ms and quadrupled the volume it could handle. A cheap GPU performing on par with an expensive data-center-class GPU. And quality barely drops.

Why does something that convenient happen? Let me break it down here.

The real bottleneck isn’t “compute,” it’s “memory”

First, how an LLM produces text. An LLM doesn’t spit out an answer all at once. It generates words (tokens) one at a time, in order. Each time, it makes a probabilistic prediction — “given the context so far, what’s the most likely next word?” — and picks one word. Repeat that a few dozen times and you have a sentence.

And what an AI model really is, is a huge mountain of numbers (we call them “weights”). Every single time it generates one word, the GPU has to read that entire mountain out of memory, once. For a 100-word sentence, that means re-reading the mountain 100 times.

Here’s the surprising part: what’s rate-limiting isn’t the speed of the multiplications (compute). It’s the speed of reading that mountain out of memory (memory bandwidth). The bigger the mountain of numbers, the longer each word’s read takes, and the slower generation gets. The GPU’s compute units are, most of the time, sitting there waiting for data to arrive.

So if you want generation to be faster — make the mountain smaller.

Quantization = repacking the numbers, coarsely

Quantization is the technique of storing each individual number in that mountain with fewer bits. Much like rounding 3.14159… to 3.14, you take numbers you were holding in 32 or 16 bits and hold them in 4 bits instead.

The mountain of numbers then shrinks to a quarter or an eighth of its size.

The effects chain together.

• Latency comes down: the mountain is smaller, so each word’s read is faster.
• Cost comes down: one GPU handles 4x the volume, so the fixed cost per request falls to a quarter.
• Accuracy barely drops: coarsening the numbers barely changes the model’s answers (−1 point on pass rate).

People tend to assume “latency and cost are usually a trade-off,” but quantization brings both down at once. Shrinking the mountain is the common cause of both the speed and the cheapness.

(To be precise: the reason generation “speed” only goes to 1.8x rather than 4x is that what scales with byte count is only the time spent reading the weights — the computation itself and the fixed overhead remain. The total volume one machine can handle — throughput — on the other hand grows by nearly 4x, in step with the mountain shrinking to 1/4. So the gains land in different places: 1.8x on speed, 4x on volume and cost.)

(An aside: this is also the flip side of the principle that “latency is decided by the byte count of the weights.” Which is why, for instance, “a larger model quantized hard” and “a smaller model quantized lightly” come out at roughly the same speed if the byte count is the same. Think in bytes, not in parameters.)

Why coarsening the numbers doesn’t cost you smarts

It’s counterintuitive, but there’s a reason the answers barely change when you coarsen the numbers. Most of the model’s decisions are made with room to spare (when it picks a word, there’s a wide gap in confidence between the candidates). Rounding the numbers a little doesn’t move that majority.

Degradation shows up only in the decisions that were close calls to begin with — it concentrates in the just-under-10% of “borderline” cases. And that just-under-10% is only the set that could be affected; the answers that actually change are a subset of that again. Which is why average quality costs you just −1 point. (There are extra tricks for holding down the breakdown in that “borderline” part — the worst case — but that’s another story.)

In any case, the “smarts” this small model shows on a specific task are something you build in through fine-tuning (SFT, CPO), which is next time. Quantization is the technique for taking those built-in smarts and making them fast and cheap while shaving away almost none of them — which is why the two work as a set.

Do it wrong and the gains vanish

That said, quantization isn’t as simple as “apply it and you win.” In practice, I hit two pitfalls.

1. Always measure through the machinery you deploy. The speed benefit of quantization only appears when you go through the dedicated fast execution path (the kernel). Measure in a stock environment and you’ll wrongly conclude “it doesn’t get any faster.”
2. Watch how fine-tuning (next time) and quantization mesh. The specialization fine-tuning I’ll describe later sits “thin and wide” across the model. Aggressive quantization can shave that off first. So fine-tuning and quantization have to be held to the same conditions. And evaluation, too, must always be done at the production quantization.

Summary

1. The rate limiter isn’t compute, it’s memory bandwidth. What sets generation latency is how fast you can read out the mountain of weights. So speed is decided not by “parameter count” but by byte count.
2. Quantization brings latency and cost down at the same time. Shrink the mountain to a quarter and accuracy barely drops. The loss is localized in the “borderline” decisions, and the backbone of the translation holds.
3. If you don’t hold the measurement method constant, this win is an illusion. Speed only materializes on the same path you deploy on, and quantization eats into what fine-tuning gained. Hold training and evaluation to the same quantization as production.

So now my own small model is fast and cheap. But that’s only half the job. Left as it is, a small model loses on quality to the models above it. The remaining half — making that model smart — is next time.

---

Appendix: raw data