Generation speed is decided by the byte count of the weights, not the parameter count

Measured on the same GPU, 4B-INT8 (weights about 4.0GB) and 8B-INT4 (about 4.3GB) came out at nearly the same speed. 5.98ms against 6.50ms per token. Even though the parameter counts differ by 2x.

The reason is simple: every time it generates one token, it reads out the model’s weights in full. What decides it isn’t the amount of compute and isn’t the parameter count — it’s the number of bytes it has to read. Which is also why, when you pick a GPU, you look at memory bandwidth and not compute performance.

This law comes with premises, though. In fact, my first measurement got the method wrong and I misread a “should be 2x faster” as 1.3x — and under a different measurement method, a 2.35x gap disappeared, and then inverted.

”A 4B should be 2x faster” only came out at 1.3x

The first measurement went like this. Swap the 8B model for a 4B and you halve the weights you read, so it should be close to 2x faster. But what I measured stopped at 1.24–1.36x.

“Going to 4B doesn’t buy you as much as you’d think” — if I had drawn my conclusion right there, I would have gotten the whole model-size decision wrong.

The cause wasn’t the model. It was that the setups I was measuring weren’t held constant. The 4B side alone was carrying a double handicap.

• A different quantization kernel (one of them was a scheme that unpacks less efficiently on that GPU generation)
• The adapter was still running bolted on the outside (every generation step pays an extra small matrix multiply)

Re-measure with the same quantization scheme as the 8B side, and with the adapter merged into the model itself — 2.3x. It had followed the law all along.

Hold the serving setup constant before you compare. A comparison that isn’t held constant makes 2x look like 1.3x.

Every token reads all of the weights

Why does the byte count decide it? The mechanism is plain.

When a language model generates one token, it touches every weight in the model exactly once. So at batch=1 (generating for a single user only), the read out of memory dominates rather than the compute. The amount read is the byte count of the weights, so generation speed is inversely proportional to that byte count.

The same law carries straight over to quantization bit width. Bit width is byte count.

The size ratio is 2.92x but the measured value is 2.35x. The gap is what’s left over — the parts that aren’t quantized (reads and writes of the cache and the activations). Byte count decides everything, but there are reads and writes beyond the model’s weights.

And as long as memory bandwidth is the rate limiter, the number to look at when choosing a GPU is bandwidth too. Not compute performance, and not VRAM capacity.

Get one thing wrong in how you measure and the 2.35x disappears

Now for the real point. Same model, same GPU, measured three times changing nothing but the measurement method. The results split three ways.

In the first, the fast quantization kernel never got used and compute was the rate limiter. In the second, the fixed per-step overhead dominated, and on top of that only the unpacking cost of quantization got added. Only in the third did the fixed cost come out and the bandwidth-bound shape appear.

The speed of quantization only shows up when “fast kernel × graph execution × batch=1 bandwidth-bound” all line up. Drop any one of the three and it doesn’t just look like it “doesn’t work” — it looks actively harmful. When you’re reading benchmark numbers, check first that all three line up.

Open questions: a law has a domain where it holds

This law isn’t unconditional. I’ve confirmed two conditions, by measurement, where it breaks.

The first is when you’re handling several users at once. Process the generation in a batch and several requests can share a single read of the weights. Throughput scales with bandwidth, but the wait for any one user stays proportional to the byte count. Everything in this article is about the speed of generating for a single user.

The second is when the weights get too small. Quantize a small model hard and put it on a fast GPU, and the weight read finishes so quickly that fixed costs like kernel launch become the dominant term. In practice, 4B-INT4 came out at 168 against 177 tok/s across the top two generations of data-center GPU — barely any difference at all, even with 3.3x the bandwidth. Measure the same 8B-INT4 on a previous-generation consumer GPU and it ran at exactly half the speed of the 4B (= plainly bandwidth-bound).

So don’t try to confirm this law with a comparison that spans GPUs. If one side is bandwidth-bound and the other is fixed-cost-bound, the same law will hold in one place and fail in the other.

Lessons

1. Generation speed is decided by the byte count of the weights, not the parameter count. 4B-INT8 and 8B-INT4 came out at nearly the same speed. So design around “light in bytes” rather than “a small model,” and choose GPUs on memory bandwidth rather than compute performance.
2. Hold the serving setup constant before you compare. A difference in nothing but the quantization kernel and how the adapter was handled turned 2x into 1.3x. Before you attribute a gap to model size, hold the way it’s run constant.
3. A law has a domain where it holds. Step outside the batch=1, bandwidth-bound premise and neither size nor bandwidth buys you anything. In comparisons that span GPUs, suspect that the premise itself has changed.

This is the manifesto’s first principle — Measure First — in its inference-performance form. Same model, same GPU, and still, one choice in how you measure makes 2.35x look like “no difference,” or like an inversion.

Appendix: raw data

All measured at batch=1 (generating for a single user). TPOT = generation time per token.