Your Quantized LLM Is Not Slow Because of the Quantization

The Symptom

I spent months building a 2-bit quantization scheme for Qwen3. The model went from 8 GB to 2.6 GB, a 4.5x reduction. Then I measured throughput.

It was barely faster than the FP16 baseline.

That did not add up. Token decoding in an LLM is memory bound, not compute bound. You read every weight once per token, and the arithmetic per byte read is tiny. This is well established: Pope et al. identify the memory traffic needed to load parameters and KV cache from high-bandwidth memory as a primary constraint on generative inference [1], and the AWQ authors frame model size as raising both a memory-size barrier for serving and a memory-bandwidth barrier for token generation [2].

If the weights are four times smaller, you read four times less, and you should go meaningfully faster. Something was eating the gain.

The Obvious Suspect

My first assumption was that my own quantization was at fault: dequantization cost, no fused kernel, irregular memory access.

That is a fair suspicion. Extreme quantization schemes pay a decode cost on every weight, and if you have not written a fused kernel, you are materializing full-precision weights back into memory before every matmul, which defeats the entire point of compressing them.

The suspicion was partly correct. It was also wrong in an expensive way, because it kept me staring at my own code for weeks.

Profile Before You Optimize

The procedure that eventually found the problem:

1. Time a full forward pass at the top level. That is your baseline.
2. Time each component separately: attention, MLP, embeddings, and the output projection.
3. Build a memory-traffic model. Count the bytes that must move per token, then divide by your device's achievable bandwidth. That is the floor you are trying to reach.
4. If you are far from the floor, the gap is not where you think it is.

Step 2 is the one people skip, and the reason is structural. You optimized the quantized layers, so you profile the quantized layers. The parts you did not touch are, by construction, the parts you are not looking at.

When I finally instrumented everything, the quantized layers were fine. The time was going somewhere else.

The Culprit

Qwen3-4B has a hidden size of 2560 and a vocabulary of 151936 tokens. The output projection, lm_head, is a 151936 x 2560 matrix. In bf16:

151936 x 2560 x 2 bytes = 778 MB

This is the largest single tensor in the model. It is also, in most setups, left in full precision, because quantizing the output projection tends to cost more in quality than it saves in memory.

In candle 0.9.2, the logits were computed as a matmul against the transposed weight. Transposing produces a non-contiguous tensor. The matmul path did not accept that layout, so it materialized a contiguous copy first.

778 MB copied. Per token.

Not read, copied: 778 MB read plus 778 MB written, roughly 1.5 GB of memory traffic, purely to reorder bytes before any arithmetic happened. On every token generated.

The large-vocabulary head, being the one that quietly dominates, is not a new phenomenon. Wijmans et al. show that on the training side, growing vocabularies shifted the memory footprint disproportionately onto the cross-entropy layer, to the point where it can account for the large majority of training memory, and they solve it with a fused kernel that never materializes the full logit matrix [3]. Different mechanism, same structural cause: the vocabulary dimension is large, and anything that touches it in full is expensive.

Quantifying It

You do not need a benchmark to see the shape of the problem. The architecture is public, so the traffic is arithmetic.

Qwen3-4B has 3.63B parameters in its 36 transformer blocks and 389M in the tied embedding and output projection. Per decoded token, weight traffic is the block weights at whatever precision you quantized them to, plus the lm_head at bf16, plus the copy.

Two things worth noting. First, the FP16 column comes out at 8.045 GB, which matches the 8.04 GB I actually measured serving the model. The model is sound. Second, look at the red band. The copy is a fixed 1.5 GB regardless of how hard you compress, so its share climbs from 16% at FP16 to 48% at 2 bits.

That has a direct consequence for the speedup you can achieve. If decode is bandwidth-bound, throughput is inversely proportional to traffic, which gives a ceiling:

At 2 bits, the copy caps you at 2.96x when the traffic model allows 4.77x. You lose 38% of the available gain, and you lose it to a memory copy that does no arithmetic at all.

The Fix

The GEMM does not need the copy. cuBLAS, and every serious BLAS, takes transpose flags for its operands. A transposed matrix is not a different matrix; it is the same bytes with different strides. The transpose should be free.

The fix routes the operation, so the transposed weight reaches the GEMM directly instead of being materialized into a fresh contiguous buffer.

That work is upstream:

• Issue: huggingface/candle#3871
• PR: huggingface/candle#3872

If you are running candle, you already have it.

What It Was Worth

About half of the end-to-end throughput improvement I had been attributing to my quantization work came from this fix.

Half. Months of work on lattice quantization, and a comparable share of the measured speedup came from deleting a memory copy in code I did not write and had not thought to look at.

Note that the analytical model above predicted 48% at 2 bits, before I had measured anything. That convergence is the useful part. This is not an anecdote about a single bug in a single framework; it is a predictable consequence of compressing part of a model while leaving the rest alone.

What Generalizes

Fixed costs grow in relative terms as you compress. Quantization shrinks what you quantized. Everything else stays the same size and becomes a larger share of your runtime. If you are working on extreme quantization, this is not a footnote; it is the next problem. Past a certain compression ratio, the uncompressed components are your bottleneck by definition, and no further work on the quantization scheme will move your throughput.

Layout bugs hide well. A transposed copy is functionally correct. Tests pass. Output is bit-identical. Nothing is broken; it is just slow, and slowness does not throw. These survive a long time in mature codebases because everyone assumes the framework handles it.

Count bytes moved, not FLOPs. This is the central lesson of FlashAttention, which got its speedup not by reducing arithmetic but by avoiding materializing a large intermediate in HBM, and which actually performs more FLOPs than the standard implementation while running several times faster [4]. The transposed copy is the same failure mode in miniature: zero arithmetic, all traffic.

Do not trust your priors about where time goes, least of all in your own project. I had every reason to believe the bottleneck was in my quantization code. I had just written it; it was the novel part, and it was where all my attention was. That is precisely the bias profiling exists to correct.

Checklist

Before optimizing an inference pipeline:

• Build the traffic model first. Bytes moved per token divided by achievable bandwidth. Know your floor before you start.
• List every tensor above 100 MB. For each, ask what precision it is in and whether it is copied on the hot path.
• Profile the components you did not modify.
• Look for .t(), .transpose() or .permute() immediately before a matmul. Any of them can trigger a materialized copy.
• Recompute your ceiling after every compression step. The bottleneck moves as you compress.

The bottleneck is rarely where the interesting work is. That is what makes it a bottleneck.

Notes on the Figures

Both figures are analytical, derived from the published Qwen3-4B configuration, not measured. They count weight traffic only and exclude the KV cache, activations, norms, and biases, so they are a lower bound on real traffic. They assume decode is fully bandwidth-bound, which is the standard regime at batch size 1. The script that produces them is a few dozen lines and reproduces from the config file alone.

References

[1] R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, A. Levskaya, J. Heek, K. Xiao, S. Agrawal, J. Dean. Efficiently Scaling Transformer Inference. MLSys 2023 (Outstanding Paper Award). arXiv:2211.05102

[2] J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, S. Han. AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration. MLSys 2024 (Best Paper Award), Proceedings of MLSys 6:87-100. arXiv:2306.00978

[3] E. Wijmans, B. Huval, A. Hertzberg, V. Koltun, P. Krähenbühl. Cut Your Losses in Large-Vocabulary Language Models. ICLR 2025. arXiv:2411.09009

[4] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, C. Ré. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS 2022. arXiv:2205.14135