LookAtMyQuant

Abstract

Every piece of a model costs something different to compress

A quantization format applies one rule to a whole model. GGUF's Q4_K_M, for example, stores most matrices at about 4.5 bits per weight and a fixed list of tensors at Q6_K. The rule is the same for a 360-million-parameter model and a 70-billion-parameter one, for a model with a large vocabulary and one with a small one, and for a phone and a server.

But the damage each piece does to the model is not uniform. In Gemma 4 E2B, one 0.4-million-weight matrix costs nearly as much at 3 bits as twenty whole layers. Half of the model's weights are a lookup table that tolerates 4 bits well. The 256 most frequent tokens make up a third of all text, while the rarest half of the vocabulary makes up a fraction of a percent. A fixed recipe cannot see any of this.

LookAtMyQuant treats quantization as compilation. A plan names one program for every stored piece of the model. The compiler finds the plan that does the least damage for a given budget on a given device. We make four contributions:

• A quantization compiler. It allocates bits across weight matrices, frequency buckets of the embedding and per-layer tables, and the key–value cache, with prices that depend on the target device.
• A cheap way to measure. One forward pass with shaped noise predicts each matrix's damage at 3 bits, and the ratios between codes are the same on every model we tested. Measuring a few layers with real codes is then enough to predict every other code.
• Compiled GGUF files for Gemma 4 E2B that beat every public quantization we measured, scored the same way, at every size where a public file exists.
• A grammar of quantization programs in which published methods are short programs, so the compiler's menu can mix code families.

A compiled Gemma 4 E2B file has 12 times lower KL than Q4_K_M at the same size

We compared the compiler's files with every widely used quantization of Gemma 4 E2B that we could load. These are Google's quantization-aware-trained q4_0 GGUF, unsloth's UD-Q4_K_XL, bartowski's Q4_K_M and LM Studio's MLX 4-bit. We also built llama.cpp's stock recipes, Q2_K to Q8_0, with an importance matrix from the same calibration text the compiler uses.

Every file is turned back into full-precision weights inside the same Hugging Face model, so the same code scores every file on the same text. The text is 64 windows of WikiText-2 and 8 held-out windows from each of 13 sources: English, ten other languages, code and chat. Each window has 512 tokens and starts with the beginning-of-sequence token. KL divergence measures how far a file's next-token predictions are from the original model's; lower is closer. Sizes count the bytes of the text model's weights, for every file.

The gap holds on every one of the 13 sources and on WikiText-2, which is not part of the calibration text. On WikiText-2 the 3.39 GB compiled file has a KL of 0.008, and Q4_K_M has 0.143.

Benchmarks agree. Multiple-choice tasks barely separate the files. Except for Q2_K and Google's QAT file, every file is within about two points of the original on MMLU, ARC-Challenge and HellaSwag. Multi-step math separates them (Figure 2). Public 4-bit files lose up to 27 points on GSM8K, and compiled files at most 6. With 250 problems, each GSM8K figure is uncertain by about ±6 points.

Most of the gain comes from the per-layer lookup table

Gemma 4 E2B feeds each of its 35 layers a 256-wide vector looked up by token. Together these vectors form a table of 262,144 rows by 8,960 columns: 2.35 billion of the model's 4.63 billion text weights. The table is never multiplied, only read one row per token. The compiler measured it as cheap to compress: per weight, it costs less than a tenth as much as the least sensitive matrices.

llama.cpp's Q4_K_M recipe stores this table at Q6_K, 6.56 bits per weight. That is 1.93 GB, 56% of the file. The compiler, choosing among the same llama.cpp types, stores it at IQ4_XS, 4.25 bits, and gives the 0.68 GB it saves to the matrices, most of which it stores at Q8_0 or BF16 (Figure 3). Figure 4 replays the compiled file one piece at a time.

With its own codes, the compiler beats every public file under 3.5 GB at only 2 GB

The compiler's native menu has trellis codes and Lloyd–Max codes, described below, which have no inference kernel yet. With them, its 2.00 GB plan has a KL of 0.074, lower than every public file under 3.5 GB, and solves 60.4% of GSM8K problems. At 3.41 GB its KL is 0.0057. These plans show what the allocation can reach once such codes run.

On Qwen3.5-4B, the embedding table is the largest lever

Qwen3.5-4B ties a 248,320-row embedding table to its output head. At 16 bits the table takes about as many bytes as all the matrices at 3 bits. At the same size as a model with 3-bit matrices and a 16-bit embedding, the compiler's plan stores the embedding at 3.1 bits on average and the matrices at 5.35. Perplexity rises 0.13% instead of 4.9%, with 12 times less KL (Table 3).

The plan gives the 256 most frequent rows 16 bits, the next 8,000 rows 8 bits, and the rarest 117,000 rows 2 bits. Rare rows are cheap. The 183,000 rows whose tokens never appear in WikiText's training split cost 0.0007 KL in total when all of them are stored at 2 bits.

Each device gets its own plan

What a code costs depends on the device. On disk, only stored bits count. On a laptop generating one sequence, the weights and the key–value cache are both read for every token, so cache bits count too. On a server running 32 sequences, one read of the weights serves all of them, so weights become cheap and the cache dominates. On a device limited by compute, decode work counts. For SmolLM2-360M the compiler makes a different plan for each (Table 4). The server plan stores matrices at 7.4 bits because 32 sequences share each read, and the compute-limited plan avoids trellis codes, which take more work to decode.

Choosing a bit width per matrix, and nothing else, lowers KL by only 3 to 5%

The large gains come from pieces that uniform formats leave at 16 bits or treat as one block: the embedding table, the per-layer table and the cache. On top of a strong base quantizer, choosing a width per matrix helps little. On SmolLM2-360M, at the same average bits as the uniform 3-bit and 4-bit programs, per-matrix widths lower KL by 3% and 5% (Table 5). Which layer a matrix is in matters more than what kind of matrix it is. Keeping the widths chosen for each kind of matrix but assigning them to random layers gives 21% more KL than the uniform program.

Quantization methods are short programs in a grammar of rewrites

LookAtMyQuant writes a model as a tensor network. Each weight matrix is a node with named indices, and nodes that share an index are joined. A quantization method is a sequence of rewrites of this network, built from five kinds of word, named after parts of speech. Figure 5 shows one layer's MLP in this form.

• Nouns are the nodes and the indices between them.
• Types are the values a node may take: an integer grid, a finite set of levels, paths through a trellis, or a randomized Hadamard transform stored as its signs. The type sets the storage price.
• Exact verbs change how the network is written but keep its output the same: a change of basis on an index, splitting out a scale, splitting one index into several.
• Lossy verbs restrict values to a cheaper type. All error comes from these.
• Control words order the steps and choose the data each one is fitted to. One example is the sweep that settles columns one block at a time and re-solves the rest, as GPTQ does.

Nine published methods are short programs in this grammar, each reproduced exactly, layer by layer, against its reference code: RTN, GPTQ, AWQ, SVD, SVD-LLM, SliceGPT, magnitude pruning, Wanda and SparseGPT. The same words also build MXFP4, NVFP4, QuIP#-style codebooks and QTIP-style trellis codes. Figure 6 steps through the program the compiler uses most, and Figure 7 shows the rewrites behind five published methods.

The compiler measures what each piece costs, then picks the combination with the least damage

Pieces

The compiler finds the pieces by running the model once. Linear layers that read the same tensor form one stage and are quantized together. Embedding tables are cut into 11 buckets of rows ranked by how often their token appears in the calibration text: 256, 256, 512 and so on, doubling up to 131,072 rows. A tied table is read two ways, as the output head at every position and as a lookup of one row, so its error is weighted by the head's inputs plus the identity. A table that is only looked up is weighted by the identity alone. Small linear layers, routers and normalization weights stay at 16 bits, and their bytes are counted.

Measure

For each piece and option, the compiler quantizes that piece alone and measures the KL divergence of the model's next-token distribution from the original's, on 16 held-out windows of 512 tokens. The windows come from the calibration split, never the test split. On SmolLM2-360M we measured all 2,312 piece-option pairs this way.

For larger models, each matrix gets one cheap probe instead. The probe adds noise shaped like the error that LDLQ leaves behind: Gaussian noise multiplied by the square root of LDLQ's error variances and by the inverse of the triangular factor of H, at a fixed level. This takes one forward pass and no code search. A few layers, which we call spot layers, are measured with every real code. On Gemma 4 E2B, the 275 matrices got a probe each, four layers and all 22 table buckets got real codes, and measuring took about one H100-hour.

Predict

An option that was not measured is predicted from the probe and the code ratios:

KL_i(c) = r_probe · KL_i(probe) · ρ(c),   where ρ(c) = median over spot matrices of KL(c) / KL(tr3)

Here r_probe is the median ratio of real 3-bit KL to probe KL on the spot layers. Both ratios turn out to be properties of the codes rather than of the model (Figures 10 and 11).

Price

A code's cost on a device comes from three numbers: the bits it stores, the cache bits it adds per token of context, and the operations needed to decode it.

cost = stored bits / batch + decode operations · c / batch + context · cache bits

Here batch is the number of sequences generated together, context is the number of tokens in the cache, and c is the price of one decode operation in bits.

On disk, only stored bits count. The laptop setting generates one sequence with a 2,048-token context. The server setting generates 32 sequences with 16,384 tokens each. The compute-limited setting charges 2 bits per decode operation.

Allocate

The compiler chooses one code o_i for every piece i to

minimize Σ_i KL_i(o_i)   subject to   Σ_i price_i(o_i) ≤ B

Dynamic programming solves this in seconds, with the budget divided into 100,000 steps and each cost rounded up to a whole step. Adding up each piece's KL assumes that damage adds up; we test this below.

Run

The model is quantized layer by layer with each piece's program, on 64 calibration windows from a separate split of the same text. Each matrix is fitted with the cross term, so it targets the original model's output given the already-quantized inputs. For llama.cpp, the same plan becomes one --tensor-type override per tensor in llama-quantize, and we check that every tensor in the output file has the planned type.

For files that run today, the menu is llama.cpp's own types

The types are Q2_K, Q3_K, IQ4_XS, Q4_K, Q5_K, Q6_K, Q8_0 and BF16. Each of Gemma 4 E2B's 278 tensors is measured alone in each type, inside real llama.cpp files, with the same KL measurement. The compiler then picks one type per tensor in the same way.

Analysis

One noisy forward pass predicts a matrix's 3-bit damage

Across 224 matrices of SmolLM2-360M and 211 of Qwen3.5-4B, the KL of the real 3-bit trellis code is 1.33 times the probe's KL, with a log-scale correlation of 0.99 on both models. Eighty percent of matrices fall between 1.22 and 1.59 times. On the 32 Gemma 4 E2B matrices measured with real codes, the median ratio is 1.35 with a wider spread (Figure 10).

Ratios between codes are the same on every model tested

Each bit removed multiplies a matrix's KL by about four, as Gaussian rate–distortion theory predicts. The ratio of each code's KL to the 3-bit trellis's KL is nearly the same on SmolLM2-360M, Qwen3.5-4B and Gemma 4 E2B (Figure 11). So the spot layers of a new model mostly confirm ratios already known. On SmolLM2, a plan chosen from three measured layers plus predictions is as good as a plan chosen from every measurement: 0.01333 summed KL against 0.0133.

Damage adds up, except where it is smallest

The compiler assumes that the KL of a plan is the sum of its pieces' KL. Measured KL is within 18% of the sum for every compiled plan on SmolLM2 and Qwen3.5 and for every Gemma GGUF plan (Figure 12). Uniform plans measure 10–14% below the sum, because the cross term repairs them better than predicted. Gemma plans with the compiler's own codes measure 36% to 93% above the sum from 2.5 GB up, where each piece's KL is smallest. Searching around the compiler's plan with real evaluations, as EvoPress does, would target this gap.

In Gemma 4 E2B, a few pieces carry most of the sensitivity

Layers 15 to 34 reuse the keys and values that layers 13 and 14 compute, and have no key or value projections of their own. They cost less than half as much per layer (Figure 13). One matrix, layer 13's per-layer projection, costs 0.106 KL at 3 bits, nearly as much as all twenty later layers together. Every plan keeps it at 16 bits.

Per weight, key and value projections cost 15 to 19 times as much as query projections. In both tables, KL per row falls 2,000 to 5,000 times from the most frequent bucket to the rarest, so the compiler gives frequent rows 8 or 16 bits, and the rarest half of the per-layer table gets 2 bits in every plan.

Two exact rewrites did not help

Exact rewrites are free moves for a search: they change how the network is written without changing its output, so they can make it easier to quantize. We tried two on SmolLM2-360M.

• Diagonal rescaling across a matrix pair (value and output projections, up and down projections). We searched the rescaling strength per block. 63 of 64 blocks chose no rescaling, and the result was slightly worse: 16.735 perplexity against 16.721. The randomized Hadamard transform spreads quantization error evenly across channels, and rescaling makes some channels small, so the compensating factor amplifies their error.
• Choosing Hadamard signs. Keeping the best of four random sign seeds per matrix changed nothing: 16.76 against 16.72 at 3 bits. The per-matrix error picks among seeds that differ only by noise.

On top of incoherence processing, these rewrites have nothing left to fix. The exact rewrites worth searching are those the code does not already perform.

Limitations

• Decoding speed was not measured. The compiled GGUF stores most matrices at Q8_0 or BF16, which decode with different kernels than Q4_K. File size, and so memory traffic per token, matches the public files, but tokens per second may differ.
• The compiler's own codes don't run in any inference engine yet. Their quality numbers still hold. Every file in this paper, public or ours, was scored the same way, by turning its stored values back into full-precision weights. Their speed is unknown until an engine runs them.
• One model so far against public files. The public comparison covers Gemma 4 E2B only.
• Calibration text. The compiler and our llama.cpp builds calibrate on a split of the same 13-source mix used for testing; the unsloth and bartowski files calibrate on their own text. WikiText-2 is in neither, and the gap there is as large.
• Small evaluation sets. KL uses 104 held-out windows and 64 WikiText-2 windows of 512 tokens; GSM8K uses 250 problems.
• No fine-tuning. Every result here is post-training quantization.

Further work

Every plan here is post-training quantization, and the Gemma plans are priced by file size. Each step below either improves quality or prices a resource that file size misses.

• Fine-tuning after allocation. Fine-tuning helps most where bits are fewest. On Llama-2-7B, fine-tuning each layer after quantizing lowered the perplexity of the 2-bit trellis from 6.76 to 6.26, but moved the 4-bit trellis only from 5.177 to 5.167. QTIP adds end-to-end fine-tuning and reaches 5.86 at 2 bits. So we expect the largest gains for the smallest plans, such as the 1.5 GB and 2.0 GB files. The allocation can use this too. A low-bit code that fine-tuning repairs well does less damage than its measurement says. End-to-end fine-tuning against the original model also works on the whole plan at once, which targets the gap between predicted and measured KL.
• GPU memory as the budget. The plans here count every byte the same, which matches file size and the unified memory of Macs and phones. On a GPU, lookup tables and offloaded mixture-of-experts layers can stay in CPU memory, so only the pieces that live on the GPU use its memory. A GPU-memory target prices the other pieces at a cheaper host-memory rate. For Gemma 4 E2B, whose per-layer table is half the model, that plan would give the table more bits and the matrices fewer than the plans here.
• Inference speed as the budget. The compiler's decode costs are assumed for now: one operation per weight for 16-bit values, two for a scalar code and three for a trellis. Measuring tokens per second for each type on each device lets a plan target speed directly, for example the most accurate GGUF that decodes at least as fast as Q4_K_M on a given GPU.
• Long contexts. The cache is priced at the context length a plan targets, but plans have been evaluated at 512 and 2,048 tokens. The Qwen3.5-4B laptop plan, priced at 32,768 tokens, should be evaluated at that length, and the recurrent state of hybrid models like Qwen3.5 should be priced too.
• Kernels for the compiler's own codes. With trellis and Lloyd–Max codes, the 2.00 GB plan beats every public file under 3.5 GB, but no inference engine runs these codes yet. The plans ask for three things: rows of different widths in one table, trellis decoding inside llama.cpp, and keys and values cached at different widths.
• More models. The rest of Gemma 4 is next: E4B, 12B, 31B and the 26B-A4B mixture of experts, each compared with its public files on the same protocol. The compiler already treats experts as pieces, grouped by how often the router picks them.
• Search around the plan. Where the sum of per-piece KL misses the measured KL, trying small changes to the plan on the real model, as EvoPress does, would find plans the sum cannot.

Related work

Bits per layer. HIGGS [1] proves that perplexity increase is linear in each layer's error and allocates widths by dynamic programming; our measurements agree, matrix by matrix. ExLlamaV2's EXL2 format [2] measures each module at about a dozen settings and optimizes to a target bits per weight. NVIDIA ModelOpt's auto_quantize [3] searches a format per layer under an effective-bits budget. EvoPress [4] searches with real model evaluations because summed per-layer errors can rank configurations wrongly, and AMQ [5] uses a learned quality predictor. HAWQ-V3 [6] solves an integer program under memory, latency and bit-operation budgets for convolutional networks, the closest prior version of per-device pricing.

Cache and embeddings. KVTuner [7] searches per-layer key and value widths and also finds keys more sensitive than values; PM-KVQ [8] allocates cache precision under a memory budget. Recommendation models quantize embedding rows by feature frequency [9]. We found no peer-reviewed work that does this for language-model vocabularies.

Codes and rewrites. GPTQ [10], QuIP# [11] and QTIP [12] supply the base quantizers: error-compensating sweeps, incoherence processing and trellis codes. Learned rotations such as SpinQuant and FlatQuant, and graph-rewrite systems such as TASO and Tensat, search exact rewrites, the latter for speed rather than quantization.

What is new here. One allocation across matrices, frequency buckets of the embedding and per-layer tables, and the cache, with prices that depend on the device; a menu that mixes code families; plans run with cross-term targets so later matrices absorb earlier error; and compiled files in a standard format, compared with public files on one protocol.

References

1. HIGGS. Malinovskii et al., Pushing the Limits of Large Language Model Quantization via the Linearity Theorem. NAACL 2025. arXiv:2411.17525
2. EXL2. turboderp, ExLlamaV2. github.com/turboderp-org/exllamav2
3. ModelOpt. NVIDIA Model Optimizer, auto_quantize. Documentation
4. EvoPress. Sieberling et al., evolutionary search for dynamic model compression. ICML 2025. arXiv:2410.14649
5. AMQ. AutoML search over per-layer bit widths with a quality predictor. EMNLP 2025. ACL Anthology
6. HAWQ-V3. Yao et al., HAWQ-V3: Dyadic Neural Network Quantization. ICML 2021. PMLR
7. KVTuner. Li et al., layer-wise mixed-precision key–value cache quantization. ICML 2025. PMLR
8. PM-KVQ. Key–value cache precision allocated under a memory budget. arXiv:2505.18610
9. Mixed-precision embeddings. Embedding precision chosen per feature-frequency group in recommendation models. arXiv:2409.20305
10. GPTQ. Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR 2023. arXiv:2210.17323
11. QuIP#. Tseng et al., QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks. ICML 2024. arXiv:2402.04396
12. QTIP. Tseng et al., QTIP: Quantization with Trellises and Incoherence Processing. NeurIPS 2024. arXiv:2406.11235

Citation

@misc{boson2026lookatmyquant,
  title  = {LookAtMyQuant: A Quantization Compiler},
  author = {{Shin Megami Boson}},
  year   = {2026},
  month  = oct
}