LookAtMyQuant 2

Qwen3.5-4B in 1.2 GB: 330 of 540 math problems to Bonsai 2 Ternary's 328, and closer to the original on text

Shin Megami Boson

November 2026

Ours Bonsai 2 Ternary 4B Smaller Qwen3.5 models at more bits per weight, and the original

Math problems solved, of 540

KL from the original on text (lower is better)

Figure 1. Left: exact-match math problems solved, of 540. Right: KL(original ‖ model) between next-token distributions, averaged over held-out text from 13 sources. Sizes are language-model weights.

At 1.2 GB our Qwen3.5-4B solves 330 of 540 math problems and Bonsai 2 Ternary solves 328

Bonsai 2 Ternary 4B [1] is Prism ML's ternary-weight version of Qwen3.5-4B [2]. It stores 2-bit codes with a scale and a bias per 128 weights, and its language-model weights take 1.20 GB. The original's take 8.4 GB in bf16.

Our file is the same model at 2.3 bits per weight, 1.23 GB. It is post-training quantization with trellis codes from the LookAtMyQuant compiler, followed by 4,000 steps of distillation from the original. The packed file is a release of the repository.

The math test is 540 generated problems with exact integer or fraction answers, in twelve families at three difficulties, from long addition to determinants and modular exponentiation. Scoring is exact match on the final boxed answer. Decoding is greedy with thinking disabled and at most 2,048 new tokens.

Ours solves 330, Bonsai 328 and the original 359. 62 problems are solved only by ours and 60 only by Bonsai (McNemar p = 0.93), so the test does not separate the two files.

On text, ours is closer to the original. KL(original ‖ model) on held-out text from 13 sources is 0.45 nats for ours and 0.59 for Bonsai, and ours is lower on 11 of the 13 (Figure 2). Bonsai is lower on chat and Hindi.

All sizes are the language model alone. Qwen3.5-4B also has a 0.67 GB vision tower, which Bonsai's 1.87 GB download keeps in bf16. Our file omits it, and we did not evaluate image inputs.

Table 1. Every model on the same math test and the same text. KL is KL(original ‖ model) in nats per token, on 52 windows of 512 tokens from 13 sources and on 64 windows of the WikiText-2 test split.

ModelWeights (GB)Math, of 540KL, 13 sourcesKL, WikiText-2
Qwen3.5-4B, ours
2.3 bits per weight, trellis-coded
1.233300.450.42
Bonsai 2 Ternary 4B
Prism ML, ternary weights
1.203280.590.50
Qwen3.5-2B at 5 bits
1.9B parameters, MLX-style affine quantization
1.292840.510.45
Qwen3.5-2B at 4 bits
1.9B parameters, MLX-style affine quantization
1.062420.600.51
Qwen3.5-0.8B at 8 bits
0.8B parameters, MLX-style affine quantization
0.802050.750.70
Qwen3.5-4B, original
bf16
8.4135900
Ours Bonsai 2 Ternary 4B
Figure 2. KL(original ‖ model) on each of the 13 sources: English, ten other languages, code and chat, four windows of 512 tokens each. Lower is better.

A 4B model at 2.3 bits beats a 2B at 5 bits and a 0.8B at 8 bits

The same space can hold a smaller model at a higher bit rate. We quantized the two smaller Qwen3.5 models the way MLX's standard files do [8]: affine quantization in groups of 64 weights, each with a 16-bit scale and bias, which costs N + 0.5 bits per weight at N bits.

Qwen3.5-2B at 5 bits is 1.29 GB and solves 284. At 4 bits it is 1.06 GB and solves 242. Qwen3.5-0.8B at 8 bits is 0.80 GB and solves 205; more bits would not help it, since its relative weight error at 8 bits is already 0.5%. Their KL from the original 4B on text is 0.51, 0.60 and 0.75, against 0.45 for ours.

At 2.3 bits, two files with nearly equal KL solve 1 and 328 problems

A scalar Lloyd–Max build at this size, calibrated on generic text, has WikiText-2 KL 0.60. Bonsai's is 0.50. The build solves 1 problem of 540 and Bonsai solves 328. That likelihood metrics miss reasoning at low bit rates is known. ThinkBit [16] and LAQuant [17] report it for perplexity at 2 and 3 bits. This pair shows the same for KL from the original.

The two metrics also separate in the other direction. Our post-training build already solves 317 at a KL of 1.41 on the 13 sources. Fine-tuning then cuts that KL to 0.45 and adds 13 problems. So we selected every component below on the benchmark and report KL beside it. Figure 3 adds the components one at a time.

Figure 3. Cumulative ablation on the 540. Each bar adds one component to the build above it, at the same file size. Each blue bar is a single build, and builds of one recipe have differed by 31 problems.

Calibrating on the model's own long worked solutions, with digit tokens weighted 10×, nearly doubled the score at fixed codes

Layer-wise quantization minimizes error under the input second moment H = E[w·xxᵀ], so the calibration set and the token weights w decide which errors count. Weighting tokens in H is RSQ's method [12], which weights by attention score. Calibrating on the model's own reasoning traces is AYOT's [13]. Here the weight follows the kind of token a position predicts, chosen for the capability to keep.

With trellis codes fixed and half of the set the original's GSM8K solutions [10], the build solves 117.

An all-math set with w = 1 at positions that predict a digit and w = 0.1 elsewhere gives 147. Replacing 256 sequences of 512 tokens with 64 of 2,048, the original's greedy solutions to MATH training problems [9], gives 224. These have long chains of intermediate results, like the benchmark's.

Bits belong to rows: 10% of a matrix's rows can carry 79% of its Fisher information

For each output row i we measure si = E[gi²]·E[yi²] on the original model. Here y is the row's output and g is the gradient of the log-likelihood of tokens sampled from the model's own predictions, taken at the positions the calibration weights most, those that predict a digit. In the matrices that read a linear-attention layer's input, 10% of the rows carry 79% of the total.

Mixed precision between output rows, ranked across the whole model, is MixLLM's [14]. RoleQ [15] gives a second ternary plane to units chosen by Fisher saliency. We apply the idea to trellis codes, at four rates inside a matrix.

A global allocation of 1 to 4 bits per row group, across all matrices and under the same byte budget, took the score from 265 to 300. It puts 13% of the matrix weights at 1 bit, 47% at 2, 36% at 3 and 4% at 4. Rotating the output side as well would average this structure away, so the incoherence transform is on the input side only.

e-means codebooks cut trellis MSE by 4% to 9% at the same bytes and added 17 problems

A trellis code decodes each weight from a 4,096-entry codebook. We had fit it with six full-batch Lloyd iterations on a fixed set of Gaussian sequences. e-means [7] replaces full passes with mini-batch updates through exponential moving averages. Fit that way, with fresh Gaussian samples in every batch, an EMA of the sum and count for each codeword, and a batch weight on a cosine schedule from 0.5 to 0.01, the codebook's MSE under the deployed encoder falls 4.0%, 4.5%, 6.2% and 8.8% at 1 to 4 bits.

The file size and the decoder do not change. The score went from 300 to 317, and KL on the 13 sources fell 4%. With e-means codebooks at 2 and 3 bits only, the score was 311.

Error propagation at half strength beat full strength by 41 problems

QEP [6] adds a cross term to the layer objective, so that the quantized layer on quantized inputs matches the original layer on clean inputs, with a strength α. At α = 1 the build solves 224. At α = 0.5 it solves 265, and WikiText-2 KL falls from 0.85 to 0.66.

Re-quantizing every layer from the original weights mid-training cut KL on text by another 43%

Our fine-tuning is distillation on latent full-precision weights W that are pulled toward their quantized values P and re-quantized every 1,000 steps (Figure 5). Its parts are published. TVQ [18] trains trellis codes through latent weights and recomputes their indices every 16 steps, with a straight-through estimator. The pull is ProxQuant's and BinaryRelax's [20] [21]. It took the score from 317 to 321 and KL on the 13 sources from 1.41 to 0.79. The codes hardly moved: 0.1% of a layer's codes change at a re-quantization, because a weight held near its quantized value quantizes back to the same code. BCJR-QAT [19] describes the same effect: trained latent weights stay in the basin of the post-training solution.

So between steps 1,000 and 1,750, each layer is re-quantized once from the original weights W0 in place of W. H comes from an exponential moving average over the training batches' own activations, about 2,000 sequences of the partly fine-tuned model, eight times the build's calibration tokens. 97% of the codes change, W restarts at the new P, and training continues. We know of no earlier method that returns to the original weights during training.

On two builds this gave 330 and 324 problems at KL 0.45 and 0.45, against 321 and 314 at 0.79 and 0.80 without it (Table 2). Part of the gain on text comes from the data mix: the running H is 30% non-math text, and the build's calibration set is all math.

Fine-tuned Fine-tuned, with mid-training re-quantization (our file) Steps 1,000 to 1,750: re-quantization from the original weights

KL from the original on text (lower is better)

Figure 4. KL from the original on the training run's held-out text, for the quantized model P at steps 0, 1,000, 2,000, 3,000 and 4,000. Both runs start from the same build.

Table 2. Two builds, each with post-training quantization only, fine-tuned, and fine-tuned with mid-training re-quantization.

TrainingMath, of 540KL, 13 sourcesKL, WikiText-2
Build A, e-means codebooks at every bit rate
Post-training quantization only3171.411.07
Fine-tuned3210.790.64
Fine-tuned, with mid-training re-quantization (our file)3300.450.42
Build B, e-means codebooks at 2 and 3 bits only
Post-training quantization only3111.431.05
Fine-tuned3140.800.64
Fine-tuned, with mid-training re-quantization3240.450.44

The recipe is post-training quantization against a digit-weighted objective, then distillation that ends exactly on the codes

Table 3. The final recipe.

ComponentSetting
Post-training quantization
CodesTrellis-coded quantization as in QTIP [3]: a bitshift trellis with L = 12 and V = 1, at 1 to 4 bits per weight, decoded from a 4,096-entry codebook per bit rate. Codebooks are fit with e-means.
BasisA randomized Hadamard transform on each matrix's input side [4], shared by the matrices that read the same activations. One scale per row.
RoundingViterbi along each row, 16 columns at a time with 8 of look-ahead, with LDLQ error feedback [5] under H = E[w·xxᵀ].
Calibration64 sequences of 2,048 tokens: the original's greedy solutions to MATH training problems. w = 1 at digit-predicting positions and 0.1 elsewhere.
Error propagationQEP [6] at α = 0.5.
Bit allocationPer-matrix rates from the compiler, then a global allocation over 1,520 row groups in 190 matrices by each row's Fisher information. The 200 matrices average 2.32 bits per weight.
EmbeddingThe tied embedding, 636M weights, in frequency buckets at 1 to 8 bits, 1.88 on average.
Fine-tuning, 4,000 steps
StudentLatent full-precision weights W, initialized at the dequantized build. P is W's current quantized value.
LossKL(original ‖ student) over the original's top 64 tokens plus one bucket for the tail, at 128 positions of each 512-token sequence, with digit-predicting positions weighted 10×. Batches of 8, half math and half general text, science, code, instructions and tool calls. The math is the original's own solutions to GSM8K and MATH training problems and, for about an eighth of it, to problems from the benchmark's generator at another seed, with wrong calculations corrected and only right answers kept. Every test question is excluded by hash.
OptimizerAdam, learning rate 10⁻⁵, with ε set per tensor to its initial gradient RMS. Norm weights and the embedding's scales and codebooks train as well.
PullAfter each step, W ← W + β(P − W) with β = 10⁻³. Over the last 600 steps β rises to 0.03 and the learning rate falls to 0. Then W ← P.
Re-quantizationEvery 250 steps, P ← Q(W) for a quarter of the layers with the build's own quantizer, and row scales and codebooks are refit. Between steps 1,000 and 1,750, each layer's P ← Q(W0) once, under a running H.

The latent weights are there because a trellis-coded weight has no small step: one path bit changes the decoded values of the weights after it. Quantizing one layer takes 20 to 40 seconds, so a straight-through estimator that re-quantizes at every step is out of reach. Our projection is the build's full quantizer, on a quarter of the layers every 250 steps, and the forward pass uses the latent weights under the pull. PV-Tuning [22] is another route past straight-through estimation. The pull has to be strong. A run with β = 3·10⁻⁴, and re-quantization twice as often, left the weights nearly twice as far from their codes and lost the gain on math.

InitW ← PThe latent weights start at the dequantized build.
Each stepDistillOne Adam step on the token-weighted KL to the original's top-64 next-token probabilities.
Each stepW ← W + β(P − W)A pull toward the quantized values, with β = 10⁻³.
Every 250 stepsP ← Q(W), a quarter of the layersWith the build's own quantizer. Once mid-training, from W0.
Last 600 stepsAnnealβ rises to 0.03 and the learning rate falls to 0. Then W ← P.
Figure 5. The fine-tuning loop. The result is exactly representable in the format: re-quantizing four of its matrices changed no code.

The compiler now allocates bits to rows against a digit-weighted objective, and fine-tunes after allocation

LookAtMyQuant 1 measured the KL that each stored piece of a model costs when it alone is compressed, and chose one code per piece under a byte budget. It stopped at post-training quantization and listed fine-tuning after allocation as further work. Table 4 lists what this recipe changes. The code for all of it is in the public repository, with a job for each step.

Table 4. The compiler in LookAtMyQuant 1 and in this recipe.

StepLookAtMyQuant 1This recipe
PiecesWhole matrices, embedding buckets by token frequency, key–value caches.Also row groups inside each matrix.
SensitivityKL on held-out text with one piece compressed.Also Fisher information per output row.
CalibrationA text mix with every token weighted equally.A set chosen for the target, with token weights in H.
Codesllama.cpp's types, or Lloyd–Max and trellis codes with Lloyd codebooks.Trellis codes at 1 to 4 bits per row group, with e-means codebooks.
After allocationNothing.Distillation that ends on the codes, with one re-quantization from the original weights.
AcceptanceKL from the original.An exact-answer benchmark as well as KL.

Calibration and training cost about $34 of GPU time in all

Table 5 prices each step of making the file once, at Modal's on-demand rates [11] for the GPU, CPU and memory the step used. Everything before fine-tuning costs $13 and fine-tuning costs $21, 5.4 GPU-hours in all. Calibration uses 131k tokens and fine-tuning sees 16M.

Table 5. The time and cost of each step.

StepGPUHoursCost
Per-matrix sensitivity measurement for the bit allocationH1001.6$7.74
Calibration set (the original generates 384 long solutions)H1000.3$1.24
Row Fisher information and four codebook fitsL40S0.2$0.50
Quantization (LDLQ and Viterbi, 32 layers)H1000.6$3.32
Fine-tuning, 4,000 stepsB2002.7$20.98
Total5.4$33.78

Limitations

References

  1. Bonsai 2 Ternary 4B. Prism ML. huggingface.co/prism-ml/Ternary-Bonsai-2-4B-mlx-2bit
  2. Qwen3.5-4B. Qwen team. huggingface.co/Qwen/Qwen3.5-4B
  3. QTIP. Tseng, Sun, Hou and De Sa, QTIP: Quantization with Trellises and Incoherence Processing. NeurIPS 2024. arXiv:2406.11235
  4. QuIP#. Tseng et al., QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks. ICML 2024. arXiv:2402.04396
  5. GPTQ. Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR 2023. arXiv:2210.17323
  6. QEP. Arai and Ichikawa, Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization. NeurIPS 2025. arXiv:2504.09629
  7. e-means. Bessa, Mazaré, Szilvasy, Touvron, Picard, Douze and Jégou, e-means: outperforming k-means (in one-tenth the epochs). 2026. arXiv:2611.09501, e-means.github.io
  8. MLX. Apple machine learning research. github.com/ml-explore/mlx
  9. MATH. Hendrycks et al., Measuring Mathematical Problem Solving With the MATH Dataset. NeurIPS 2021. arXiv:2103.03874
  10. GSM8K. Cobbe et al., Training Verifiers to Solve Math Word Problems. 2021. arXiv:2110.14168
  11. Modal. Pricing, read on 10 November 2026. modal.com/pricing
  12. RSQ. Sung, Yadav, Li, Yoon and Bansal, RSQ: Learning from Important Tokens Leads to Better Quantized LLMs. 2025. arXiv:2503.01820
  13. AYOT. Wang, Li, Kang, Fan and Yao, Attend to Your Own Thoughts: Breaking the Barrier for Post-Training Quantization of Reasoning LLMs through the Lens of 1.58-Bit Quantization. 2026. arXiv:2608.01078
  14. MixLLM. Zheng, Song and Liu, MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Design. MLSys 2026. arXiv:2412.14590
  15. RoleQ. Knittel, Helbling and Pfister, RoleQ: Role-Aligned Scales for Converting Language Models to Ternary. 2026. arXiv:2610.21808
  16. ThinkBit. Sukthanker and Cameron, ThinkBit: Search Codewords, Recalibrate Scales, Select by Reasoning. 2026. arXiv:2610.34886
  17. LAQuant. Choi, Song and Yoo, LAQuant: A Simple Overhead-free Large Reasoning Model Quantization by Layer-wise Lookahead Loss. 2026. arXiv:2605.08755
  18. TVQ. Prutianova, Bogdanova, Shutov, Poimanov, Gorkov and Zharikov, Trainable Vector Quantization for Large Language Models. ECML PKDD 2026. arXiv:2610.41822
  19. BCJR-QAT. Iyengar, BCJR-QAT: A Differentiable Relaxation of Trellis-Coded Weight Quantization. 2026. arXiv:2605.10655
  20. ProxQuant. Bai, Wang and Liberty, ProxQuant: Quantized Neural Networks via Proximal Operators. 2018. arXiv:1810.00861
  21. BinaryRelax. Yin, Zhang, Lyu, Osher, Qi and Xin, BinaryRelax: A Relaxation Approach For Training Deep Neural Networks With Quantized Weights. 2018. arXiv:1801.06313
  22. PV-Tuning. Malinovskii, Mazur, Ilin, Kuznedelev, Burlachenko, Yi, Alistarh and Richtárik, PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression. 2024. arXiv:2405.14852

Citation

@misc{boson2026lookatmyquant2,
  title  = {LookAtMyQuant 2: Qwen3.5-4B in 1.2 GB},
  author = {{Shin Megami Boson}},
  year   = {2026},
  month  = nov
}