Can We Quantize Everything to 4 Bits: Weights, Activations, and the KV Cache?

This post is based on our paper “ReQAT: Achieving Full-Precision Reasoning Accuracy with 4-bit Floating-Point Quantization-Aware Training”, joint work with Hanyang University presented at ICML 2026 (Oral). The code and training recipe are available at github.com/aiha-lab/ReQAT.

Why Rebellions Cares About the Model Side

A 4-bit number format only lowers cost per token if the model still gives the right answers at 4 bits. Hardware support for low-precision formats has moved faster than models’ ability to tolerate them, and that gap between what the hardware can do and what the model can tolerate is where most low-precision deployments stall.

For reasoning workloads, the gap has been widest. These workloads generate the most tokens and have the most to gain from FP4, but they are also where accuracy breaks down first, so the efficiency stays on the spec sheet instead of showing up in production.

Rebellions builds AI inference infrastructure, and what a customer gets out of it depends on two things: the hardware, and a model that runs on it at the precision it was designed for, with accuracy the customer can accept. Most of that second part happens upstream of us, in how the model is trained and quantized, but we have a direct interest in how it turns out. That interest led us to partner with Hanyang University on low-precision model training, starting from one question: can a reasoning model run with weights, activations and the KV cache all at 4 bits and still match BF16 accuracy?

Background: Low-precision Inference

Quantization reduces the cost of inference by representing a model’s weights and intermediate values in fewer bits. It can be applied after training (post-training quantization, PTQ) or folded into the training loop (quantization-aware training, QAT), and in both cases the aim is a smaller memory footprint, lower bandwidth demand, and higher compute throughput where the hardware supports the reduced precision.

Weight-only quantization was the first form to see broad use. Decoding is usually memory-bound because the full set of weights is read for every generated token, so compressing the weights alone to 4 bits reduces the dominant traffic. GPTQ and AWQ established this approach, and because the arithmetic still ran at 16 bits, the accuracy cost was usually small.

Native hardware support for low-precision floating-point formats, first FP8 and more recently FP4, extended quantization to activations and the KV cache (Figure 1). Quantizing activations allows the matrix multiplications (GEMMs) to run at low precision, which raises compute throughput, and quantizing the KV cache reduces the memory that limits batch size at long context lengths. Microscaled formats such as MXFP4 and NVFP4, which share one scale factor across a small block of elements, made 4-bit floating point workable for activations with large outliers.

FP4 compute cores are now shipping in current-generation GPUs, and native MXFP4 and NVFP4 support is becoming more common in inference hardware, but accuracy has remained the limiting factor on the model side. Activation distributions vary from token to token, and the key cache carries per-channel outliers combined with the rotational structure of RoPE (Rotary Positional Embedding). Each extension of quantization, from weights to activations to the KV cache, has widened the accuracy gap. As a result, most publicly released 4-bit models quantize only the weights or keep sensitive parts of the network at higher precision, and models that run with weights, activations and the KV cache all at 4 bits (W4A4KV4) without a measurable accuracy loss remain uncommon.

The gap is largest for Large Reasoning Models (LRMs) such as the DeepSeek-R1 family, which solve problems by generating long chains of thought, and a single math problem can produce a trace of more than 16K tokens. Quantizing a model distilled from DeepSeek-R1 to NVFP4 W4A4KV4 with a standard PTQ pipeline reduces math reasoning accuracy by a wide margin, and the usual remedies recover only part of it. QAT and quantization-aware distillation (QAD) both improve on PTQ for these models but remain well below BF16 full fine-tuning accuracy, and quantizing the KV cache widens the gap further.

What Breaks at FP4

Our initial assumption was that the fine-tuning budget was the limiting factor. On R1-Llama-8B, however, raising the number of fine-tuning epochs from one to five left the post-PTQ accuracy drop between 11 and 15 points, so we looked instead at where in a reasoning trace quantization errors appear, using the entropy of the next-token distribution at each decoding step.

Grouping more than 1.5 million generated tokens by entropy gave two populations. Low-entropy tokens, which make up roughly 80% of a trace, are mostly digits, variable names and operators (1, x, α, +) that the model predicts almost deterministically. High-entropy tokens are discourse markers such as Then, Let and Hmm, at the points where the reasoning path branches. FP4 quantization flattens the next-token distribution for both populations (Figure 2). A reasonable prior is that the high-entropy positions are the sensitive ones, since a small perturbation there can change the top-1 token.

Two controlled experiments pointed to the low-entropy positions. Routing each next-token prediction to a BF16 or an FP4 model by its entropy showed that sending only the low-entropy tokens to BF16 recovered a large share of the lost accuracy, while sending only the high-entropy tokens recovered little. Injecting Gaussian noise into the logits as a proxy for quantization error gave the same answer: noise on the lowest-entropy 25% of positions caused a steep drop in AIME-2025 accuracy, while the same noise on the highest-entropy 25% had a much smaller effect. The pattern held on R1-Qwen-7B and R1-Qwen-14B and on the more recent Qwen3-4B and Qwen3-8B (Figure 3).

The mechanism is visible in the tail of the distribution. In the low-entropy buckets, FP4 rarely changed the top-1 token, but the probability mass outside it grew substantially relative to BF16. Under greedy decoding this has little effect, but under the temperature sampling used in production serving it raises the probability of an incorrect token at each of thousands of positions in a trace, and one incorrect digit or operator typically invalidates the rest of the derivation. Perturbations at high-entropy positions usually changed the wording without changing the final answer.

In our analysis, the dominant cause of FP4 reasoning failure is increased sampling error at the positions where the model is most confident, and standard QAT, which applies a uniform loss across all tokens, does not target this failure mode.

Our Approach: ReQAT

The analysis above reduces the QAT design problem to two questions, where the training updates should concentrate and how strongly confidence should be reinforced there, plus the separate problem of KV cache quantization. ReQAT (Reasoning-centric QAT), the training framework we propose in the paper, addresses each with one component (Figure 4).

TAQ (Trace-Aligned QAT). TAQ is a two-stage procedure. Stage 1 fine-tunes the model in BF16 on a reasoning dataset, and Stage 2 runs QAT with quantization noise enabled on a subset of the same traces used in Stage 1. When we tracked token entropy during training, fine-tuning alone and QAT alone changed entropy mainly in the high-entropy buckets, while fine-tuning followed by QAT on the same traces also changed the low-entropy buckets. We attribute this to the model having already learned each trace in Stage 1, which leaves the Stage 2 gradient to act on the low-entropy predictions that quantization made fragile. The same additional QAT on different traces did not produce this effect. On R1-Qwen-14B under MXFP4 W4A16 with a 350M-token budget, two-stage training without trace alignment raised AIME accuracy from 62.29% to 62.60%, and trace alignment raised it to 67.29%.

SEM (Selective Entropy Minimization). TAQ alone did not close the gap at W4A4KV4, which suggested that where the signal goes and how sharp the distribution becomes are separate requirements. SEM adds an entropy minimization term to the supervised fine-tuning (SFT) loss, applied only to the lowest-entropy 75% of tokens in each minibatch with a soft weight that increases as entropy decreases. The highest-entropy 25% are excluded so that the branch points of the reasoning keep their original distribution. The benefit scaled with task difficulty: it added little on GSM8K and MATH-500 and produced a clear gain on AIME-120, where traces are longest.

Q-FIT (Quantization-Friendly Initialization via Transformation). TAQ recovered most of the accuracy at W4A4, but accuracy dropped again when the KV cache was quantized to 4 bits, which pointed to the initialization used for QAT. Two transformations are known to reduce key cache quantization error without changing the attention output: channel-pair scaling before RoPE and per-channel shifting after RoPE. The appropriate one differs by layer. Some layers have asymmetric outliers within a RoPE channel pair, where a shared scale is counterproductive, and others have key magnitudes that oscillate across token positions, where a fixed offset drifts over long sequences (Figure 5). Q-FIT calibrates both transformations jointly for each layer to minimize the difference between BF16 and 4-bit KV attention outputs. Neither transformation alone gave a reliable gain in our ablations, while the joint calibration raised NVFP4 W4A4KV4 accuracy from 63.13% to 65.94%.

Accuracy and Throughput Results

The main experiments fine-tune R1-Qwen-14B on the math subset of OpenThoughts3 and evaluate on AIME-120 (AIME 2022~2025, averaged over 8 seeds), with the same total fine-tuning budget for every method.

BF16 fine-tuning and QAT both saturate at around 280M tokens, while ReQAT continued to improve with budget and reached the highest accuracy in every setting we tested, including against AWQ, QuaRot, FlatQuant and QAD.

Matching BF16 accuracy means the throughput advantage of FP4 can be taken at face value. We measured end-to-end output-token throughput relative to BF16 on a single NVIDIA B200 GPU, the platform our university partners used for these experiments, with 1,000 requests of 512-token prompts, batching up to 256 requests, and the KV cache allocated up to the memory limit.

Q-FIT adds a 4~5% overhead relative to native NVFP4 because it disables RoPE fusion, and with that overhead ReQAT reaches 3.05× at 8K output tokens.

The breakdown by output length (Figure 6) shows the source of the speedup. At 2K output tokens, both BF16 and NVFP4 fit a batch of 256 in memory, so the 1.8~1.9× there reflects the compute advantage of 4-bit GEMM alone. At 8K tokens, BF16 can no longer sustain that batch size because of KV cache growth while NVFP4 can, and the speedup exceeds 3×. For long reasoning outputs, most of the FP4 gain comes from the larger batch sizes that KV cache compression permits, so on this workload the benefit of FP4 depends as much on holding accuracy with a 4-bit KV cache as on GEMM speed.

Conclusion

We set out to answer whether a reasoning model can be served with weights, activations and the KV cache all at 4 bits without giving up the accuracy of BF16. For R1-Qwen-14B, the answer is yes: ReQAT reaches 65.94% on AIME-120 under NVFP4 W4A4KV4, against 65.46% for BF16 full fine-tuning with the same training budget, and the resulting model serves at 3.05× the output-token throughput of BF16 at 8K output tokens. The result rests on one observation, that FP4 damages reasoning mainly through sampling errors at the tokens the model is most confident about, and on three changes to the training recipe that follow from it: revisiting the same traces during QAT, sharpening the distribution at low-entropy positions, and calibrating the KV cache transformations per layer before training starts.

Two points from this work carry beyond the specific models and formats we tested. First, where the training signal goes matters more than how much of it there is: within our budgets, more tokens or epochs did not reduce the FP4 gap, while targeting low-entropy positions did. Second, KV cache compression is where most of the throughput gain for long outputs comes from, so holding accuracy with a 4-bit KV cache is a requirement for reasoning workloads.

Results like these are why we care about what happens on the model side. The hardware sets the ceiling, and the training recipe decides how close a deployment gets to it. ReQAT is one answer to that second question, and we expect to keep following it.

References

1. Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. “OPTQ: Accurate Quantization for Generative Pre-trained Transformers” (GPTQ). ICLR 2023.
2. Lin, J. et al. “AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration.” MLSys 2024.
3. Ashkboos, S. et al. “QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs.” NeurIPS 2024.
4. Sun, Y. et al. “FlatQuant: Flatness Matters for LLM Quantization.” 2025.
5. Rouhani, B. D. et al. “Microscaling Data Formats for Deep Learning.” 2023.
6. Open Compute Project. “OCP Microscaling Formats (MX) Specification v1.0.”
7. Alvarez, E. et al. “Introducing NVFP4 for Efficient and Accurate Low-Precision Inference.” 2025.
8. NVIDIA. “Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery.” 2026.
9. Guo, D. et al. “DeepSeek-R1 Incentivizes Reasoning in LLMs through Reinforcement Learning.” Nature, 2025.
10. Yang, A. et al. “Qwen3 Technical Report.” 2025.
11. Guha, E. et al. “OpenThoughts: Data Recipes for Reasoning Models.” 2025.
12. Su, J. et al. “RoFormer: Enhanced Transformer with Rotary Position Embedding.” Neurocomputing, 2024.