Quantization Explained: Q4 vs Q8 vs Q16

Quantization reduces model size by using fewer bits to represent each weight. Here's what that means for quality and performance.

What Is Quantization?

When models are trained, weights are stored in high precision (FP32 = 32 bits, FP16 = 16 bits). Quantization converts these to lower precision formats (INT8 = 8 bits, INT4 = 4 bits).

The Quality/Speed Tradeoff

Format

Size vs FP16

Quality

Speed

-------------------------------------

FP16

100%

Perfect

Baseline

Q8 (INT8)

50%

~99%

1.3-1.5x faster

Q4 (INT4)

25%

~95-98%

1.5-2x faster

Q2 (INT2)

12.5%

~85-90%

2-3x faster

When to Use Each

- Q8: When you have the VRAM and want minimal quality loss

- Q4: The sweet spot for most users — 4-bit quantization is good enough for almost all tasks

- Q2: Only for extreme VRAM constraints — quality suffers noticeably

What Affects Quality Loss

- Model size: Larger models tolerate lower quantization better

- Task type: Reasoning tasks are more sensitive than casual chat

- Calibration data: Good quantization calibration data matters

Recommendation

For most users: Q4_K_M or Q5_K_S — these are well-tested formats with excellent quality/size ratios. Use [VRAM Calculator](/tools/vram/) to see what fits in your GPU.