Can my GPU run this model?

Accurate VRAM estimation using the full formula — weights, KV cache (GQA-aware), activations, and CUDA overhead.

1. Select GPU

2. Choose model

3. Quantization

4. Context length — 8k tokens

5128k32k131k

5. Inference engine

6. Use case

The formula behind the estimate

V_weights

params × bits_per_param ÷ 8 ÷ 1.07
Q4_K_M ≈ 4.5 bits → 70B × 4.5/8 ≈ 39.4 GiB

V_KV (GQA-aware)

2 × n_layers × n_kv_heads × d_head × context × 2 bytes ÷ 1.07
Llama 3: 8 KV heads (not 64) = 8× less KV cache

V_activations

~1 GB inference / ~3 GB activations (fine-tuning)
Forward pass activations only — full fine-tuning needs significantly more (gradients + optimizer states).

V_overhead

CUDA context + engine overhead
llama.cpp ≈ 0.8 GB, vLLM ≈ 3.5 GB

GQA note: Grouped-Query Attention (Llama 3, Qwen 2.5, DeepSeek R1) drastically reduces KV cache — a model with 8 KV heads uses 8× less KV cache than if it had 64 full attention heads.