How Much VRAM Do You Need to Run an LLM?

VRAM (Video Random Access Memory) is the most critical resource when running Large Language Models locally. This guide breaks down exactly what you need.

The Simple Formula

For a model to run, you need roughly:

- 1 byte per parameter for FP32 (full precision)

- 0.5 bytes per parameter for FP16/BF16 (half precision)

- 0.25 bytes per parameter for INT8 (8-bit quantization)

- 0.125 bytes per parameter for INT4 (4-bit quantization)

VRAM by Model Size

Model

FP16

INT8

INT4

-------------------------

7B params

14GB

7GB

3.5GB

13B params

26GB

13GB

6.5GB

33B params

66GB

33GB

16.5GB

70B params

140GB

70GB

35GB

Practical Recommendations

- 7B models (Q4): Any modern GPU with 8GB+ — RTX 3060, A4000

- 13B models (Q4): 12-16GB VRAM — RTX 3080, A5000

- 70B models (Q4): Multi-GPU or cloud — or use a provider like Groq

- 33B models (Q4): 24GB VRAM — RTX 3090/4090

KV Cache

Don't forget the KV cache! It also consumes VRAM proportional to context length. A 4096-token context can add several GB overhead on large models.

Use the [VRAM Calculator](/tools/vram/) to estimate exactly what you need for your specific setup.

The Bottom Line

If you have 24GB VRAM, you can comfortably run most 7B and 13B models, and some 33B models in Q4. For 70B models, consider cloud GPUs or providers like Groq.