LLM Fine-Tuning VRAM Planner: Full vs LoRA vs QLoRA

Quick answer

It depends far more on the method than the model. Full fine-tuning needs roughly 16 bytes per parameter (weights + gradients + Adam optimizer states) before activations — about 100+ GB for a 7B model, i.e. multiple data-centre GPUs. LoRA freezes the base weights, so you only hold them in FP16 (~2 B/param, ~14 GB for 7B) plus a tiny trainable adapter — a single 24 GB card. QLoRA quantizes the frozen base to 4-bit (~0.5 B/param, ~4 GB for 7B), which is how a 65B model was fine-tuned on one 48 GB GPU. The planner below computes the parameter/optimizer memory exactly and adds an activation estimate you can tune, then checks it against real GPUs.

Model size (billions of params)

Fine-tuning method

Batch size

Sequence length (tokens)

Gradient checkpointing (recompute activations — big VRAM saving, ~20% slower)

How the numbers are built

The parameter memory is exact for mixed-precision Adam, in bytes per parameter:

Component	Full	LoRA	QLoRA
Base weights	2 (FP16)	2 (FP16, frozen)	0.5 (4-bit, frozen)
Gradients	2 (FP16)	0 (base frozen)	0 (base frozen)
Optimizer (Adam m+v, FP32)	8	~0 (adapter only)	~0 (adapter only)
FP32 master copy	4	0	0
Total (per param)	~16 B	~2 B	~0.5 B

LoRA/QLoRA also train a small adapter (typically well under 1% of the base parameters), which adds a little for its own weights, gradients and optimizer state — small enough that the frozen base dominates. Activations are the genuinely variable part: they scale with batch size and sequence length and are only a rough estimate here. Gradient checkpointing (on by default above) trades compute for memory and cuts activation VRAM dramatically — turn it on for large models on a small card. Always leave headroom; these are floor estimates, not guarantees.

Where this fits

This planner is the training-time companion to the inference tools: GPU ↔ local-LLM fit (inference) · AI-GPU hardware database · quantization format comparison · local AI runtime comparison · self-hosted LLM security.

Related products, repair, and setup paths

Last reviewed July 21, 2026.