LoRA Rank

LoRA rank, written as r, is the central hyperparameter of Low-Rank Adaptation. It sets the inner dimension of the two small matrices that LoRA inserts to approximate a weight update. Instead of training a full weight-update matrix, LoRA learns a product of a down-projection and an up-projection whose shared inner dimension is r, so the rank directly determines how many trainable parameters the adapter adds — and, by extension, how much of your GPU actually gets spent on learning rather than on holding frozen weights.

What the rank actually controls

For a frozen weight matrix of size d × k, a LoRA adapter adds roughly r × (d + k) trainable parameters instead of d × k. On the large projection matrices inside a transformer, that difference is enormous: a rank of 16 on a 4096-dimensional layer trains well under one percent of what full fine-tuning would touch. The mathematical bet behind this is that the change a task demands has low intrinsic rank — the base model already knows how to write and reason, and the adapter only needs to steer it. The rank is the width of that steering channel.

Choosing a rank

A higher rank gives the adapter more capacity to capture complex task-specific patterns, but it increases memory use, slows each step slightly, and raises the risk of overfitting on small datasets. A lower rank is cheaper and often generalizes better when training data is limited. Common values run from about 8 or 16 for style and formatting tasks up to 64, 128, or 256 for harder, data-rich scenarios such as teaching a new domain vocabulary. Because the adapter weights can be merged back into the base model after training, the chosen rank affects training cost but not inference latency. Rank rarely acts alone: it is paired with the scaling hyperparameter described under LoRA alpha, which is typically set relative to r (a common convention is alpha equal to r or to 2r), and with the choice of which layers receive adapters — attention projections only, or every linear layer in the network.

Rank on a single consumer GPU

For someone fine-tuning an open-weight model on the same workstation that runs their node and their Ollama stack, rank is the main dial for balancing quality against the VRAM you actually have. A sensible workflow is to start low — rank 8 or 16 — confirm the task learns at all, then raise the rank only if validation results plateau below target. Doubling the rank doubles adapter memory but often yields diminishing returns past the point where the task's real complexity is captured; published ablations repeatedly show small ranks matching large ones on narrow tasks. When VRAM is the binding constraint, combining a modest rank with a quantized base model — the QLoRA recipe described under quantization — usually beats cranking the rank on a smaller model.

Rank in the wider PEFT family

One subtlety worth knowing: rank interacts with data quantity more strongly than with model size. A 70B model fine-tuned on three hundred examples usually wants a lower rank than a 7B model fine-tuned on fifty thousand, because the adapter's capacity should track what the dataset can actually support. If training loss falls while validation loss climbs, the first suspect is a rank oversized for the data. Conversely, if the model stubbornly refuses to pick up a genuinely complex behavior no matter how long you train, capacity — rank, or the set of adapted layers — is the dial to reach for before blaming the data.

LoRA rank carries over to variants such as DoRA, which applies a LoRA-style low-rank update to the directional component of each weight while training magnitude separately, and to the broader family of parameter-efficient fine-tuning methods that all trade a little expressiveness for a large drop in trainable parameters. Whatever the variant, the discipline is the same one a miner applies at the tuning bench: change one dial at a time, measure, and let the validation numbers — not intuition — tell you when more capacity is actually buying you something. For the parent technique itself, see LoRA (Low-Rank Adaptation).