Definition
LoRA alpha (often written lora_alpha) is the scaling hyperparameter that controls how strongly a LoRA adapter's low-rank update is applied to the frozen base weights. In the standard formulation the update is multiplied by the ratio of alpha to rank, written as α/r, before being added back into the model. Alpha therefore governs the strength of the fine-tuning signal, while the rank governs its capacity.
The alpha-over-rank ratio
Scaling by α/r is meant to keep the magnitude of the update roughly consistent as you change the rank, so that increasing rank does not also inadvertently amplify the adapter's effect. Setting alpha equal to the rank applies the update at unscaled (1x) strength; a larger alpha relative to rank pushes the adapter harder, while a smaller one softens it. This is why alpha cannot be reasoned about in isolation from the LoRA rank it is divided by.
Common conventions
Two patterns dominate in practice. One fixes alpha at a constant such as 16 or 32 regardless of rank. The other ties alpha to rank, commonly α = 2r, which the original Microsoft implementation favours (for example r=8 with alpha=16). Some quantized-training recipes deliberately use a smaller alpha relative to rank. A practical tip: if you raise the rank but leave alpha fixed, you are also weakening the per-parameter update, so adjust the two together.
Alpha is part of the same hyperparameter set inherited by LoRA variants like DoRA, and it complements the capacity set by LoRA rank.
In Simple Terms
LoRA alpha (often written lora_alpha) is the scaling hyperparameter that controls how strongly a LoRA adapter’s low-rank update is applied to the frozen base weights.…