Regularization

Regularization is a collective name for techniques that deliberately constrain a model during training so it generalizes better to unseen data. In practice this usually means adding a penalty to the loss function that grows as the model's parameters become large or complex, nudging the optimizer toward simpler solutions that are less likely to memorize noise.

Common forms

Two of the most familiar penalties are L2 regularization (also called weight decay), which penalizes the squared magnitude of the weights and shrinks them smoothly toward zero, and L1 regularization, which penalizes their absolute magnitude and tends to drive some weights exactly to zero, effectively performing feature selection. Beyond these, deep learning relies heavily on dropout, which randomly disables neurons during training, and early stopping, which halts training once validation performance stops improving. Data-side methods such as augmentation act as a form of regularization too.

Why it is a balance

Regularization is a dial, not a switch. Too little leaves a model free to overfit; too much pushes it toward underfitting, where it can no longer capture real structure. The right strength is itself a tuning decision, set before training and adjusted by watching validation results. For anyone training or fine-tuning models on their own hardware, understanding regularization is what separates a model that holds up in the field from one that looks great in training and falls apart in use.

See also our entries on overfitting, underfitting, and hyperparameter tuning.