KL Penalty (KL Divergence)

The KL penalty is a regularization term used throughout reinforcement-learning fine-tuning of language models. It measures the Kullback-Leibler (KL) divergence, a statistical distance, between the model being trained (the policy) and a frozen reference model, usually the supervised fine-tuned checkpoint the training started from. By penalizing large divergence, it keeps the model anchored to coherent, fluent language even as reinforcement learning pushes it toward higher reward.

Why the leash is necessary

During RLHF, the reward model is only an approximation of human preference. Without a constraint, the policy will rapidly drift into regions of probability space that score high under the reward model but produce repetitive, nonsensical, or stylistically broken text, a failure mode known as reward hacking. The reward model has blind spots, and an unconstrained optimizer will find and exploit them. A coefficient called beta sets how strong the penalty is: too small and the model breaks; too large and it barely learns. Practitioners often keep the running KL between roughly 0 and 10.

Where it appears

The KL penalty is a core ingredient in PPO, GRPO, and RLAIF. Interestingly, RL with a KL penalty can be interpreted as Bayesian inference, with the reference model as a prior. For anyone fine-tuning an open model, the KL term is the safety leash that lets you push for new behavior without forgetting how to write.

It works hand in hand with reward shaping to keep optimization honest.